The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,465 other followers

Archive for the ‘ESXi6.7’ Category

Some links to post about ESXi 6 and ESXi 7 storage and storage speed issues

Posted by jpluimers on 2021/09/09

For my link archive (most via [Wayback] sata very slow after ESXi 6.7 update – Google Search):

Two takeaways already:

–jeroen

Posted in ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

“fixing” ESXi “rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]”

Posted by jpluimers on 2021/08/30

Reminder to self: create a static ESXi binary for a recent rsync release.

Quite a few people have bumped into rsync erroring out with “large” sets of files (where large can be as low as ~1000), like for instance Tj commenting on my post “ESXi 5.1 and rsync – damiendebin.net.”:

ERROR: out of memory in receive_sums [sender] │······
rsync error: error allocating core memory buffers (code 22) at util2.c(102) [sender=3.1.1] │······
rsync: [generator] write error: Broken pipe (32) │······

I bumped into this myself as well, even when updating from rsync 3.1.0 to 3.1.2.

There are various static rsync for ESXi around. Just a few of them for completeness:

There is also 3.0.9 (via [Wayback] VMware ESXi 5.1 rsync 3.0.9 statically linked binary erstellen – bachmann-lan.de), but it has a VMFS bug ([Wayback] 8177 – Problems with big sparsed files) as per [Wayback] ESXi 5.1 and rsync – damiendebin.net.)

The good news is that it is fixed in 3.2.2 as a user-configurable setting, but since there is no ESXi build yet (see reminder above)…

Anyway: [Wayback] 12769 – error allocating core memory buffers (code 22) depending on source file system

Wayne Davison 2020-06-26 03:56:35 UTC
I fixed the allocation args to be size_t values (and improved a bunch of allocation error checking while I was at it).

I then added an option that lets you override this allocation sanity-check value. The default is still 1G per allocation, but you can now specify a much larger value (up to "--max-alloc=8192P-1").

If you want to make a larger value the default for your copies, export RSYNC_MAX_ALLOC in the environment with the size value of your choice.

Committed for release in 3.2.2.

This is what happens with 3.1.2 and 3.1.3:

time rsync -aiv --info=progress2 --progress --partial --existing --inplace /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
sending incremental file list
              0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1000/1259)
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]
Command exited with non-zero status 22
real    0m 0.87s
user    0m 0.10s
sys 0m 0.00s
time rsync -aiv --info=progress2 --progress --partial --ignore-existing --sparse /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
sending incremental file list
              0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1000/1259)
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]
Command exited with non-zero status 22
real    0m 0.28s
user    0m 0.12s
sys 0m 0.00s
Finished

I was lucky that [Wayback] “rsync error: error allocating core memory buffers” protocol version “3.1.2” – Google Search got me a result so quickly: add a --protocol-29 and you are set.

The first result (Wayback has the results reversed from what got) didn’t fix it. The second did.

  1. [Wayback] 225761 – net/rsync long path causes buffer overflow (update to 3.1.3)
  2. [Wayback/Archive.is] AIX Open Source – IBM Power Systems Community: rsync out of memory

    As a work around, I added “--protocol=29” to one of our servers that was consistently failing with “ERROR: out of memory in flist_expand [receiver]” “rsync error: error allocating core memory buffers (code 22) at util2.c(105) [receiver=3.1.3]” in rsync-3.1.3-2.ppc

    I read the man page and started experimenting with the protocol version until I lowered it enough to get it to work consistently.

The problem might be that running on the ESXi gives you limited memory, but then some 10k files should not use more than like half a megabyte of memory.

Sometime I will dig deeper into the protocol version differences, for now a list of files I think will be relevant for that (mainly look for protocol_version):

Some web pages mentioning the --protocol option and might give me more insight in the protocol differences:

With --protocol=29, time estimation is way off, but there are no errors:

time rsync -aiv --info=progress2 --progress --partial --existing --inplace --protocol=29 /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
building file list ... 
9059 files to consider
.d..t...... isos/
         27,593   0%    0.00kB/s    0:00:06 (xfr#1, to-chk=0/9059)   

sent 212,594 bytes  received 268 bytes  20,272.57 bytes/sec
total size is 3,055,677,645,398  speedup is 14,355,204.99
real    0m 13.31s
user    0m 1.35s
sys 0m 0.00s

time /vmfs/volumes/5791a3e1-0b9368de-4965-0cc47aaa9742/local-bin/rsync -aiv --info=progress2 --progress --partial --ignore-existing --sparse --protocol=29 /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
building file list ... 
9059 files to consider
>f+++++++++ isos/EN-Windows-XP-SP3-VL.iso
...
cd+++++++++ ESXi65.filesystem-root/usr/share/
216,868,164,639   7%   40.64MB/s    1:24:48 (xfr#2571, to-chk=0/9059)   

sent 216,894,938,870 bytes  received 57,858 bytes  42,582,702.80 bytes/sec
total size is 3,055,677,645,398  speedup is 14.09
real    1h 24m 58s
user    34m 5.59s
sys 0m 0.00s
Finished

Even not on ESXi, there were just a few people bumping into this, so I wonder why there are so few matches on [Wayback] “ERROR: out of memory in flist_expand [sender]” “sender=3.1” – Google Search:

–jeroen

Posted in *nix, *nix-tools, ash/dash development, Development, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, rsync, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: finding unmounted VMFS volumes

Posted by jpluimers on 2021/08/26

Sometimes, especially when ESXi thinks a volume is bad, but the ESXi S.M.A.R.T. logging does not indicate so, it boots without mounting some of the VMFS volumes as datastores.

It took me a while to find the right command to list those, but I’m glad I found it:

# esxcfg-volume -l
Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs).
VMFS UUID/label: 552f5788-33e30274-8dba-001f29022aed/850EVO1TBR1B
Can mount: Yes
Can resignature: Yes
Extent name: naa.600605b00aa054a0ff0000210221eaf8:1 range: 0 - 953087 (MB)

Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
VMFS UUID/label: 5ad4af1b-f3ae285c-e0f4-0cc47aaa9742/IntelNVMe1TB-BTPY750500091P0H
Can mount: Yes
Can resignature: Yes
Extent name: eui.0000000001000000e4d25cd29be94e01:1 range: 0 - 976639 (MB)

Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
VMFS UUID/label: 5ad4aeea-6954841c-470e-0cc47aaa9742/IntelNVMe1TB-BTPY7425047S1P0H
Can mount: Yes
Can resignature: Yes
Extent name: eui.0000000001000000e4d25c0e8dc74e01:1 range: 0 - 976639 (MB)

Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs).
VMFS UUID/label: 552d3e82-ccee005a-b719-001f29022aed/850EVO1TBR1A
Can mount: Yes
Can resignature: Yes
Extent name: naa.600605b004b87fb01cc22b3487cbf9a9:1 range: 0 - 953087 (MB)

It is a similar esxcfg-* command with very long output:

# esxcfg-scsidevs -m
t10.ATA_____WDC_WD3200BEKT2D22F3T0________________________WD2DWXE708P5C200:1 /vmfs/devices/disks/t10.ATA_____WDC_WD3200BEKT2D22F3T0________________________WD2DWXE708P5C200:1 502124c5-cc47df0d-3ef7-a0369f0e1091  0  simplesata
t10.ATA_____SAMSUNG_MZHPV512HDGL2D00000______________S1X1NYAGB09589______:1  /vmfs/devices/disks/t10.ATA_____SAMSUNG_MZHPV512HDGL2D00000______________S1X1NYAGB09589______:1  5791a3e1-0b9368de-4965-0cc47aaa9742  0  Samsung512NVME
t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0HA01043L_____:1   /vmfs/devices/disks/t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0HA01043L_____:1   59a018dd-07a9931a-7135-0cc47aaa9742  0  Samsung850-2TB-S3D4NX0HA01043L
t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S2KMNCAGB04321L_____:1   /vmfs/devices/disks/t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S2KMNCAGB04321L_____:1   59a01b5c-c46ae5be-f4eb-0cc47aaa9742  0  Samsung850-2TB-S2KMNCAGB04321L
naa.5000c50087762d1b:1                                                       /vmfs/devices/disks/naa.5000c50087762d1b:1                                                       59a33f7b-66df7c00-11b0-0cc47aaa9742  0  ST6000VX0001-1SH-Z4D3DZZV
naa.600605b00aa054a0ff000021022683ae:1                                       /vmfs/devices/disks/naa.600605b00aa054a0ff000021022683ae:1                                       532cd010-6e8c01d1-45be-001f29022aed  0  Raid6SSD

or a totally different command than the other way around (listing all mounted datastores):

# esxcli storage vmfs extent list
Volume Name                     VMFS UUID                            Extent Number  Device Name                                                                 Partition
------------------------------  -----------------------------------  -------------  --------------------------------------------------------------------------  ---------
simplesata                      502124c5-cc47df0d-3ef7-a0369f0e1091              0  t10.ATA_____WDC_WD3200BEKT2D22F3T0________________________WD2DWXE708P5C200          1
Samsung512NVME                  5791a3e1-0b9368de-4965-0cc47aaa9742              0  t10.ATA_____SAMSUNG_MZHPV512HDGL2D00000______________S1X1NYAGB09589______           1
Samsung850-2TB-S3D4NX0HA01043L  59a018dd-07a9931a-7135-0cc47aaa9742              0  t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0HA01043L_____            1
Samsung850-2TB-S2KMNCAGB04321L  59a01b5c-c46ae5be-f4eb-0cc47aaa9742              0  t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S2KMNCAGB04321L_____            1
ST6000VX0001-1SH-Z4D3DZZV       59a33f7b-66df7c00-11b0-0cc47aaa9742              0  naa.5000c50087762d1b                                                                1
Raid6SSD                        532cd010-6e8c01d1-45be-001f29022aed              0  naa.600605b00aa054a0ff000021022683ae                                                1

Yes, this has to do with my post earlier today: NVMe and SATA health data on ESXi: some links to investigate.

You can mount the volume persistently (with the -M or --persistent-mount option) or transiently (with the -m or --mount option), where you can either refer to the volume by name or by uuid:

esxcfg-volume
-l|--list               List all volumes which have been
                        detected as snapshots/replicas.
-m|--mount              Mount a snapshot/replica volume, if 
                        its original copy is not online.
-u|--umount             Umount a snapshot/replica volume.
-r|--resignature        Resignature a snapshot/replica volume.
-M|--persistent-mount   Mount a snapshot/replica volume
                        persistently, if its original copy is
                        not online.
-U|--upgrade            Upgrade a VMFS3 volume to VMFS5.
-h|--help               Show this message.

So in my case, I mounted two of the volumes by hand (of which later one of the NVMe devices – IntelNVMe1TB-BTPY750500091P0H – died within warranty and the other – IntelNVMe1TB-BTPY7425047S1P0H – was causing read errors within warranty, so sent both for RMA):

  • # esxcfg-volume --persistent-mount IntelNVMe1TB-BTPY750500091P0H
    Persistently mounting volume IntelNVMe1TB-BTPY750500091P0H
  • # esxcfg-volume --persistent-mount IntelNVMe1TB-BTPY7425047S1P0H
    Persistently mounting volume IntelNVMe1TB-BTPY7425047S1P0H

Alternatively, you can use esxcli to do the same: [Wayback] Mount a Datastore with ESXCLI

Mount a Datastore with ESXCLI

The esxcli storage filesystem commands support mounting and unmounting volumes. You can also specify whether to persist the mounted volumes across reboots by using the --no-persist option.

Use the esxcli storage filesystem command to list mounted volumes, mount new volumes, and unmount a volume. Specify one of the connection options listed in Connection Options for vCLI Host Management Commands in place of <conn_options>.

Procedure:

  1. List all volumes that have been detected as snapshots.

    esxcli <conn_options> storage filesystem list
  2. Run esxcli storage filesystem mount with the volume label or volume UUID.

    esxcli <conn_options> storage filesystem volume mount --volume-label=<label>|--volume-uuid=<VMFS-UUID>

    Note

    This command fails if the original copy is online.

What to do next

You can later run esxcli storage filesystem volume unmount to unmount the snapshot volume.

esxcli <conn_options> storage filesystem volume unmount --volume-label=<label>|--volume-uuid=<VMFS-UUID>

References:

Knowing the command made me – through [Wayback] “esxcfg-volume -l” – Google Search – found out many ran in the same issue, including myself (:

–jeroen

Posted in ESXi6, ESXi6.5, ESXi6.7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

The curious Kabri | How to force VMware to generate a new MAC address for a virtual machine

Posted by jpluimers on 2021/08/06

[WayBack] The curious Kabri | How to force VMware to generate a new MAC address for a virtual machine

  1. Shut down the Guest OS.
  2. Open up the .vmx file.
  3. Delete the following lines (that begin with…):
    ethernet0.addressType
    uuid.location =
    uuid.bios =
    ethernet0.generatedAddress =
    ethernet0.generatedAddressOffset =
  4. Boot up the Guest OS again, and it should generate new details in the vmx file (I’d check afterwards to be doubly sure).

In my experience, start with the bold values.

If the address is the same, fiddle with ethernet0.generatedAddressOffset

Be careful with the other values, as it might force your OS to think so much hardware has changed, that license keys have become invalid.

Via: [WayBack] Re-generate MAC addresses for VMs |VMware Communities

Related:

–jeroen

Posted in ESXi6, ESXi6.5, ESXi6.7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi 6.7 and up: embedded busybox wget understands https (finally!)

Posted by jpluimers on 2021/07/16

At last, somewhere around ESXi 6.7, the built-in BusyBox tool wget started to support the https protocol.

Yay!

Which means that workarounds in these answers are not needed any more:

It was a big BusyBox version bump (from 1.22.x to 1.29.x)  between ESXI 6.5 and 6.7, especially since 1.26.x versions (that introduced wget) have been available way before ESXi 6.5 came out: [WayBack] BusyBox.

From an ESXi 6.5U2 host

[root@ESXi-X10SRH-CF:/tmp] wget https://www.example.org
wget: not an http or ftp url: https://www.example.org
[root@ESXi-X10SRH-CF:/tmp] wget --help
BusyBox v1.22.1 (2018-07-23 19:34:04 PDT) multi-call binary.

Usage: wget [-csq] [-O FILE] [-Y on/off] [-P DIR] [-U AGENT] URL...

Retrieve files via HTTP or FTP

    -s  Spider mode - only check file existence
    -c  Continue retrieval of aborted transfer
    -q  Quiet
    -P DIR  Save to DIR (default .)
    -O FILE Save to FILE ('-' for stdout)
    -U STR  Use STR for User-Agent header
    -Y  Use proxy ('on' or 'off')

[root@ESXi-X10SRH-CF:/tmp] vmware -l
VMware ESXi 6.5.0 Update 2

From an ESXi 6.7U2 host

[root@ESXi-X9SRI-3F:/tmp] wget https://www.example.org
Connecting to www.example.org (93.184.216.34:443)
index.html           100% |******************************************************************************************|  1270  0:00:00 ETA
[root@ESXi-X9SRI-3F:/tmp] wget --help
BusyBox v1.29.3 (2018-11-02 15:37:50 PDT) multi-call binary.

Usage: wget [-c|--continue] [--spider] [-q|--quiet] [-O|--output-document FILE]
    [--header 'header: value'] [-Y|--proxy on/off] [-P DIR]
    [-S|--server-response] [-U|--user-agent AGENT] URL...

Retrieve files via HTTP or FTP

    --spider    Only check URL existence: $? is 0 if exists
    -c      Continue retrieval of aborted transfer
    -q      Quiet
    -P DIR      Save to DIR (default .)
    -S          Show server response
    -O FILE     Save to FILE ('-' for stdout)
    -U STR      Use STR for User-Agent header
    -Y on/off   Use proxy
[root@ESXi-X9SRI-3F:/tmp] vmware -l
VMware ESXi 6.7.0 Update 2

–jeroen

Posted in ESXi6.5, ESXi6.7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

 
%d bloggers like this: