The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘ESXi5.5’ Category

« Previous Entries

VMware ESXi: How to Kill an Unresponsive (Stuck) Virtual Machine | Windows OS Hub

Posted by jpluimers on 2022/04/21

For my link archive: [Wayback] VMware ESXi: How to Kill an Unresponsive (Stuck) Virtual Machine | Windows OS Hub

Commands covered:

  • esxcli vm process list
  • esxcli vm process kill --type=[soft,hard,force] --world-id=WorldNumber
  • esxtop
  • ps
  • kill

–jeroen

Posted in ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

Run ESXi from a USB Flash Drive: A How-To-Guide

Posted by jpluimers on 2022/04/13

Guessing the [Wayback] Run ESXi from a USB Flash Drive: A How-To-Guide by just the abstract does not show the value enough:

A USB flash drive can be used not only for installation media – you can also run ESXi from USB flash drives or SD flash cards and boot from these devices.

In fact, the article shows way more, including:

  • how the partitions on USB/SD devices are built as compared to HDD devices, and how they even differ depending on USB/SD sizes
  • how to backup/restore the USB/SD boot devices (so you can stock them in case of failure)

This is very important, because every now and then, these USB and SD devices fail (see for instance [Wayback] Solved: Remount boot filesystem on a running system. – VMware Technology Network VMTN), so knowing what to do then is key and helps handling errors like this one:

Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage.

A every useful article for my link archive!

Related: ESXi: storing an ISO 8601 time-stamped backup tarball locally

Read the rest of this entry »

Posted in ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

When some virtual machines cannot run VMware Tools: Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly)

Posted by jpluimers on 2022/04/12

An interesting set of scripts from [Wayback/Archive.is] No Joke IT: Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly).

If all ESXi virtual machines support running of VMware Tools, then the solution is a plain /sbin/shutdown.sh && /sbin/poweroff (see [Wayback/Archive.is] No Joke IT: Shut down ESXi 5.1 guest VMs and the host (free edition) via SSH – the easy way!).

Code is in the repository at [Wayback/Archive.is] sixdimensionalarray/esxidown: A shell script to shutdown VMware ESXi host servers, with these two main files:

Note: the No Joke IT web-site has vanished, so only the [Wayback] and [Archive.is] links of it still work. The github code was still there at the time of writing.

Via: [Wayback] Solved: Read only Files – VMware Technology Network VMTN

Related: Some notes on replacing parts of a text file with template text using sed on a Busybox system.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: for my link archive – links about “vim-cmd vmsvc/message” (lots of interesting scripts for deployment scenarios)

Posted by jpluimers on 2022/02/16

In ESXi: on the console/ssh, when a moved VM pauses during power-on: show which VMs have messages waiting, then answer them, I searched for [Wayback] “vim-cmd vmsvc/message” – Google Search in order to see which messages were available.

That search revealed a lot more links, so here are the ones I found most interesting:

 

–jeroen

Read the rest of this entry »

Posted in ESXi4, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: for my link archive “Developing for VMware ESXi”

Posted by jpluimers on 2022/02/15

This post amends the post last week on rsync backup of your ESXi box: How to make a statically linked rsync binary « The Wiert Corner – irregular stream of stuff.

Two weeks ago, I posted about Source: ESXi: searching for “vim-cmd vmsvc/message” lead me to the “Managing ESXi Without VI Client” series of blog posts.

It got me looking more deeply in the VM-Help site, and I found [Wayback] Developing for VMware ESXi – Virtual Machine and VPS Tutorials, for which I have materialised the links below and checked their WayBack machine status.

Compiling Utilities for ESXi

Given that ESXi is not based on Linux you won’t find any installer which you could use to install any Linux components that you might want to add to ESXi. However, ESXi does make use of a number of Open Source packages such as OpenSSL, Python, and Openwsman (WS-Management). The key to compiling a utility for ESXi is creating a statically linked version of the tool. With a statically linked version, there are no dependencies on other libraries that may not be present on ESXi. The downside to this method of compiling is that the utility may be larger than a dynamically linked version. With a dynamically linked version the utility assumes that other libraries are present and can rely on subroutines within those libraries.

Compiling rsync – [Wayback] How to compile a statically linked rsync binary for ESXi
Compiling Busybox – [Wayback] How to compile Busybox for ESXi … kind of Part 1
Discussion of compiling UNFS – http://www.vm-help.com/forum/viewtopic.php?f=16&t=2280&p=10185&e=10185 (not archived in the WayBack machine nor available on-line)
Notes on compling binaries – [Wayback] Stjepan Groš – Homepage

Compiling Drivers for ESXi

Given the common misunderstanding that ESXi is Linux based, a new user often inquires about the process of copying a Linux driver to their ESXi install. This is not possible. ESXi includes a Linux driver compatibility module. This allows for Linux source code to be used to compile drivers for ESXi, but the drivers are still specific to ESXi. The following links provides some samples and notes for compiling drivers for ESXi.

Compiling a Silicon Image 3132 driver – [Archive.is] Wayback: Adding Driver Support to VMware ESXi 4 | Tip’s Notebook
Compiling a Marvell sky2 driver – [Wayback] Using a Marvell LAN card with ESXi 4 – KernelCrash

(Note: This post was initially written when ESXi 4.0 was available. As of late 2010, ESXi 4.1 has been released, and it does actually include a sky2 driver that may or may not work with various Marvell LAN chipsets. The post is still relevant (especially the comments)  if your particular Marvell chipset does not work with the sky2 driver in ESXI 4.1. Also, the post is relevant if you’re interested in porting other network drivers to ESXi)

Open Virtualization Drivers development notes

Being from the ESXi 4 and 5 era, the links seem to hold up remarkably well. Despite ESXi 3 being Linux based (see [Archive.is] VMware ESX Server – Wikipedia, the free encyclopedia), as opposed to ESXi 4 and up that run a microkernel, Linux based tools still can be used to develop tooling and drivers.

–jeroen

Posted in ESXi4, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: searching for “vim-cmd vmsvc/message” lead me to the “Managing ESXi Without VI Client” series of blog posts

Posted by jpluimers on 2022/02/02

Last week, I wrote about ESXi: on the console/ssh, when a moved VM pauses during power-on: show which VMs have messages waiting, then answer them.

The “vim-cmd vmsvc/message” – Google Search there lead me to this very interesting series of blog posts on managing ESXi Without VI Client.

It was written in the ESXi 4 and 5 era (for a time-frame, see [Wayback] VMware ESXi Release and Build Number History | virten.net) where VIC – the [Wayback] VMware Infrastructure Client, later named VC:  [Wayback] vSphere Client – ruled.

The series content holds remarkably well now that VIC/VC got first replaced by the [Wayback] vSphere Web Client (often called [Wayback] vSphere HTML 5 Client and started out of the [Wayback] ESXi Embedded Host Client fling).

Anyway, here are the posts in the series:

  1. [Wayback] Managing ESXi Without the VI Client – Part 1 – Virtual Machine and VPS Tutorials – initial setup and creating a VM
  2. [Wayback] Managing ESXi Without the VI Client – Part 2 – Virtual Machine and VPS Tutorials – add a license key, enable VM automatic startup and shutdown and unregister a VM
  3. [Wayback] Managing ESXi without the VI Client – Part 3 – Virtual Machine and VPS Tutorials – create a virtual switch and configure a firewall VM
  4. [Wayback] Managing ESXi without the VI Client – Part 4 – Virtual Machine and VPS Tutorials – install an update for ESXi
  5. [Wayback] Managing ESXi Without VI Client – Part 5 – Virtual Machine and VPS Tutorials – creating a datastore, then moving a VM between datastores

–jeroen

Posted in ESXi4, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

VMware ESXi and vSphere: vmNIC speeds are limited by your CPU and RAM speeds, only in part by the vNIC drivers

Posted by jpluimers on 2022/01/11

Still a lot of people think that network speed depends on the vNIC driver and vNIC speed settings.

This is not true: it mainly depends on CPU and RAM speeds as that is where the bottleneck of virtual network processing is.

What does matter is the VM/host overhead is far less when drivers use paravirtualisation (i.e. shortcutting calls from the guest OS to the hypervisor) like PVSCSI for disk or VMXNET3 for networking. This means that VMXNET3 has even more performance than E1000.

Read the rest of this entry »

Posted in ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

Some links and graphs on ESXi capping/throtteling disk speeds

Posted by jpluimers on 2022/01/06

As promised in “Solution” on ESXi 6.7 smartinfo throwing error Cannot open device, here are a few links in capping throttling disk speeds by ESXi followed by a few graphs of my own:

My own observations on ESXi 6.7 update 3:

  1. One rsync operation:
    1 rsync from 860 EVO SSD to 960 PRO NVMe

    1 rsync from 860 EVO SSD to 960 PRO NVMe

  2. Two rsync operations:
    2 rsync from 860 EVO SSD to 960 PRO NVMe

    2 rsync from 860 EVO SSD to 960 PRO NVM

  3. Resume actions were about 10 times faster than the single rsync read speeds:
    Resume action

    Resume action

  4. Suspend actions were between 4 and 6 times faster than rsync write speeds:
    Start of suspend action

    Start of suspend action

     

    Finish of suspend action

    Finish of suspend action

For each rsync operation, I had a separate SSH session going, and the speed doubled.

The resume action of all Virtual Machines was almost a flat speed curve.

The suspend action of all Virtual Machines started fast (when all machines were suspending) and finished slower (when only the largest virtual machines were still suspending)

–jeroen

Posted in ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

“fixing” ESXi “rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]”

Posted by jpluimers on 2021/08/30

Reminder to self: create a static ESXi binary for a recent rsync release.

Quite a few people have bumped into rsync erroring out with “large” sets of files (where large can be as low as ~1000), like for instance Tj commenting on my post “ESXi 5.1 and rsync – damiendebin.net.”:

ERROR: out of memory in receive_sums [sender] │······
rsync error: error allocating core memory buffers (code 22) at util2.c(102) [sender=3.1.1] │······
rsync: [generator] write error: Broken pipe (32) │······

I bumped into this myself as well, even when updating from rsync 3.1.0 to 3.1.2.

There are various static rsync for ESXi around. Just a few of them for completeness:

There is also 3.0.9 (via [Wayback] VMware ESXi 5.1 rsync 3.0.9 statically linked binary erstellen – bachmann-lan.de), but it has a VMFS bug ([Wayback] 8177 – Problems with big sparsed files) as per [Wayback] ESXi 5.1 and rsync – damiendebin.net.)

The good news is that it is fixed in 3.2.2 as a user-configurable setting, but since there is no ESXi build yet (see reminder above)…

Anyway: [Wayback] 12769 – error allocating core memory buffers (code 22) depending on source file system

Wayne Davison 2020-06-26 03:56:35 UTC
I fixed the allocation args to be size_t values (and improved a bunch of allocation error checking while I was at it).

I then added an option that lets you override this allocation sanity-check value. The default is still 1G per allocation, but you can now specify a much larger value (up to "--max-alloc=8192P-1").

If you want to make a larger value the default for your copies, export RSYNC_MAX_ALLOC in the environment with the size value of your choice.

Committed for release in 3.2.2.

This is what happens with 3.1.2 and 3.1.3:

time rsync -aiv --info=progress2 --progress --partial --existing --inplace /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
sending incremental file list
              0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1000/1259)
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]
Command exited with non-zero status 22
real    0m 0.87s
user    0m 0.10s
sys 0m 0.00s
time rsync -aiv --info=progress2 --progress --partial --ignore-existing --sparse /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
sending incremental file list
              0   0%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1000/1259)
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]
Command exited with non-zero status 22
real    0m 0.28s
user    0m 0.12s
sys 0m 0.00s
Finished

I was lucky that [Wayback] “rsync error: error allocating core memory buffers” protocol version “3.1.2” – Google Search got me a result so quickly: add a --protocol-29 and you are set.

The first result (Wayback has the results reversed from what got) didn’t fix it. The second did.

  1. [Wayback] 225761 – net/rsync long path causes buffer overflow (update to 3.1.3)
  2. [Wayback/Archive.is] AIX Open Source – IBM Power Systems Community: rsync out of memory

    As a work around, I added “--protocol=29” to one of our servers that was consistently failing with “ERROR: out of memory in flist_expand [receiver]” “rsync error: error allocating core memory buffers (code 22) at util2.c(105) [receiver=3.1.3]” in rsync-3.1.3-2.ppc

    I read the man page and started experimenting with the protocol version until I lowered it enough to get it to work consistently.

The problem might be that running on the ESXi gives you limited memory, but then some 10k files should not use more than like half a megabyte of memory.

Sometime I will dig deeper into the protocol version differences, for now a list of files I think will be relevant for that (mainly look for protocol_version):

Some web pages mentioning the --protocol option and might give me more insight in the protocol differences:

With --protocol=29, time estimation is way off, but there are no errors:

time rsync -aiv --info=progress2 --progress --partial --existing --inplace --protocol=29 /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
building file list ... 
9059 files to consider
.d..t...... isos/
         27,593   0%    0.00kB/s    0:00:06 (xfr#1, to-chk=0/9059)   

sent 212,594 bytes  received 268 bytes  20,272.57 bytes/sec
total size is 3,055,677,645,398  speedup is 14,355,204.99
real    0m 13.31s
user    0m 1.35s
sys 0m 0.00s

time /vmfs/volumes/5791a3e1-0b9368de-4965-0cc47aaa9742/local-bin/rsync -aiv --info=progress2 --progress --partial --ignore-existing --sparse --protocol=29 /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
building file list ... 
9059 files to consider
>f+++++++++ isos/EN-Windows-XP-SP3-VL.iso
...
cd+++++++++ ESXi65.filesystem-root/usr/share/
216,868,164,639   7%   40.64MB/s    1:24:48 (xfr#2571, to-chk=0/9059)   

sent 216,894,938,870 bytes  received 57,858 bytes  42,582,702.80 bytes/sec
total size is 3,055,677,645,398  speedup is 14.09
real    1h 24m 58s
user    34m 5.59s
sys 0m 0.00s
Finished

Even not on ESXi, there were just a few people bumping into this, so I wonder why there are so few matches on [Wayback] “ERROR: out of memory in flist_expand [sender]” “sender=3.1” – Google Search:

–jeroen

Posted in *nix, *nix-tools, ash/dash development, Development, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, rsync, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Windows on VMware ESXi: fixing “Unknown USB Device” or “The USB device returned an invalid USB device descriptor”

Posted by jpluimers on 2021/08/27

Sometimes, the Windows device manager shows any of the below errots for an USB 3.x device that is physically connected to the ESXi host and passes through a VM (via a virtual USB controller and virtual USB device).

This solution usually works: use a USB 2.0 cable instead of a USB 3.x cable.

If it doesn’t work, then the solutions are the same as when this happens in a pure physical solution:

In my case, I needed a USB connection, because it is the only way to reset a Fujitsu ScanSnap scanner to correct WiFi settings after it gets confused (which happens a few times per year).

Sreenshots

The difference for devices with a known driver:

  • USB 3 cable 
  • USB 2 cable 

The difference for devices with an unknown driver:

  •  USB 3 cable 
  • USB 2 cable 

Related information

This happened with me on both a ScanSnap iX500 and iX1500, and this did not help (as I was on a more recent Windows version): [WayBack] When setting up your scanner 502: My computer does not recognize ScanSnap iX500 with a USB 3.0 port. – Fujitsu Global.

More on USB passthrough:

More on USB virtualisation:

–jeroen

Posted in ESXi5.5, ESXi6, ESXi6.5, Fujitsu ScanSnap, Hardware, ix500, Power User, Scanners, Virtualization, VMware, VMware ESXi | Leave a Comment »

« Previous Entries