The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,862 other subscribers

Archive for the ‘VMware’ Category

« Previous Entries
Next Entries »

Alternatives to VMware ESXi: working around “[Errno 28] No space left on device” when updating (especially when booting from USB-stick)

Posted by jpluimers on 2021/05/07

Yesterday I talked about VMware ESXi: working around “[Errno 28] No space left on device” when updating (especially when booting from USB-stick).

There are some alternative workarounds mentioned on the interwebz. Below are a the ones I found. I discuss which ones won’t work, and why I dislike others.

Alternative workarounds that failed

Configuring host-swap

This was suggested by:

Host swap was already configured, and it still failed.

Just in case you ever want to configure host swap, it is under an URL like https://esxi67.example.org/ui/#/host/manage/system/swap and looks like this:

ESXi 6.7: configuring host swap

ESXi 6.7: configuring host swap

You get there by:

  1. logging on to the web UI
  2. clicking Host
  3. clicking Manage under Host
  4. clicking Swap under the System tab
  5. clicking Edit settings when you want to change them
    ESXi 6.7: edit host swap settings

    ESXi 6.7: edit host swap settings

More information about host swap:

Alternative workarounds I like less

Below are a few alternative workarounds. I will include them as they gained me more knowledge, but I will also describe why I like them less.

  • [Wayback] ESXI 6.7 update: No space left on device | eknori.de after explaining that directing the swap space to a datastore fails, also mentions alternative this:

    Unfortunately, in this situation, host swap already was enabled.

    There is though, a workaround. You can use an image that doesn’t have the tools vib included with this command:

    esxcli software profile update -p ESXi-6.7.0-20190802001-no-tools -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

    You can then manually install the troublesome vib (if you have a need for tools) with this command:

    esxcli software vib install -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_10.3.10.12406962-14141615.vib

    I had to edit it as the post itself shows the filename as

    vmw-depot
 -index.xml

    Yup: bitching again, as markup issues make code unreliable. It also allows me to explain why I do not like the solution, which is because of two reasons:

    1. It doesn’t explain why this solution works and if it is future proof. Does a future upgrade that includes changed VMware_locker_tools-light also fail? If it does not fail, does it update the VMware_locker_tools-light?
    2. It does not explain how to get the path of https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_10.3.10.12406962-14141615.vib.  I did some mor research on this, and it is actually pretty straightforward: the [Wayback] VMware ESXi 6.7 Patch History has it in the table
      ESXi-6.7.0-20210304001-standard patch table

      ESXi-6.7.0-20210304001-standard patch table

      The “Version” link for “tools-light” [Wayback] 11.2.5.17337674-17700514 actually links to the VMware_locker_tools-light_11.1.1.16303738-16701467.vib file.

  • [Wayback] ESXi 6.7.0 – [Errno 28] No space left on devicevibs = VMware_locker_tools-light_11.1.1.16303738-16701467 does not explain where to get the VMware_locker_tools-light_11.1.1.16303738-16701467.vib link from, does not have the code formatted as such (so I did that below), but does actually answers part of the above questions, but not if a future upgrade will also fail. In short: re-running the upgrade after manually installing the VMware_locker_tools-light_11.1.1.16303738-16701467.vib will succeed:

    Unfortunately swap was already enabled to I had to manually install the tools-light with this command:

    esxcli software vib install -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_11.1.1.16303738-16701467.vib

    Then re-ran the upgrade and it was successful.

  • [Wayback] ESXi 6 Update error – No Space left on device /locker which suggests to delete “find all big files in /locker and remove it”. I think that is a bad idea, as the /locker directory is maintained by your ESXi system and you should not remove any big file without knowing if it is relied upon by ESXi.
  • While updating VMware ESXi servers, VMware vSphere users may encounter the “No space left on device” error that pops up while executing “esxcli software vib update” command. Interestingly, the problem occurs even though disks are doing well and have enough free space and df -h command proves that.[Wayback] No Space Left on Device? Updating VMware ESXi | StarWind Blog has bad code markup, but explains
    • how to get disk usage with df -h where the vfat volumes usually indicate the ones on USB or SD-card media.
    • that hardly the number of inodes is a problem, and that stat -f / can help you figure out if that is the case on the volume where the upgrade files are stored
    • how to find large files not in data stores; I have changed added -h to the ls command so it becomes human readable:
      find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -lh '{}' \;
    • suggests how to put the swap space on a data store (which doesn’t work on ESXi 6.7 systems any more)
  •  

    [Wayback] Intel NUC Kit NUC5i3RYH met ESXi 6.0 updaten naar 6.7 – Gahan Zwart’s Blog

    • I like the upgrade copying the ISO to an USB stick with Rufus
    • I do not like the VMware_locker_tools-light... intermediate step, as the last step (download the full depot to a datastore, then update from there)
    • ESXi 7.0 has the same ErrNo 28 update problem as ESXi 6.7 and 6.5, so I will default to the depot download.

–jeroen

Posted in ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

VMware ESXi: working around “[Errno 28] No space left on device” when updating (especially when booting from USB-stick)

Posted by jpluimers on 2021/05/06

A while ago, I had this error while upgrading an ESXi 6.7 system:

# esxcli system maintenanceMode get
Enabled
# esxcli software profile update -p ESXi-6.7.0-20210304001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-
index.xml
 [InstallationError]
 [Errno 28] No space left on device
       vibs = VMware_locker_tools-light_11.2.5.17337674-17700514
 Please refer to the log file for more details.

This seems to be kind of a known issue, especially on systems that boot from USB sticks (see [Archive.is] “ESXi” “[Errno 28] No space left on device” – Google Search). Despites these sticks having gotten larger over time, VMware uses very little storage on them, so runs out of storage spice while processing installs from the depot.

This is kind of disappointing, and with the ever growing updates, the only workaround is to download an offline bundle, then install it.

Below I will explain how to get the offline bundle, as it involves a few more steps all aimed at this simple installation of the offline bundle file:

# esxcli system maintenanceMode get
Enabled
# esxcli software profile update -p ESXi-6.7.0-20210304001-standard -d /vmfs/volumes/EVO860_500GB/OfflineBundles/ESXi670-202103001.zip 
Update Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: VMW_bootbank_ixgben_1.7.1.16-2vmw.670.3.104.16075168, VMW_bootbank_lpfc_11.4.33.26-14vmw.670.3.104.16075168, VMW_bootbank_net-e1000_8.0.3.1-5vmw.670.3.112.16701467, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.670.3.104.16075168, VMW_bootbank_nfnic_4.0.0.44-0vmw.670.3.104.16075168, VMW_bootbank_ntg3_4.1.5.0-0vmw.670.3.116.16713306, VMW_bootbank_nvme_1.2.2.28-4vmw.670.3.132.17167734, VMW_bootbank_qfle3f_1.0.25.0.2-15vmw.670.3.104.16075168, VMW_bootbank_sfvmk_1.0.0.1003-7vmw.670.3.104.16075168, VMW_bootbank_vmkusb_0.1-1vmw.670.3.143.17700523, VMW_bootbank_vmw-ahci_2.0.7-2vmw.670.3.143.17700523, VMware_bootbank_cpu-microcode_6.7.0-3.139.17700514, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.2-3.89.15160138, VMware_bootbank_esx-base_6.7.0-3.143.17700523, VMware_bootbank_esx-ui_1.33.7-15803439, VMware_bootbank_esx-update_6.7.0-3.143.17700523, VMware_bootbank_native-misc-drivers_6.7.0-3.89.15160138, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.36-2vmw.670.3.116.16713306, VMware_bootbank_vsan_6.7.0-3.143.17661912, VMware_bootbank_vsanhealth_6.7.0-3.143.17665851, VMware_locker_tools-light_11.2.5.17337674-17700514
   VIBs Removed: VMW_bootbank_ixgben_1.7.1.16-1vmw.670.3.73.14320388, VMW_bootbank_lpfc_11.4.33.25-14vmw.670.3.73.14320388, VMW_bootbank_net-e1000_8.0.3.1-5vmw.670.0.0.8169922, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.670.2.48.13006603, VMW_bootbank_nfnic_4.0.0.29-0vmw.670.3.73.14320388, VMW_bootbank_ntg3_4.1.3.2-1vmw.670.1.28.10302608, VMW_bootbank_nvme_1.2.2.28-1vmw.670.3.73.14320388, VMW_bootbank_qfle3f_1.0.25.0.2-14vmw.670.0.0.8169922, VMW_bootbank_sfvmk_1.0.0.1003-6vmw.670.3.73.14320388, VMW_bootbank_vmkusb_0.1-1vmw.670.2.48.13006603, VMW_bootbank_vmw-ahci_1.2.8-1vmw.670.3.73.14320388, VMware_bootbank_cpu-microcode_6.7.0-2.69.14141615, VMware_bootbank_elx-esx-libelxima.so_11.4.1184.1-2.48.13006603, VMware_bootbank_esx-base_6.7.0-3.73.14320388, VMware_bootbank_esx-ui_1.33.4-14093553, VMware_bootbank_esx-update_6.7.0-3.73.14320388, VMware_bootbank_native-misc-drivers_6.7.0-2.48.13006603, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.36-2.48.13006603, VMware_bootbank_vsan_6.7.0-3.73.14263135, VMware_bootbank_vsanhealth_6.7.0-3.73.14263064, VMware_locker_tools-light_10.3.10.12406962-14141615
   VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.670.0.0.8169922, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.670.0.0.8169922, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-via_0.3.3-2vmw.670.0.0.8169922, VMW_bootbank_block-cciss_3.6.14-10vmw.670.0.0.8169922, VMW_bootbank_bnxtnet_20.6.101.7-24vmw.670.3.73.14320388, VMW_bootbank_bnxtroce_20.6.101.0-20vmw.670.1.28.10302608, VMW_bootbank_brcmfcoe_11.4.1078.25-14vmw.670.3.73.14320388, VMW_bootbank_char-random_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.670.0.0.8169922, VMW_bootbank_elxiscsi_11.4.1174.0-2vmw.670.0.0.8169922, VMW_bootbank_elxnet_11.4.1097.0-5vmw.670.3.73.14320388, VMW_bootbank_hid-hid_1.0-3vmw.670.0.0.8169922, VMW_bootbank_i40en_1.8.1.9-2vmw.670.3.73.14320388, VMW_bootbank_iavmd_1.2.0.1011-2vmw.670.0.0.8169922, VMW_bootbank_igbn_0.1.1.0-5vmw.670.3.73.14320388, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.670.1.28.10302608, VMW_bootbank_ipmi-ipmi-si-drv_39.1-5vmw.670.1.28.10302608, VMW_bootbank_iser_1.0.0.0-1vmw.670.1.28.10302608, VMW_bootbank_lpnic_11.4.59.0-1vmw.670.0.0.8169922, VMW_bootbank_lsi-mr3_7.708.07.00-3vmw.670.3.73.14320388, VMW_bootbank_lsi-msgpt2_20.00.06.00-2vmw.670.3.73.14320388, VMW_bootbank_lsi-msgpt35_09.00.00.00-5vmw.670.3.73.14320388, VMW_bootbank_lsi-msgpt3_17.00.02.00-1vmw.670.3.73.14320388, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.670.0.0.8169922, VMW_bootbank_misc-drivers_6.7.0-2.48.13006603, VMW_bootbank_mtip32xx-native_3.9.8-1vmw.670.1.28.10302608, VMW_bootbank_ne1000_0.8.4-2vmw.670.2.48.13006603, VMW_bootbank_nenic_1.0.29.0-1vmw.670.3.73.14320388, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.670.0.0.8169922, VMW_bootbank_net-bnx2x_1.78.80.v60.12-2vmw.670.0.0.8169922, VMW_bootbank_net-cdc-ether_1.0-3vmw.670.0.0.8169922, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.670.0.0.8169922, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.670.0.0.8169922, VMW_bootbank_net-enic_2.1.2.38-2vmw.670.0.0.8169922, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.670.0.0.8169922, VMW_bootbank_net-forcedeth_0.61-2vmw.670.0.0.8169922, VMW_bootbank_net-igb_5.0.5.1.1-5vmw.670.0.0.8169922, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.670.0.0.8169922, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.670.0.0.8169922, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.670.0.0.8169922, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.670.0.0.8169922, VMW_bootbank_net-nx-nic_5.0.621-5vmw.670.0.0.8169922, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.670.0.0.8169922, VMW_bootbank_net-usbnet_1.0-3vmw.670.0.0.8169922, VMW_bootbank_nhpsa_2.0.22-3vmw.670.1.28.10302608, VMW_bootbank_nmlx4-core_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx4-en_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx4-rdma_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx5-core_4.17.13.1-1vmw.670.3.73.14320388, VMW_bootbank_nmlx5-rdma_4.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nvmxnet3-ens_2.0.0.21-1vmw.670.0.0.8169922, VMW_bootbank_nvmxnet3_2.0.0.29-1vmw.670.1.28.10302608, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_pvscsi_0.1-2vmw.670.0.0.8169922, VMW_bootbank_qcnic_1.0.2.0.4-1vmw.670.0.0.8169922, VMW_bootbank_qedentv_2.0.6.4-10vmw.670.1.28.10302608, VMW_bootbank_qfle3_1.0.50.11-9vmw.670.0.0.8169922, VMW_bootbank_qfle3i_1.0.2.3.9-3vmw.670.0.0.8169922, VMW_bootbank_qflge_1.1.0.11-1vmw.670.0.0.8169922, VMW_bootbank_sata-ahci_3.0-26vmw.670.0.0.8169922, VMW_bootbank_sata-ata-piix_2.12-10vmw.670.0.0.8169922, VMW_bootbank_sata-sata-nv_3.5-4vmw.670.0.0.8169922, VMW_bootbank_sata-sata-promise_2.12-3vmw.670.0.0.8169922, VMW_bootbank_sata-sata-sil24_1.1-1vmw.670.0.0.8169922, VMW_bootbank_sata-sata-sil_2.3-4vmw.670.0.0.8169922, VMW_bootbank_sata-sata-svw_2.3-3vmw.670.0.0.8169922, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.670.0.0.8169922, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.670.0.0.8169922, VMW_bootbank_scsi-aic79xx_3.1-6vmw.670.0.0.8169922, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.670.0.0.8169922, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.670.0.0.8169922, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.670.0.0.8169922, VMW_bootbank_scsi-hpsa_6.0.0.84-3vmw.670.0.0.8169922, VMW_bootbank_scsi-ips_7.12.05-4vmw.670.0.0.8169922, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.670.0.0.8169922, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.670.0.0.8169922, VMW_bootbank_scsi-mpt2sas_19.00.00.00-2vmw.670.0.0.8169922, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.670.0.0.8169922, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libata-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libata-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfc-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfc-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-3-0_6.7.0-0.0.8169922, VMW_bootbank_smartpqi_1.0.1.553-28vmw.670.3.73.14320388, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.670.0.0.8169922, VMW_bootbank_usbcore-usb_1.0-3vmw.670.0.0.8169922, VMW_bootbank_vmkata_0.1-1vmw.670.0.0.8169922, VMW_bootbank_vmkfcoe_1.0.0.1-1vmw.670.1.28.10302608, VMW_bootbank_vmkplexer-vmkplexer_6.7.0-0.0.8169922, VMW_bootbank_xhci-xhci_1.0-3vmw.670.0.0.8169922, VMware_bootbank_esx-dvfilter-generic-fastpath_6.7.0-0.0.8169922, VMware_bootbank_esx-xserver_6.7.0-3.73.14320388, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-16vmw.670.1.28.10302608, VMware_bootbank_lsu-intel-vmd-plugin_1.0.0-2vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-drivers-plugin_1.0.0-1vmw.670.2.48.13006603, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-13vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-9vmw.670.2.48.13006603, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-9vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-7vmw.670.0.0.8169922, VMware_bootbank_lsu-smartpqi-plugin_1.0.0-3vmw.670.1.28.10302608, VMware_bootbank_qlnativefc_3.1.8.0-5vmw.670.3.73.14320388, VMware_bootbank_rste_2.0.2.0088-7vmw.670.0.0.8169922
# reboot

Note that I always perform a  esxcli system maintenanceMode get in order to verify the system is indeed in maintenance mode.

Steps to get the offline bundle

0. determining the profile

Actually there is a pre-step: checking which update profile is needed. I use the VMware Patch Tracker from VFrontDe, as you can subscribe through RSS (by ESXi version) Twitter (all versions) for that.

For ESXi 7, I use moved from the [Wayback] regular ESXi 6.7 patch RSS feed to the Feedly ESXi 6.7 patch RSS feed (which integrates in my daily news reading) of [Wayback] VMware ESXi 6.7 Patch History:

Keep track of VMware ESXi patches, subscribe by [Archive.is] RSS and [Archive.is] Twitter! – Brought to you by @VFrontDe

For the current (as of writing) update, it shows two out of four profiles:

  1. Imageprofile ESXi-6.7.0-20210304001-standard (Build 17700523) … (For more information see [Wayback] Release Notes.)
  2. Imageprofile ESXi-6.7.0-20210301001s-standard (Build 17700514) … (For more information see [Wayback] Release Notes.)

There can be two more “...-no-tools” profiles as described by [Wayback] VMware ESXi Image Profiles | virten.net

Example – ESXi 5.5 Patch 2 contains 4 Image Profiles:
  1. ESXi-5.5.0-20140704001-standard – Contains all patches
  2. ESXi-5.5.0-20140704001-no-tools – Contains all patches but no VMware Tools
  3. ESXi-5.5.0-20140701001s-standard – Contains security patches only
  4. ESXi-5.5.0-20140701001s-no-tools – Contains security patches only and no VMware Tools

…

vSphere ESXi 6.7 Image Profiles
Image Profile ESX Build Release Date
ESXi-6.7.0-20210304001-standard 17700523 2021-03-18
ESXi-6.7.0-20210304001-no-tools 17700523 2021-03-18
ESXi-6.7.0-20210301001s-standard 17700514 2021-03-18
ESXi-6.7.0-20210301001s-no-tools 17700514 2021-03-18

Given that I my environment as complete as possible (the original ESXi install contains everything, so I do want to include the “VMware Tools“, and update the full software, nut just security issues), I opt for “###...-standard” and not for “###s...-standard“.

1. determine the download

The page [Wayback] VMware ESXi 6.7 Patch History (and similar pages for other ESXi versions) contains a tiny link in the upper left hand side named “[MyVMware Patch Download]” (I saved it in the Wayback machine) which brings you a patch search page on the VMware patch site.

The big drawback is that you need to be signed on into yout VMware account to actually see this page. The VMware account validation is implemented in a “corporate” way, so often does not work (especially when you need it, which for me often is on weekends, a period in the week that corporates seem to ar less monitor their systems than during USA work-day hours). I might sound cynical, but this set me back quite some time even trying to get an ESXi 7 license.

Here are some screenshots:

VMware HomeP/roduct Patches

VMware Home/Product Patches

The site is so corporate, that when your screen is not wide enough, the selection box is drawn outside your screen area:

VMware being very corporate: drawing selection box outside the screen area.

VMware being very corporate: drawing selection box outside the screen area.

Anyway the one ending in “and Installable” is the one we want, cutting off the other part (you are a corporate or you aren’t, right?):

"ESXi (Embedded and ..." is cut off as the layout has been designed to cut off either way.

“ESXi (Embedded and …” is cut off as the layout has been designed to cut off either way.

So for ESXi 6.7.0, when writing this, the offline bundles were these:

ESXi 6.7.0 offline bundles

ESXi 6.7.0 offline bundles (unlike the YYYYMM### release name and YYYYMM### bulletin numbers, the dates are in MM/DD/YYYY format)

Release Name Release Date Build Number Bulletin Number

ESXi670-202103001

Product:ESXi (Embedded and Installable) 6.7.0

Download Size:476.7 MB

03/18/2021

17700523

 ESXi670-202103102-SG

ESXi670-202103103-SG

ESXi670-202103001

ESXi670-202103101-SG

ESXi670-202103401-BG

ESXi670-202103403-BG

ESXi670-202103402-BG

Download

So, for imageprofile ESXi-6.7.0-20210304001-standard (Build 17700523), the release name is ESXi670-202103001 (as it matches Build Number [Wayback] 17700523).

From the above, we now have these links and numbers:

The two common parts above are:

  1. ESXi670-202103001 (both as identifier and as filename ESXi670-202103001.zip)
  2. https://docs.vmware.com/en/VMware-vSphere/6.7/rn/esxi670-202103001.html which contains a table like this that has both a (deprecated!) md5sum and sha1sum value:
    Download Filename: ESXi670-202103001.zip
    Build: 17700523
    Download Size: 476.7 MB
    md5sum: 26f177706bce4a0432d9e8af016a5bad
    sha1checksum: 0263a25351d003d34321a062b391d9fe5c312bd2
    Host Reboot Required: Yes
    Virtual Machine Migration or Shutdown Required: Yes

 

Back to the download button, where you can validate the filename of the off-line bundle:

The Download button points to a complex URL with these parts:

  • scheme:[//authority]path = https://download2.vmware.com/patch/software/VUM/OFFLINE/release-767-20210316-739878/ESXi670-202103001.zip
  • [?query] is of the form ?HashKey=<32-hexadecimal-digits>&params=<URI-escaped-JSON-string>&AuthKey=<10-decimal-digits>_<32-hexadecimal-digits>
  • Sample values included in params:
    • "custnumber":"ABCDEFGHIJKLMNOP==" 16-digit == padded base64 encoded customer identification
    • "sourcefilesize":"476.7+MB"
    • "dlgcode":"ESXi670-202103001"
    • "languagecode":"en"
    • "source":"PATCH"
    • "downloadtype":"manual"
    • "downloaduuid":"12345678-490a-abcd-efgh12345678" (version 4 variant 1 uuid)
    • "productname":"ESXi+(Embedded+and+Installable)"
    • "productversion":"6.7.0"

2. downloading the offline bundle file and verifying it

The steps taken until now:

  1. Find the most recent imageprofile name at [Wayback] VMware ESXi 6.7 Patch History (in my case ESXi-6.7.0-20210304001-standard)
  2. Find the “Release notes” link right under the table below it (in my case https://docs.vmware.com/en/VMware-vSphere/6.7/rn/esxi670-202103001.html#esxi-6-7-0-20210301001s-standard-resolved)
  3. Browse to the top of that page for the table with the offline bundle identifier and filename

Now

  1. search for the filename (like [Wayback/Archive.is] “ESXi670-202103001.zip” – Google Search) to download, or get the download URL from the VMware patch site (note this URL expires usually within 20 minutes).
  2. download the file using that URL:
    # wget "download-URL" -O /vmfs/volumes/EVO860_500GB/OfflineBundles/ESXi670-202103001.zip
Connecting to download.example.org (192.0.2.0:443)
ESXi670-202103001.zi 100% |****************************************************************************************************************|  454M  0:00:00 ETA

    Yes, that strange IP address is for documentation purposes as you can see in [Wayback] RFC 5737 – IPv4 Address Blocks Reserved for Documentation and Reserved IP addresses – Wikipedia.

  3. verify the download with the above sha1sum (which is included in the ESXi BusyBox shell)
    # sha1sum /vmfs/volumes/EVO860_500GB/OfflineBundles/ESXi670-202103001.zip 
0263a25351d003d34321a062b391d9fe5c312bd2  /vmfs/volumes/EVO860_500GB/OfflineBundles/ESXi670-202103001.zip
  4. shutdown all the VM’s, put the machine in maintenance mode, upgrade, verify result, reboot, put the machine out of maintenance mode, start any VM’s that should be up

[Archive.is] [OSError] [Errno 28] No space left on device even … – VMware Technology Network VMTN very importantly mentions to do a dry-run before actually upgrading:

esxcli software profile update --dry-run -p ESXi-6.7.0-20201103001-standard -d /vmfs/volumes/datastore1/update/ESXi670-202011001.zip

If the dry run is successful, do the update with

esxcli software profile update -p ESXi-6.7.0-20201103001-standard -d /vmfs/volumes/datastore1/update/ESXi670-202011001.zip

—jeroen

Posted in ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: where are my log files actually stored? Actually, most of them are in `/scratch/log` which points to a hidden `.locker` directory in a datastore.

Posted by jpluimers on 2021/05/05

A summary of the full gist at [Wayback/Archive.is] ESXi-where-are-my-log-files-stored.txt:

# ls -al / /var/ /var/log/ /var/run/ /scratch/ /scratch/log/ | grep "/\|log\|-\>"
/:
...
lrwxrwxrwx    1 root     root            57 Apr  4 18:16 scratch -> /vmfs/volumes/5ce2d440-72311161-75c5-0025907d9d5c/.locker
...
/scratch/:
drwxr-xr-x    1 root     root        106496 Apr 10 08:40 log
/scratch/log/:
-rw-------    1 root     root           411 Apr  4 18:20 Xorg.log
...
-rw-------    1 root     root         78835 Apr  4 10:30 syslog.0.gz
-rw-------    1 root     root         61136 Mar 18 15:05 syslog.1.gz
-rw-------    1 root     root         60589 Feb 24 00:30 syslog.2.gz
-rw-------    1 root     root         60373 Feb  1 08:01 syslog.3.gz
-rw-------    1 root     root         60203 Jan  9 15:50 syslog.4.gz
-rw-------    1 root     root         59889 Dec 17 23:20 syslog.5.gz
-rw-------    1 root     root         60398 Nov 25 06:50 syslog.6.gz
-rw-------    1 root     root         60563 Nov  2 14:25 syslog.7.gz
-rw-------    1 root     root        531794 Apr 10 09:35 syslog.log
...
-rw-------    1 root     root        157255 Apr  4 18:17 vvold.log
/var/:
...
drwxr-xr-x    1 root     root           512 Apr  5 19:19 log
...
/var/log/:
-rw-r--r--    1 root     root           416 Apr  4 18:16 .vmsyslogd.err
...
-rw-r--r--    1 root     root         38069 Apr  4 18:20 configRP.log
-rw-r--r--    1 root     root             0 Apr  4 18:16 cryptoloader.log
...
-rw-r--r--    1 root     root            87 Apr  5 21:57 esxcli.log
...
-rw-------    1 root     root          3350 Apr  4 18:16 init.log
-rw-r--r--    1 root     root           966 Apr  4 18:16 iofilter-init.log
...
-rw-r--r--    1 root     root         21769 Apr  4 18:16 jumpstart-esxcli-stdout.log
-rw-r--r--    1 root     root         18857 Apr  4 18:16 jumpstart-native-stdout.log
-rw-r--r--    1 root     root         10837 Apr  4 18:16 jumpstart-stdout.log
-rw-r--r--    1 root     root             0 Apr  4 18:16 kickstart.log
...
-rw-------    1 root     root         10916 Apr  4 18:16 sysboot.log
...
-rw-------    1 root     root            64 Apr 10 09:13 tallylog
...
/var/run/:
lrwxrwxrwx    1 root     root            12 Apr  4 18:16 log -> /scratch/log
  • Almost all log files (most from /var/log and all from /var/run/log) are actually persistently stored in /scratch/log and survive reboots. Just a few are non-persistent.
  • /var/log/syslog is being archived as .gz files (compressed by gzip).
  • syslog is special: the location can be configured, and even be external: [Wayback] Configuring syslog on ESXi (2003322)

    VMware vSphere ESXi 5.0 and higher hosts run a Syslog service (vmsyslogd) that provides a standard mechanism for logging messages from the VMkernel and other system components. By default in ESXi, these logs are placed on a local scratch volume or a ramdisk. To preserve the logs further, ESXi can be configured to place these logs to an alternate storage location on disk and to send the logs across the network to a Syslog server.

  • A summary of some of the above log files is at [Wayback] ESXi Log File Locations

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, gzip, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Running ArchiveTeam Warrior version 3.2 on ESXi

Posted by jpluimers on 2021/05/05

A while ago I wrote about Helping the WayBack ArchiveTeam team: running their Warrior virtual appliance on ESXi.

Since it was scheduled before my cancer treatment started and got posted when still recovering from it, I missed that version 3.2 of the [Wayback] ArchiveTeam Warrior appliance appeared in the [Wayback] Releases · ArchiveTeam/Ubuntu-Warrior at [Wayback] Release v3.2 · ArchiveTeam/Ubuntu-Warrior. You can download it form these places:

These two sites have not yet been updated, so they contain the older versions:

The source code now has been moved three times:

Read the rest of this entry »

Posted in *nix, *nix-tools, ArchiveTeamWarrior, Cloud, Containers, diff, Docker, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Infrastructure, Internet, InternetArchive, Kubernetes (k8n), KVM Kernel-based Virtual Machine, patch, Power User, VirtualBox, Virtualization, VMware, VMware ESXi, VMware Workstation, WayBack machine | Leave a Comment »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 5

Posted by jpluimers on 2021/04/30

Yesterday’s post got a bit longer than anticipated as there were most steps than I hoped for to create the listing script vim-cmd-list-all-VMs.sh:

#!/bin/sh
# https://wiert.me/2021/04/29/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-4/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
done

This means that today there is a new try to cover the rest of these lists I started yesterday with:

Available commands

  • vim-cmd vmsvc/power.getstate vmid
  • vim-cmd vmsvc/power.hibernate vmid
  • vim-cmd vmsvc/power.off vmid
  • vim-cmd vmsvc/power.on vmid
  • vim-cmd vmsvc/power.reboot vmid
  • vim-cmd vmsvc/power.reset vmid
  • vim-cmd vmsvc/power.shutdown vmid
  • vim-cmd vmsvc/power.suspend vmid
  • vim-cmd vmsvc/power.suspendResume vmid

Unavailable commands

  • vim-cmd vmsvc/power.startup vmid
  • vim-cmd vmsvc/power.resume vmid
  • vim-cmd vmsvc/power.wakeup vmid

So here we go:

vim-cmd-hibernate-running-VMs.sh and vim-cmd-suspend-running-VMs.sh

These files are almost the same: the echo and command are different (Hibernating versus Suspending and power.hibernate versus suspend)

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on  VM with id ${vmid} and name: $name"
        echo "Hibernating VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.hibernate ${vmid}
    fi
done

This hibernates all VMs with power state Powered on and waits for each VM to complete hibernation.

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on VM with id ${vmid} and name: $name"
        echo "Suspending VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.suspend ${vmid}
    fi
done

This suspends all VMs with power state Powered on and waits for each VM to complete suspending.

vim-cmd-resume-suspended-VMs.sh and vim-cmd-wakeup-suspended-VMs.sh

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Suspended" ] ; then
        echo "Suspended VM with id ${vmid} and name: $name"
        echo "Resuming  VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.on ${vmid}
    fi
done

This resumes (as there is no vim-cmd vmsvc/power.resume vmid) all VMs with power state Suspended and waits for each VM to complete resume.

vim-cmd-power-off-powered-on-VMs.sh 

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on   VM with id ${vmid} and name: $name"
        echo "Powering off VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.off ${vmid}
    fi
done

This powers off all VMs with power state Power on and waits for each VM to start powering off (but does not wait for them to complete powering off, so parts run in parallel).

vim-cmd-power-on-powered-off-VMs.sh, vim-cmd-power-on-shutdown-VMs.sh and vim-cmd-startup-shutdown-VMs.sh

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered off" ] ; then
        echo "Powered off VM with id ${vmid} and name: $name"
        echo "Powering on VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.on ${vmid}
    fi
done

These powers on all VMs with power state Powered off and waits for each VM to complete power on (but does not wait for them to complete boot, so part runs in parallel!).

These are exactly the same, as you cannot distinguish a VM that has been shutdown by vim-cmd vmsvc/power.off vmid from vim-cmd vmsvc/shutdown vmid.

vim-cmd-reboot-powered-on-VMs.sh 

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on VM with id ${vmid} and name: $name"
        echo "Rebooting  VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.reboot ${vmid}
    fi
done

This reboots all VMs with power state Power on and waits for each VM to start rebooting (but does not wait for them to complete reboot, so parts run in parallel).

vim-cmd-reset-powered-on-VMs.sh 

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on VM with id ${vmid} and name: $name"
        echo "Resetting  VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.reset ${vmid}
    fi
done

This resets all VMs with power state Power on and waits for each VM to start rebooting (but does not wait for them to complete reboot, so parts run in parallel).

vim-cmd-suspendResume-powered-on-VMs.sh 

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on      VM with id ${vmid} and name: $name"
        echo "SuspendResuming VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.suspendResume ${vmid}
    fi
done

This suspends and resumes on all VMs with power state Power on and waits for each VM to start this shirt cycle (but does not wait for them to complete the suspend and resume cycle, so parts run in parallel).

Future episodes

For now, this is OK enough for me, but I might write a new installment trying to run more of these in parallel (as parts already are done on the ESXi side).

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 4

Posted by jpluimers on 2021/04/29

Yesterday we ended with an overview of available and unavailable vim-cmd vmsvc commands and the promise to try running the various power commands on all relevant VMs.

Let’s start with a summary of the commands, so it will be easier to make a list of scripts to run them on relevant VMs.

Available commands

  • vim-cmd vmsvc/power.getstate vmid
  • vim-cmd vmsvc/power.hibernate vmid
  • vim-cmd vmsvc/power.off vmid
  • vim-cmd vmsvc/power.on vmid
  • vim-cmd vmsvc/power.reboot vmid
  • vim-cmd vmsvc/power.reset vmid
  • vim-cmd vmsvc/power.shutdown vmid
  • vim-cmd vmsvc/power.suspend vmid
  • vim-cmd vmsvc/power.suspendResume vmid

Unavailable commands

  • vim-cmd vmsvc/power.startup vmid
  • vim-cmd vmsvc/power.resume vmid
  • vim-cmd vmsvc/power.wakeup vmid

List the vmid values, power status and name of all VMs

Getting the vmid

Yesterday I showed a small statement that gives the list of vmid values on an ESXi system:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)s+((S.+S)?)s+([S+])s+(.+.vmx)s+(S+)s+(vmx-[[:digit:]]+)s*?((S.+)?)$/1/p"

What I ideally want is not just the vmid and name for each VM from vim-cmd vmsvc/getallvms, but also get the power state information from vim-cmd vmsvc/power.getstate vmid.

For that, we need to parse the output of vim-cmd vmsvc/power.getstate vmid, which can be three outputs:

  • Retrieved runtime info
    Powered off
  • Retrieved runtime info
    Powered on
  • Retrieved runtime info
    Suspended

So basically it involves deleting the first line which was covered in part 2 of this installment, for example on my system:

# vim-cmd vmsvc/power.getstate 10 | sed '1d'
Powered on

Getting VM name

Extracting both vmid and name from vim-cmd vmsvc/getallvms at the same time is not easy, heck even impossible, so I decided to go the vim-cmd vmsvc/get.config vmid way.

Getting multiple values out of some output is already very hard in bash, where usually the less difficult way is to use arrays. Since Busybox has an ash shell (see Busybox sh (actually ash derivative dash): checking exit codes), and ash does not do arrays, that route is gone.

To give you an idea how hard this is in bash and how to sort of workaround the lack of array support in ash:

This partial vim-cmd vmsvc/get.config vmid sample output on one of my VMs that shows how to use head -n 31 to get just the first 31 lines of output:

# vim-cmd vmsvc/get.config 10 | head -n 31
Configuration:

(vim.vm.ConfigInfo) {
   changeVersion = "2021-04-07T22:08:30.548274Z", 
   modified = "1970-01-01T00:00:00Z", 
   name = "X9SRI-3F-W10P-EN-MEDIA", 
   guestFullName = "Microsoft Windows 10 (64-bit)", 
   version = "vmx-14", 
   uuid = "564d51ac-f6cf-e40b-b686-2f53a28a4bea", 
   createDate = "2019-05-17T21:37:11.408173Z", 
   instanceUuid = "52403d0e-7ccd-48da-bb21-7e966defccf7", 
   npivNodeWorldWideName = , 
   npivPortWorldWideName = , 
   npivWorldWideNameType = , 
   npivDesiredNodeWwns = , 
   npivDesiredPortWwns = , 
   npivTemporaryDisabled = true, 
   npivOnNonRdmDisks = , 
   locationId = "564d6b18-ecd1-2261-0127-146b3f3bc636", 
   template = false, 
   guestId = "windows9_64Guest", 
   alternateGuestName = "", 
   annotation = "", 
   files = (vim.vm.FileInfo) {
      vmPathName = "[EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx", 
      snapshotDirectory = "[EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA", 
      suspendDirectory = "[EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA", 
      logDirectory = "[EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA", 
      ftMetadataDirectory = 
   }, 
   tools = (vim.vm.ToolsConfigInfo) {

The reason to go the vim-cmd vmsvc/get.config vmid way is that it contains all the configuration info in a kind of JSON format (except the first two lines) and should be relatively easy to parse. Or so at least I hoped.

Basically I am interested in the value of name = "X9SRI-3F-W10P-EN-MEDIA", however, there are multiple name fields in the total configuration:

# vim-cmd vmsvc/get.config 10 | sed -n -E '/name =/p'
   name = "X9SRI-3F-W10P-EN-MEDIA", 
         name = "EVO860_500GB",

So what I really want is the value of name = "X9SRI-3F-W10P-EN-MEDIA", in between the (vim.vm.ConfigInfo) { and files = (vim.vm.FileInfo) { parts.

This can be done using sed as it allows to specify a range using a start and end value using addresses:

  • [Wayback] sed: Addresses in sed

    An address is either a decimal number that counts input lines cumulatively across files, a '$' character that addresses the last line of input, or a context address (which consists of a BRE, as described in Regular Expressions in sed , preceded and followed by a delimiter, usually a slash).

    An editing command with no addresses shall select every pattern space.

    An editing command with one address shall select each pattern space that matches the address.

    An editing command with two addresses shall select the inclusive range from the first pattern space that matches the first address through the next pattern space that matches the second. (If the second address is a number less than or equal to the line number first selected, only one line shall be selected.) Starting at the first line following the selected range, sed shall look again for the first address. Thereafter, the process shall be repeated. Omitting either or both of the address components in the following form produces undefined results:

    [address[,address]]
  • Range Addresses (sed, a stream editor)[Wayback] Range Addresses (sed, a stream editor)

    An address range can be specified by specifying two addresses separated by a comma (,). An address range matches lines starting from where the first address matches, and continues until the second address matches (inclusively):

    $ seq 10 | sed -n '4,6p'
4
5
6

    If the second address is a regexp, then checking for the ending match will start with the line following the line which matched the first address: a range will always span at least two lines (except of course if the input stream ends).

    …

  • [Wayback] Regexp Addresses (sed, a stream editor)

For example (with some characters escaped because of [Wayback] ERE syntax (sed, a stream editor)):

# vim-cmd vmsvc/get.config 10 | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/p'
(vim.vm.ConfigInfo) {
   changeVersion = "2021-04-07T22:08:30.548274Z", 
   modified = "1970-01-01T00:00:00Z", 
   name = "X9SRI-3F-W10P-EN-MEDIA", 
   guestFullName = "Microsoft Windows 10 (64-bit)", 
   version = "vmx-14", 
   uuid = "564d51ac-f6cf-e40b-b686-2f53a28a4bea", 
   createDate = "2019-05-17T21:37:11.408173Z", 
   instanceUuid = "52403d0e-7ccd-48da-bb21-7e966defccf7", 
   npivNodeWorldWideName = , 
   npivPortWorldWideName = , 
   npivWorldWideNameType = , 
   npivDesiredNodeWwns = , 
   npivDesiredPortWwns = , 
   npivTemporaryDisabled = true, 
   npivOnNonRdmDisks = , 
   locationId = "564d6b18-ecd1-2261-0127-146b3f3bc636", 
   template = false, 
   guestId = "windows9_64Guest", 
   alternateGuestName = "", 
   annotation = "", 
   files = (vim.vm.FileInfo) {

With [Wayback] BRE syntax (sed, a stream editor) the filter part would be easier: vim-cmd vmsvc/get.config 10 | sed -n -e '/(vim.vm.ConfigInfo) {/,/files = (vim.vm.FileInfo) {/p', but the print part would be more difficult:

  • # vim-cmd vmsvc/get.config 10 | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/1/p'
X9SRI-3F-W10P-EN-MEDIA

  • # vim-cmd vmsvc/get.config 10 | sed -n -e '/(vim.vm.ConfigInfo) {/,/files = (vim.vm.FileInfo) {/ s/^ +name = "(.*)",.*?/1/p'
X9SRI-3F-W10P-EN-MEDIA

Since I am used to extended regular expressions (ERE) over basica regular expressions (BRE), I prefer the first solution.

So getting the name in a variable now becomes this:

# name=`vim-cmd vmsvc/get.config 10 | sed -n -e '/(vim.vm.ConfigInfo) {/,/files = (vim.vm.FileInfo) {/ s/^ +name = "(.*)",.*?/1/p'`
# echo ${name}
X9SRI-3F-W10P-EN-MEDIA

List the vmid values, power status and name of all VMs

Back to the listing script vim-cmd-list-all-VMs.sh:

#!/bin/sh
# https://wiert.me/2021/04/29/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-4/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
done

As a bonus, next to powerState, the script also figures out vmPathName in a similar way to name.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, head, Power User, Scripting, sed, sed script, Software Development, tee, Virtualization, VMware, VMware ESXi | Leave a Comment »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 3

Posted by jpluimers on 2021/04/28

Yesterday’s installment ended with a list of power related vim-cmd vmsvc commands:

Usage: power.getstate vmid
Usage: power.hibernate vmid
Usage: power.off vmid
Usage: power.on vmid
Usage: power.reboot vmid
Usage: power.reset vmid
Usage: power.shutdown vmid
Usage: power.suspend vmid
Usage: power.suspendResume vmid

Getting vmid values

These all have a vmid parameter, so let’s create a small statement that gives the list of vmid on an ESXi system:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"

On my system the relevant VMs it returns are these:

10
5

The various power commands

The outcome of the various power commands are not as simple as one might think as they often do not just depend on the current power state of the VM, but also on either VMware Tools or open-vm-tools to be running.

Basically these tools do the same, but their origin is different: open-vm-tools is often included as part of Linux distributions; VMware Tools is often installed separately, see:

I have made a table for this hoping it makes reading easier, the explanations are by empirical usage, as the documentation in the help dump (see [Wayback] delimited vim-cmd help for each vmsvc command.txt) does not seem to match behaviour.

Command Explanation
vim-cmd vmsvc/power.getstate vmid Gets the power state of the VM, returning a line Retrieved runtime info followed by lines indicating the power state:

  • Powered off
  • Powered on
  • Suspended
vim-cmd vmsvc/power.hibernate vmid When neither VMware Tools or open-vm-tools to be running, you get an error:

(vim.fault.ToolsUnavailable) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = 
   msg = "Received SOAP response fault from []: standbyGuest
Cannot complete operation because VMware Tools is not running in this virtual machine."
}

When a hibernate/suspend task is already running, you get an error:

(vim.fault.TaskInProgress) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   task = 'vim.Task:haTask-12-vim.VirtualMachine.suspend-1006072588'
   msg = "Received SOAP response fault from []: standbyGuest
Another task is already in progress."
}

Otherwise, depends on the power state of the VM if vmware tools:

  • Powered off: keeps the power state of the VM as Powered off and shows an error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "poweredOff"
   msg = "Received SOAP response fault from []: standbyGuest
The attempted operation cannot be performed in the current state (Powered off)."
}
  • Powered on: sets the power state of the VM to Suspended and shows no output, unless neither VMware Tools, nor open-vm-tools can be communicated with, then you get this error:
    (vmodl.fault.SystemError) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   reason = "Invalid fault"
   msg = "Received SOAP response fault from []: standbyGuest
vim.fault.GenericVmConfigFault"
}
  • Suspended: keeps the power state of the VM as Suspended and shows an error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "suspended"
   msg = "Received SOAP response fault from []: standbyGuest
The attempted operation cannot be performed in the current state (Suspended)."
}
vim-cmd vmsvc/power.off vmid Depends on the power state of the VM and either VMware Tools or open-vm-tools to be running:

  • Powered off: keeps the power state of the VM as Powered off and shows two lines
    Powering off VM:
Power off failed
  • Powered on: powers off the VM (bypasses any VMware Tools or open-vm-tools) and sets power state of the VM to Powered off showing one line
    Powering off VM:
  • Suspended: Hardware powers off the VM (bypasses any VMware Tools or open-vm-tools) and sets power state of the VM to Powered off showing one line
    Powering off VM:
vim-cmd vmsvc/power.on vmid Depends on the power state of the VM:

  • Powered on: keeps power state of the VM as Powered off showing two lines
    Powering on VM:
Power on failed
  • Powered off: keeps the power state of the VM as Powered off and shows one line
    Powering ofn VM:
  • Suspended: keeps the power state of the VM as Suspended showing two lines
    Powering on VM:
Power on failed

This also undoes a vim-cmd vmsvc/power.hibernate vmid, vim-cmd vmsvc/power.shutdown vmid or vim-cmd vmsvc/power.suspend vmid.
vim-cmd vmsvc/power.reboot vmid Depends on the power state of the VM and either VMware Tools or open-vm-tools to be running:

  • Powered off: keeps the power state of the VM as Powered off and shows the error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "poweredOff"
   msg = "Received SOAP response fault from []: rebootGuest
The attempted operation cannot be performed in the current state (Powered off)."
}
  • Powered on and either VMware Tools or open-vm-tools are running: uses VMware Tools or open-vm-tools to reboot the VM: keeps the power state of the VM as Powered on.
  • Powered on, but neither VMware Tools nor open-vm-tools are running: keeps the power state of the VM as Powered on and shows the error
    (vim.fault.ToolsUnavailable) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = 
   msg = "Received SOAP response fault from []: rebootGuest
Cannot complete operation because VMware Tools is not running in this virtual machine."
}
  • Suspended: keeps the power state of the VM as Suspended and shows the error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "suspended"
   msg = "Received SOAP response fault from []: rebootGuest
The attempted operation cannot be performed in the current state (Suspended)."
}
vim-cmd vmsvc/power.reset vmid Depends on the power state of the VM:

  • Powered off: keeps the power state of the VM as Powered off and shows two lines
    Reset VM:
Reset failed
  • Powered on: keeps the power state of the VM as Powered on, bypasses running VMware Tools or open-vm-tools (basically like a hardware reset button) and shows one line
    Reset VM:
  • Suspended: keeps the power state of the VM as Suspended and shows two lines
    Reset VM:
Reset failed
vim-cmd vmsvc/power.shutdown vmid Depends on the power state of the VM and either VMware Tools or open-vm-tools to be running:

  • Powered off: keeps the power state of the VM as Powered off and shows the error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "poweredOff"
   msg = "Received SOAP response fault from []: shutdownGuest
The attempted operation cannot be performed in the current state (Powered off)."
}
  • Powered on and either VMware Tools or open-vm-tools are running: uses VMware Tools or open-vm-tools to shutdown the VM and sets the power state of the VM to Powered off without showing any output.
  • Powered on, but neither VMware Tools nor open-vm-tools are running: keeps the power state of the VM as Powered on and shows the error
    (vim.fault.ToolsUnavailable) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = 
   msg = "Received SOAP response fault from []: shutdownGuest
Cannot complete operation because VMware Tools is not running in this virtual machine."
}
  • Suspended: keeps the power state of the VM as Suspended and shows the error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "suspended"
   msg = "Received SOAP response fault from []: shutdownGuest
The attempted operation cannot be performed in the current state (Suspended)."
}
vim-cmd vmsvc/power.suspend vmid When neither VMware Tools or open-vm-tools to be running, you get an error:

(vim.fault.ToolsUnavailable) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = 
   msg = "Received SOAP response fault from []: standbyGuest
Cannot complete operation because VMware Tools is not running in this virtual machine."
}

When a hibernate/suspend task is already running, you get an error:

(vim.fault.TaskInProgress) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   task = 'vim.Task:haTask-12-vim.VirtualMachine.suspend-1006072588'
   msg = "Received SOAP response fault from []: standbyGuest
Another task is already in progress."
}

Otherwise, depends on the power state of the VM:

  • Powered off: keeps the power state of the VM as Powered off and shows two lines
    Suspending VM:
Suspend failed
  • Powered on: keeps the power state of the VM as Powered on, and shows one line
    Suspending VM:

    unless neither VMware Tools, nor open-vm-tools can be communicated with, then you get this error:

    (vmodl.fault.SystemError) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   reason = "Invalid fault"
   msg = "Received SOAP response fault from []: standbyGuest
vim.fault.GenericVmConfigFault"
}
  • Suspended: keeps the power state of the VM as Suspended and shows two lines
    Suspending VM:
Suspend failed
vim-cmd vmsvc/power.suspendResume vmid Depends on the power state of the VM:

  • Powered off: keeps the power state of the VM as Powered off and shows two lines
    Suspend/Resuming the VM:
Suspend/Resume failed
  • Powered on: keeps the power state of the VM as Powered on, and shows one line
    Suspend/Resuming the VM:
  • Suspended: keeps the power state of the VM as Suspended and shows two lines
    Suspend/Resuming the VM:
Suspend/Resume failed

Note that in the VMware web console, this is shown as “Invoke FSR” which is a Fast Suspend Resume, which I think has to do with vMotion.

Notes:

  • there is no command vim-cmd vmsvc/power.startup vmid (to undo vim-cmd vmsvc/power.shutdown vmid), use vim-cmd vmsvc/power.on vmid in stead.
  • there is no command vim-cmd vmsvc/power.resume vmid (to undo vim-cmd vmsvc/power.suspend vmid), use vim-cmd vmsvc/power.on vmid in stead.
  • there is no command vim-cmd vmsvc/power.wakeup vmid (to undo vim-cmd vmsvc/power.hibernate vmid), use vim-cmd vmsvc/power.on vmid in stead.

Running the various power commands on all relevant VMs

This will be the topic for the next installment.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | 1 Comment »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 2

Posted by jpluimers on 2021/04/27

Last week ended up to be a kind of VMware ESXi heavey, and this week will be similar. So it is time for following up on VMware ESXi console: viewing all VMs, suspending and waking them up: part 1.

That one ended with

Then we get this to print some of the capture groups:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/Vmid:\1 Guest:\6 Version:\7 Name:\3 Datastore:\4 File:\5 Annotation:\8/p"

With this output:

Vmid:10 Guest:windows9_64Guest Version:vmx-14 Name:X9SRI-3F-W10P-EN-MEDIA Datastore:[EVO860_500GB] File:VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx Annotation:
Vmid:5 Guest:centos64Guest Version:vmx-11 Name:PPB Local_Virtual Machine_v4.0 Datastore:[EVO860_500GB] File:VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx Annotation:PowerPanel Business software(Local) provides the service which communicates

Figuring out power state for each VM

This will be in the next installment, as by now this already has become a big blog-post (:

I basically went the vim-cmd vmsvc way instead of the esxcli vm way. My motivation was the easier to understand ID values. They are the basis of virtually all vim-cmd vmsvc based commands:

# vim-cmd vmsvc --help
Commands available under vmsvc/:
acquiremksticket                 get.snapshotinfo                 
acquireticket                    get.spaceNeededForConsolidation  
createdummyvm                    get.summary                      
destroy                          get.tasklist                     
device.connection                getallvms                        
device.connusbdev                gethostconstraints               
device.ctlradd                   message                          
device.ctlrremove                power.getstate                   
device.disconnusbdev             power.hibernate                  
device.diskadd                   power.off                        
device.diskaddexisting           power.on                         
device.diskextend                power.reboot                     
device.diskremove                power.reset                      
device.getdevices                power.shutdown                   
device.nvdimmadd                 power.suspend                    
device.nvdimmremove              power.suspendResume              
device.toolsSyncSet              queryftcompat                    
devices.createnic                reload                           
get.capability                   setscreenres                     
get.config                       snapshot.create                  
get.config.cpuidmask             snapshot.dumpoption              
get.configoption                 snapshot.get                     
get.datastores                   snapshot.remove                  
get.disabledmethods              snapshot.removeall               
get.environment                  snapshot.revert                  
get.filelayout                   snapshot.setoption               
get.filelayoutex                 tools.cancelinstall              
get.guest                        tools.install                    
get.guestheartbeatStatus         tools.upgrade                    
get.managedentitystatus          unregister                       
get.networks                     upgrade                          
get.runtime

My “goto” for getting information is [Wayback] “vim-cmd vmsvc” site:vmware.com – Google Search, and a few sample pages are here:

  1. [Wayback] Performing common virtual machine-related tasks with command-line utilities (2012964) (showing that there are many tasks only vim-cmd vmsvc can do, but esxcli vm cannot)
  2. [Wayback] Powering on a virtual machine from the command line when the host cannot be managed using vSphere Client (1038043) (showing how to combine vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and vim-cmd vmsvc/power.on)
  3. [Wayback] Determine the power status of a virtual machine on an ESX or ESXi host (1003737) (showing vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and ps –auxwww | grep –i VM_NAME)
  4. [Wayback] Collecting information about tasks in VMware ESXi/ESX (1013003) (showing the relation between VMs and tasks using  vim-cmd vimsvc/task_list, vim-cmd vmsvc/getallvms and vim-cmd vimsvc/task_info)
  5. [Wayback] Unable to Power off a Virtual Machine in an ESXi host (1014165) (focussing on vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate, vim-cmd vmsvc/power.shutdown and vim-cmd vmsvc/power.off)
  6. [Wayback] Reloading a vmx file without removing the virtual machine from inventory (1026043) (showing vim-cmd vmsvc/getallvms and vim-cmd vmsvc/reload)
  7. [Wayback] Investigating virtual machine file locks on ESXi hosts (10051) (trying to show how to combine vim-cmd vmsvc/getallvms, grep, awk, find and xargs to find vmdk files, but fails because of parsing errors)

The pattern above is that most of the vim-cmd vmsvc examples are for power state and tasks. Not fully sure why, but my guess is it is what most people use it for. That kind of use what this series of posts also focuses on too, but certainly not the only use. Read the first numbered entry above to get a full grasp of what is possible. I hope to find time in the future to show some more examples outside the power and task realms.

Basically the only time you need to check out esxcli with VMs is when you cannot shut down a VM in a normal way. These links explain what to do in that case:

So let’s go back to basics, and start with getting info on all vim-cmd vmsvc commands.

Help on all vim-cmd vmsvc commands

Executing vim-cmd help vmsvc (preferred) or vim-cmd help vmsvc --help gives you all commands prepended with the line Commands available under vmsvc/:.

Executing vim-cmd help vmsvc/command prints the help for a single command (but vim-cmd help vmsvc/command -help first prints an error, then the help).

Here are the steps how I got the help help for all commands.

First I needed a list of all commands. This is already a multi-stage process, so below the full command I will explain the bits.

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort
  1. vim-cmd help vmsvc gives all the commands (two per line!) prepended by the line Commands available under vmsvc/:.
  2. sed '1d' stripts that line.
  3. xargs -n 1 -r echo does a lot of things:
    1. It parses the sed '1d' input line by line, splits each line into parts, combines all the parts, then executes echo with the combined parts
    2. The -n 1 ensures each invocation of echo takes only a single one of the combined parts
    3. -r is just a protection: if there is no input, then echo is never executed, resulting in empty output
  4. sort will sort all the combined output of all the echo invocations to undo the horizontal combination of parts that xargs did

Now getting the help is doing more of the above, with some more bits to explain:

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} vim-cmd help vmsvc/{}
  1. Normally, xargs will execute each command by appending the parameter inserting a space in front of each parameter
  2. -I {} will force xargs to put each argument just as is in the place where {} is used in the argument
  3. This executes vim-cmd help vmsvc/command in stead of vim-cmd help vmsvc/ command

The result is a long blob of text that is very hard to read as there are no separators between the commands. I saved it as a [Wayback] vim-cmd help for each vmsvc command.txt gist.

With a sh -c shell trick, you can add some more information and separation to the output by embedding :

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} sh -c 'echo "-----" ; echo "help for vim-cmd help vmsvc/{}" ; echo ; vim-cmd help vmsvc/{}'

I have added the output to the [Wayback] delimited vim-cmd help for each vmsvc command.txt gist.

Commands taking a vmid parameter

Now that we know how to output all help, we can filter on it.

An interesting one is to filder only commands taking a vmid parameter:

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} vim-cmd help vmsvc/{} | grep -iw vmid

On VMware ESXi 6.7, this gets you the list:

Usage: acquiremksticket vmid
Usage: acquireticket vmid ticketType
Usage: destroy vmid
Usage: device.connection vmid deviceKey connect
Usage: device.connusbdev vmid usbid
Usage: device.ctlradd vmid ctlr_type bus_number
Usage: device.ctlrremove vmid ctlr_type bus_number
Usage: device.disconnusbdev vmid usbid
Usage: device.diskadd vmid size controller_numer unit_number datastore [ctlr_type]
Usage: device.diskaddexisting vmid disk_file controller_number unit_number [ctlr_type]
Usage: device.diskextend vmid new_size controller_numer unit_number [ctlr_type]
Usage: device.diskremove vmid controller_number unit_number delete_file [controller_type]
Usage: device.getdevices vmid
Usage: device.nvdimmadd vmid size
Usage: device.nvdimmremove vmid deviceKey
Usage: device.toolsSyncSet vmid new state
Usage: devices.createnic vmid adapter-type network-id [network-type]
Usage: get.capability vmid
Usage: get.config vmid
Usage: get.config.cpuidmask vmid
Usage: get.configoption vmid
Usage: get.datastores vmid
Usage: get.disabledmethods vmid
Usage: get.environment vmid
Usage: get.filelayout vmid
Usage: get.filelayoutex vmid
Usage: get.guest vmid
Usage: get.guestheartbeatStatus vmid
Usage: get.managedentitystatus vmid
Usage: get.networks vmid
Usage: get.runtime vmid
Usage: get.snapshotinfo vmid
Usage: get.spaceNeededForConsolidation vmid
Usage: get.summary vmid
Usage: get.tasklist vmid
Usage: message vmid [messageId] [messageChoice]
Usage: power.getstate vmid
Usage: power.hibernate vmid
Usage: power.off vmid
Usage: power.on vmid
Usage: power.reboot vmid
Usage: power.reset vmid
Usage: power.shutdown vmid
Usage: power.suspend vmid
Usage: power.suspendResume vmid
Usage: queryftcompat vmid [faultToleranceType]
Usage: reload vmid
Usage: setscreenres vmid width height
Usage: snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory] [quiesced]
Usage: snapshot.get vmid
Usage: snapshot.remove vmid snapshotId [removeChildren]
Usage: snapshot.removeall vmid
Usage: snapshot.revert vmid snapshotId suppressPowerOn
Usage: snapshot.setoption [OPTIONS] vmid
Usage: tools.cancelinstall vmid
Usage: tools.install vmid
Usage: tools.upgrade vmid [args]
Usage: unregister vmid
Usage: upgrade vmid [vm_hwversion]

In the above list, the bold entries have to do with power, that is what this series is supposed to center around, so more on that tomorrow.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Awk, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, fgrep, Power User, Scripting, sed, sed script, sh, Sh Shell, Software Development, sort, Virtualization, VMware, VMware ESXi, xargs | Leave a Comment »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 1

Posted by jpluimers on 2021/04/22

I think the easiest way to list all VMs is the vim-cmd vmsvc/getallvms command, but it has a big downside: the output is a mess.

The reason is that the output:

  • has a lot of columns (Vmid, Name, Datastore, File, Guest OS, Version, Annotation),
  • more than 500 characters per line (eat that 1080p monitor!),
  • and potentially more than one line per VM as the Annotation is a free-text field that can have newlines.

Example output on one of my machines:


Vmid Name File Guest OS Version Annotation
10 X9SRI-3F-W10P-EN-MEDIA [EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx windows9_64Guest vmx-14
5 PPB Local_Virtual Machine_v4.0 [EVO860_500GB] VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx centos64Guest vmx-11 PowerPanel Business software(Local) provides the service which communicates
with the UPS through USB or Serial cable and relays the UPS state to each Remote on other computers
via a network.
It also monitors and logs the UPS status. The computer which has been installed the Local provides
graceful,
unattended shutdown in the event of the power outage to protect the hosted computer.
…

view raw

vim-cmd.vmsvc.getallvms.txt

hosted with ❤ by GitHub

As an alternative, you could use esxcli vm process list, but that gives IDs that are way harder to remember:


PPB Local_Virtual Machine_v4.0
World ID: 2099719
Process ID: 0
VMX Cartel ID: 2099713
UUID: 56 4d 74 f8 c8 22 41 27-a3 88 49 df 8b dc d6 63
Display Name: PPB Local_Virtual Machine_v4.0
Config File: /vmfs/volumes/5d35e7d8-e8df636f-46b9-0025907d9d5c/VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx
X9SRI-3F-W10P-EN-MEDIA
World ID: 2099728
Process ID: 0
VMX Cartel ID: 2099717
UUID: 56 4d 51 ac f6 cf e4 0b-b6 86 2f 53 a2 8a 4b ea
Display Name: X9SRI-3F-W10P-EN-MEDIA
Config File: /vmfs/volumes/5d35e7d8-e8df636f-46b9-0025907d9d5c/VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx

view raw

esxcli.vm.process.list.txt

hosted with ❤ by GitHub

I got both of the above commands from [Wayback] VMware Knowledge Base: Performing common virtual machine-related tasks with command-line utilities (2012964).

Back to the columns that vim-cmd vmsvc/getallvms returns:

  • Vmid is an unsigned integer
  • Name can have spaces
  • Datastore has square brackets [ and ] around it
  • File can contain spaces
  • Guest OS is an identifier without spaces (it is a value from [Wayback] the vSphere API VcVirtualMachineGuestOsIdentifier
  • Version looks like vmx-# where # is an unsigned integer
  • Annotation is multi-line free-form so potentially can have lines starting like being Vmid, but the chance that a line looks exactly like a non-annotated one is very low

So let’s find a grep or  sed filter to get just the lines without annotation continuations. Though in general I try to avoid regular expressions as they are hard to both write and read, but with Busybox there is no much choice.

I choose sed, just in case I wanted to do some manipulation in addition to matching.

Busybox sed

Though the source code [Wayback] sed.c\editors – busybox – BusyBox: The Swiss Army Knife of Embedded Linux indicates sed.c - very minimalist version of sed, the implementation actually is reasonably feature rich, just not feature complete. That’s OK given the aim of Busybox to be small.

Luckily, deep in the busybox sed code, it indicates that extended regular expressions are supported (support is in [Wayback] /uClibc/plain/libc/misc/regex/regcomp.c (look for regcomp, do not get confused by xregcomp on call sites as that is [Wayback] just a tiny wrapper to call regcomp).

The support has become better over time, like [Wayback] gnu – sed Command on BusyBox expects different syntax? – Super User shows.

This means far less escaping than basic regular expressions, capture groups are supported as well as character classes (so [[:digit:]] is more readable than [0-9]), and the + is supported to match once or more (so [0-9]+ means one or more digits, as does [[:digit:]]+, but [d]+ or \d+ don’t ). Unfortunately named capture groups are not supported (so documenting parts of the regular expression like (?<Vmid>^[[:digit:]]+) is not possible, it will give you an error [Wayback] Invalid preceding regular expression).

But first a few of the sed commandline options and their order:

vim-cmd vmsvc/getallvms | sed -n -E -e '/(^[[:digit:]]+)/p'
  1. -n outputs only matching lines that have a p print command.
  2. -E allows extended regular expressions (you can also use -r for that)
  3. -e adds a (in this case extended) regular expression
  4. '/(^[[:digit:]]+)/p' is the extended regular expression embedded in quotes
    1. / at the start indicates that sed should match the regular expression on each line it parses
    2. /p at the end indicates the matching line should be printed
    3. Parentheses ( and ) surround a capture group
    4. ^[[:digit:]]+ matches 1 or more digits at the start of the line

The grep command is indeed much shorter, but does not allow post-editing:

vim-cmd vmsvc/getallvms | grep -E '(^[[:digit:]]+)'

Building a sed filter

I came up with the below sed regular expression to filter out lines:

  1. starting with a Vmid unsigned integer
  2. having a [Datastore] before the File
  3. have a Guest OS identifier after File
  4. have a Version matching vmx-# after File where # is an unsigned integer
  5. optionally has an Annotation after Version
vim-cmd vmsvc/getallvms | sed -n -E -e  "/^([[:digit:]]+)(\s+)((\S.+\S)?)(\s+)(\[\S+\])(\s+)(.+\.vmx)(\s+)(\S+)(\s+)(vmx-[[:digit:]]
+)(\s*?)((\S.+)?)$/p"

A longer expression that I used to fiddle around with is at regex101.com/r/A7MfKu and contains named capture groups. I had to nest a few groups and use the ? non-greedy (or lazy) operator a few times to ensure the fields would not include the spaces between the columns.

Others use different expressions as for instance explained in [Wayback] Get all VMs with “vmware-vim-cmd vmsvc/getallvms” – VMware Technology Network VMTN:

Output from “vim-cmd vmsvc/getallvms” is really challenging to process. Our normal approaches such as awk column indexes, character index, and regular expression are all error prone here. The character index of each column varies depending on maximum field length of, for example, VM name. And the presence of spaces in VM names throws off processing as awk columns. And VM name could contain almost any character, foiling regex’s.

Printing capture groups

The cool thing is that it is straightforward to modify the expression to print any of the capture groups in the order you wish: you convert the match expression (/match/p) into a replacement expression (s/match/replace/p) and print the required capture groups in the replace part. A short example is at [Wayback] regex – How to output only captured groups with sed? – Stack Overflow.

There is one gotcha though: Busybox sed only allows single-digit capture group numbers, and we have far more than 9 capture groups. This fails and prints 0 after the output of capture group 1 instead of printing capture group 10, similar for 2 after group 1 instead of printing group 12:

vim-cmd vmsvc/getallvms | sed -n -E -e  "s/^([[:digit:]]+)(\s+)((\S.+\S)?)(\s+)(\[\S+\])(\s+)(.+\.vmx)(\s+)(\S+)(\s+)(vmx-[[:digit:]]+)(\s*?)((\S.+)?)$/Vmid:\1 Guest:\10 Version:\12 Name:\3 Datastore:\7 File:\8/p"

So we need to cut down on capture groups first by removing all capture groups around the \s white-space matching:

vim-cmd vmsvc/getallvms | sed -n -E -e  "/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/p"

Then we get this to print some of the capture groups:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/Vmid:\1 Guest:\6 Version:\7 Name:\3 Datastore:\4 File:\5 Annotation:\8/p"

With this output:

Vmid:10 Guest:windows9_64Guest Version:vmx-14 Name:X9SRI-3F-W10P-EN-MEDIA Datastore:[EVO860_500GB] File:VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx Annotation:
Vmid:5 Guest:centos64Guest Version:vmx-11 Name:PPB Local_Virtual Machine_v4.0 Datastore:[EVO860_500GB] File:VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx Annotation:PowerPanel Business software(Local) provides the service which communicates

Figuring out power state for each VM

This will be in the next installment, as by now this already has become a big blog-post (:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, RegEx, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

VMware ESXi 6 and 7: checking and setting/clearing maintenance mode from the console

Posted by jpluimers on 2021/04/21

Every now and then it is useful to be able to do maintenance work from the ESXi console addition to the ESXi web-user interface.

I know there are many sites having this information, but many of them forgot to format the statements with code markup, so parameters with two dashes -- (each a Wayback Unicode Character ‘HYPHEN-MINUS’ (U+002D)) now have become an [Wayback] Unicode Character ‘EN DASH’ (U+2013) which is incompatible with most console programs, especially the ESXi ones (as they are Busybox based to minimise footprint).

Note you can use this small site (which runs in-browser, so does not phone home) to get the unicode code points for any string: [Wayback] What Unicode character is this ?.

Links like below (most on the vmware.com domain) have this EN DASH and make me document things on my blog instead of trying code directly from blogs or forum posts:

So below are three commands I use that have to do with the maintenance mode (the mode that for instance you can use to update an ESXi host to the latest patch level).

    1. Check the maintenance mode (which returns Enabled or Disabled):
      esxcli system maintenanceMode get
    2. Enable maintenance mode (which returns nothing when succeeded, and Maintenance mode is already enabled. when failed):
      esxcli system maintenanceMode set --enable true
    3. Disable maintenance mode (which returns nothing when succeeded, and Maintenance mode is already disabled. when failed):
      esxcli system maintenanceMode get

Some examples, especially an the various output possibilities (commands in bold, output in italic):

# esxcli system maintenanceMode get
Disabled
# esxcli system maintenanceMode set --enable false
Maintenance mode is already disabled.
# esxcli system maintenanceMode set --enable true 
# esxcli system maintenanceMode get
Enabled
# esxcli system maintenanceMode set --enable true
Maintenance mode is already enabled.
# esxcli system maintenanceMode set --enable false
# esxcli system maintenanceMode get
Disabled

I made these scripts for this:

  • esxcli-maintenanceMode-show.sh:
    #!/bin/sh
esxcli system maintenanceMode get
  • esxcli-maintenanceMode-enter.sh:
    #!/bin/sh
esxcli system maintenanceMode set --enable true
  • esxcli-maintenanceMode-exit.sh:
    #!/bin/sh
esxcli system maintenanceMode set --enable false

Note I have not checked the exit codes for these esxcli commands yet, but did blog about how to do that: Busybox sh (actually ash derivative dash): checking exit codes.

–jeroen

Posted in BusyBox, Development, Encoding, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Software Development, Unicode, Virtualization, VMware, VMware ESXi | Leave a Comment »

« Previous Entries
Next Entries »