The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,731 other followers

ESXi: editing /etc/vmware/hostd/vmInventory.xml to fix the datastore UUID for unavailable VMs part 2

Posted by jpluimers on 2022/02/01

I started my post ESXi: editing /etc/vmware/hostd/vmInventory.xml to fix the datastore UUID for unavailable VMs with

In case I ever need this on ESXi: Insights into the VMware inventory files (vmAutoStart.xml and vmInventory.xml on ESXi; inventory.vmls on VMware Workstation/Player)

Since almost all of my blog is about things I bumped into in real life, this post was a preparation because I kind of expected this to indeed happen, and it did.

Below are the screenshots and steps I took. Of course it is an N=1 experience, so your situation might differ, but I tried to be thorough and not miss any steps.

  1. First of all, you always have to ensure that none of the virtual machines are running on the ESXi host (ideally they should be powered off, but hibernated/suspended usually works as well). For this, I use vim-cmd-hibernate-running-VMs.sh or vim-cmd-power-off-powered-on-VMs.sh from VMware ESXi console: viewing all VMs, suspending and waking them up: part 5 .
  2. Then the ESXi host needs to be in maintenance mode (I use esxcli-maintenanceMode-show.sh and esxcli-maintenanceMode-enter.sh and esxcli-maintenanceMode-exit.sh from VMware ESXi 6 and 7: checking and setting/clearing maintenance mode from the console)
  3. After taking two of the NVMe devices out (they were about to fail, I had back-ups), the inventory looked like this:
    ESXi VMs after taking out failing NVMe SSDs

    ESXi VMs after taking out failing NVMe SSDs

    None of the virtual machines are available, and the full path of their .vmx file is visible instead of the virtual machine names that normally are obtained from the .vmx files.

  4. Since I knew the old and new vmfs UUIDs, I could use vi to edit /etc/vmware/hostd/vmInventory.xml and perform these search and replace operations (and save+quit using :wq):
    :%s/5ad4aeea-6954841c-470e-0cc47aaa9742/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/g
    :%s:5ad4af1b-f3ae285c-e0f4-0cc47aaa9742:6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam:g
    :%s:/archiveteam-warrior-v3-20171013 instance 2/:/archiveteam-warrior-v3-20171013.instance-2/:g
    

    I always have to lookup the vi search/replace phrases, as they are similar, but not identical to the sed syntax, so I write this blog post that I use to find them back: Source: On learning vi/vim.

  5. Now I could restart the services, which gave an error (note the connot spelling mistake):
    # /sbin/services.sh restart
    Errors: 
    Invalid operation requested: This ruleset is required and connot be disabled
  6. I inspected the below log files, and there were no anomalies: they all looked similar to what happens after a server reboot (which takes quite some time because of the thorough boot checks of the BIOS SuperMicro on the motherboard and HBA adapters): /var/log/hostd.log, /var/log/jumpstart-stdout.log, /var/log/vmkernel.log, /var/log/vpxa.log (log files via [Wayback] Services Failing on ESXi 5.5 Host – Can’t Restart Services | [H]ard|Forum – search for “normalish” and “stableish” – and [Wayback] services.sh restart problem – This ruleset is requ… – VMware Technology Network VMTN).
  7. After fixing /etc/vmware/hostd/vmInventory.xml and restarting the services:
    ESXi VMs after editing /etc/vmware/hostd/vmInventory.xml to correct vmfs UUIDs and paths.

    ESXi VMs after editing /etc/vmware/hostd/vmInventory.xml to correct vmfs UUIDs and paths.

    The (blurred) virtual machine names are back as well as power states, disk sizes, operating systems, etc.

  8. I wasn’t done yet, because since the .vmx files themselves contain absolute references to at least .iso and .vswp files, so I needed to fix that by first getting a list of .vmx files from the vmInventory.xml file:
    # sed -n -E -e "s/^ +<vmxCfgPath>(.+)<\/vmxCfgPath>$/\1/p" /etc/vmware/hostd/vmInventory.xml
  9. This will get you the files; usually they are in entries like ide0:0.fileName, sata0:0.fileName and sched.swap.derivedName:
    # sed -n -E -e "s/^ +<vmxCfgPath>(.+)<\/vmxCfgPath>$/\1/p" /etc/vmware/hostd/vmInventory.xml | xargs -n 1 -I {} grep "vmfs" {}
  10. Appending | sed -n -E -e "s#^.+? = (\"/.*\")#\1#p" | sort | uniq | xargs ls -Alh will list the files in unique sorted order:
    # sed -n -E -e "s/^ +<vmxCfgPath>(.+)<\/vmxCfgPath>$/\1/p" /etc/vmware/hostd/vmInventory.xml | xargs -n 1 -I {} grep "vmfs" {} | sed -n -E -e "s#^.+? = (\"/.*\")#\1#p" | sort | uniq |  xargs ls -Alh

    Note that the backslash \ escaped double quotes " are inside the parenthesis () because filenames can contain spaces.

  11. For me the list was like this:
    Valid and invalid files referenced from .vmx files

    Valid and invalid files referenced from .vmx files

  12. This gave me the volume UUIDs that I had to replace, of which the first bunch until x were all .iso disk images referenced by disconnected optical devices and the last ones available .iso disk images. The remaining ones corresponded to replace operations I already had done on the vmInventory.xml file, so I could repeat that exercise, open each file in vi and repeat the search operations. This is the statement to open each .vmx file in vi:
    # sed -n -E -e "s/^ +<vmxCfgPath>(.+)<\/vmxCfgPath>$/\1/p" /etc/vmware/hostd/vmInventory.xml | xargs -n 1 -I {} grep -l "vmfs" {} | xargs -n 1 -I [] echo "vi \"[]\""

    Note that the double quotes " because filenames can contain spaces, and the same search and replace operations I used on the /etc/vmware/hostd/vmInventory.xml file:

    :%s/5ad4aeea-6954841c-470e-0cc47aaa9742/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/g
    :%s:5ad4af1b-f3ae285c-e0f4-0cc47aaa9742:6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam:g
    :%s:/archiveteam-warrior-v3-20171013 instance 2/:/archiveteam-warrior-v3-20171013.instance-2/:g
    
  13. Now the ESXi host needs to get out of maintenance mode (I use esxcli-maintenanceMode-show.sh and esxcli-maintenanceMode-exit.sh from VMware ESXi 6 and 7: checking and setting/clearing maintenance mode from the console) immediately followed by the next step (otherwise someone else could start a virtual machine with the old .vmx information)
  14. Reloading all the vmx configurations can be done with vim-cmd-reload-all-VM-vmx-configurations.sh as I described yesterday in Source: ESXi: reloading all virtual machines from their (potentially) vmx files.
  15. Note that when powering on virtual machines, you need to answer a question like this for each virtual machine:

    This virtual machine might have been moved or copied. In order to configure certain management and networking features, VMware ESX needs to know if this virtual machine was moved or copied. If you don’t know, answer “I Copied It”.

    • I Moved It
    • I Copied It

One more thing

A final note: I need to check out if .vswp files need to be there at all, as my ESXi servers have plenty of physical memory in order not to swap out to disk. More on that in a future blog post.

jeroen

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

 
%d bloggers like this: