January 2022
M	T	W	T	F	S	S
	1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30
31

Archive for the ‘Scripting’ Category

ESXi: on the console/ssh, when a moved VM pauses during power-on: show which VMs have messages waiting, then answer them

Posted by jpluimers on 2022/01/27

First the script that display messages for all virtual machines, vim-cmd-display-messages-for-all-VMs.sh:

#!/bin/sh
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    echo "Messages for VM with id ${vmid} which has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    vim-cmd vmsvc/message ${vmid}
done
exit 0

It is very similar to vim-cmd-reload-all-VM-vmx-configurations.sh from Source: ESXi: reloading all virtual machines from their (potentially) vmx files.

Messages I know either equal “`No message`” or are about “`This virtual machine may have been moved or copied.`“

If there is no available message, then you always get the stock message No message., so this is something you can use as a check in scripts.

The only messages I know of (and the only one I could find when searching for [Wayback] “vim-cmd vmsvc/message” – Google Search) are like these (where # is a number, like 18 , 23, 5910112 or 6233874: they are unique messageId values so you can refer to them when answering the message):

ESXi 4.x era (from [Wayback] Answering a virtual machine related question from the command line (1026835)):

Virtual machine message #:
msg.uuid.altered:This virtual machine may have been moved or copied.

To configure certain management and networking features VMware ESX needs to know which.

Did you move this virtual machine, or did you copy it?
If you don't know, answer "I copied it".

0. Cancel (Cancel)
1. I _moved it (I _moved it)
2. I _copied it (I _copied it) [default]

ESXi 5.x era (from [Wayback] Starting a virtual machine from the ESXi CLI pauses forever | viktorious.nl – Virtualization & Cloud Management):

Virtual machine message #:
msg.uuid.altered:This virtual machine may have been moved or copied.

To configure certain management and networking features VMware ESX needs to know which.

Did you move this virtual machine, or did you copy it?
If you don't know, answer "I copied it".

0. Cancel (Cancel)
1. I moved it (I moved it)
2. I copied it (I copied it) [default]

ESXi 5.5 era (from [Wayback] Moving a powered off VM from CLI · RR Labs):

Virtual machine message #:
This virtual machine might have been moved or copied. In order to configure certain management and networking features, VMware ESX needs to know if this virtual machine was moved or copied. If you don't know, answer "I Copied It".
   0. Cancel (Cancel)
   1. button.uuid.movedTheVM (I Moved It)
   2. button.uuid.copiedTheVM (I Copied It) [default]

ESXi 6.5 (one of my own machines):

Virtual machine message #:
This virtual machine might have been moved or copied. In order to configure certain management and networking features, VMware ESX needs to know if this virtual machine was moved or copied. If you don't know, answer "I Copied It". 
   0. button.uuid.cancel (Cancel)
   1. button.uuid.movedTheVM (I Moved It)
   2. button.uuid.copiedTheVM (I Copied It) [default]

You get the pending message for a specific machine 25 using the vim-cmd vmsvc/message 25 like here:

# vim-cmd vmsvc/message 25
Virtual machine message 5970112:
This virtual machine might have been moved or copied. In order to configure certain management and networking features, VMware ESX needs to know if this virtual machine was moved or copied. If you don't know, answer "I Copied It". 
   0. button.uuid.cancel (Cancel)
   1. button.uuid.movedTheVM (I Moved It)
   2. button.uuid.copiedTheVM (I Copied It) [default]

The “[default]” will automatically be answered on ESXi 6.5 after four (4) minutes. For this, I got two (2) identical lines of log in /var/log/hostd.log for this:

2021-05-15T13:10:27.062Z info hostd[AE81B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013/archiveteam-warrior-v3-20171013.vmx] Answered question 5970112 2021-05-15T13:10:27.062Z info hostd[AE81B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013/archiveteam-warrior-v3-20171013.vmx] Answered question 5970112

Intermezzo: `/var/log/hostd.log` extract when not answering the “moved” question

A few observations of the log file is at the bottom of this post in a gist [Archive.is].

each vim-cmd vmsvc invocation leads to

a logon event with log-entries like these where I really wonder where password comes from. as this is not a new interactive logon:

Accepted password for user root from 127.0.0.1
2021-05-15T13:05:58.568Z info hostd[AEC2B70] [Originator@6876 sub=Vimsvc opID=vim-cmd-56-584b] [Auth]: User root
2021-05-15T13:05:58.568Z verbose hostd[AEC2B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-56-584b] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:05:58.568Z info hostd[AEC2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-56-584b] Event 762 : User root@127.0.0.1 logged in as VMware-client/6.5.0
2021-05-15T13:05:58.568Z verbose hostd[AEC2B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-56-584b] RecordOp ADD: sessionList["5263dcbb-1baf-7a5a-ab7b-ea07e6499e50"], ha-sessionmgr. Applied change to temp map.

a logoff event with log-entries like these (note a slightly different opID, but the same sessionList ID):

2021-05-15T13:05:58.623Z verbose hostd[AEC2B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-56-5859 user=root] RecordOp ASSIGN: info, haTask--vim.ServiceInstance.retrieveContent-130896484. Applied change to temp map.
2021-05-15T13:05:58.623Z verbose hostd[AEC2B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-56-5859 user=root] RecordOp ASSIGN: info, haTask--vim.ServiceInstance.retrieveContent-130896484. Applied change to temp map.
2021-05-15T13:05:58.625Z verbose hostd[AE40B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-56-585a user=root] RecordOp ASSIGN: info, haTask--vim.SessionManager.logout-130896485. Applied change to temp map.
2021-05-15T13:05:58.625Z verbose hostd[AE40B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-56-585a user=root] RecordOp REMOVE: sessionList["5263dcbb-1baf-7a5a-ab7b-ea07e6499e50"], ha-sessionmgr. Applied change to temp map.
2021-05-15T13:05:58.625Z verbose hostd[AE40B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-56-585a user=root] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:05:58.625Z info hostd[AE40B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-56-585a user=root] Event 763 : User root@127.0.0.1 logged out (login time: Saturday, 15 May, 2021 13:05:58, number of API invocations: 0, user agent: VMware-client/6.5.0)
2021-05-15T13:05:58.625Z verbose hostd[AE40B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-56-585a user=root] RecordOp ASSIGN: info, haTask--vim.SessionManager.logout-130896485. Applied change to temp map.

Reading a pending message:

Accepted password for user root from 127.0.0.1
2021-05-15T13:06:32.389Z verbose hostd[B040B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-d0-588a] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:06:32.389Z info hostd[B040B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-d0-588a] Event 767 : User root@127.0.0.1 logged in as VMware-client/6.5.0
2021-05-15T13:06:32.389Z verbose hostd[B040B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-d0-588a] RecordOp ADD: sessionList["52967fcb-195c-96b3-429b-4762cbb5fdbd"], ha-sessionmgr. Applied change to temp map.
2021-05-15T13:06:32.453Z verbose hostd[BBC2B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-d0-5898 user=root] RecordOp ASSIGN: info, haTask--vim.ServiceInstance.retrieveContent-130896487. Applied change to temp map.
2021-05-15T13:06:32.453Z verbose hostd[BBC2B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-d0-5898 user=root] RecordOp ASSIGN: info, haTask--vim.ServiceInstance.retrieveContent-130896487. Applied change to temp map.
2021-05-15T13:06:32.455Z verbose hostd[BB81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-d0-5899 user=root] RecordOp ASSIGN: info, haTask--vim.SessionManager.logout-130896488. Applied change to temp map.
2021-05-15T13:06:32.455Z verbose hostd[BB81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-d0-5899 user=root] RecordOp REMOVE: sessionList["52967fcb-195c-96b3-429b-4762cbb5fdbd"], ha-sessionmgr. Applied change to temp map.
2021-05-15T13:06:32.455Z verbose hostd[BB81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-d0-5899 user=root] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:06:32.455Z info hostd[BB81B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-d0-5899 user=root] Event 768 : User root@127.0.0.1 logged out (login time: Saturday, 15 May, 2021 13:06:32, number of API invocations: 0, user agent: VMware-client/6.5.0)
2021-05-15T13:06:32.456Z verbose hostd[BB81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-d0-5899 user=root] RecordOp ASSIGN: info, haTask--vim.SessionManager.logout-130896488. Applied change to temp map.

A cron event of the automatic configuration backup:

2021-05-15T13:06:31.559Z info hostd[AE40B70] [Originator@6876 sub=SysCommandPosix] ForkExec(/sbin/backup.sh) 181092
2021-05-15T13:06:32.798Z info hostd[AE40B70] [Originator@6876 sub=Hostsvc.HostConfigSyncManager] Backup succeeded

Powering up a moved virtual machine, leading up to a cancelable task and the message:

Accepted password for user root from 127.0.0.1
2021-05-15T13:06:26.505Z info hostd[B081B70] [Originator@6876 sub=Vimsvc opID=vim-cmd-8f-585d] [Auth]: User root
2021-05-15T13:06:26.505Z verbose hostd[B081B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-585d] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:06:26.505Z info hostd[B081B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-8f-585d] Event 764 : User root@127.0.0.1 logged in as VMware-client/6.5.0
2021-05-15T13:06:26.505Z verbose hostd[B081B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-585d] RecordOp ADD: sessionList["52165943-fe83-da06-513a-6c2c38b0e470"], ha-sessionmgr. Applied change to temp map.
2021-05-15T13:06:26.548Z info hostd[AE40B70] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-8f-586a user=root] Task Created : haTask-25-vim.VirtualMachine.powerOn-130896486
2021-05-15T13:06:26.548Z verbose hostd[AE40B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-586a user=root] RecordOp ADD: recentTask["haTask-25-vim.VirtualMachine.powerOn-130896486"], 25. Sent notification immediately.
2021-05-15T13:06:26.548Z verbose hostd[AE40B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-586a user=root] RecordOp ADD: recentTask["haTask-25-vim.VirtualMachine.powerOn-130896486"], ha-taskmgr. Applied change to temp map.
2021-05-15T13:06:26.548Z verbose hostd[B940B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-586a user=root] RecordOp ASSIGN: info, haTask-25-vim.VirtualMachine.powerOn-130896486. Applied change to temp map.
2021-05-15T13:06:26.551Z verbose hostd[B940B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-586a user=root] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:06:26.551Z info hostd[B940B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-8f-586a user=root] Event 765 : archiveteam-warrior-v3-20171013 on host ESXi-X10SRH-CF in ha-datacenter is starting
2021-05-15T13:06:26.551Z info hostd[B940B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013/archiveteam-warrior-v3-20171013.vmx opID=vim-cmd-8f-586a user=root] State Transition (VM_STATE_OFF -> VM_STATE_POWERING_ON)
2021-05-15T13:06:26.551Z info hostd[B940B70] [Originator@6876 sub=Libs opID=vim-cmd-8f-586a user=root] SIOC: SIOC is notified not to start injector
2021-05-15T13:06:26.552Z verbose hostd[B940B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-586a user=root] RecordOp ASSIGN: disabledMethod, 25. Sent notification immediately.
2021-05-15T13:06:26.552Z verbose hostd[B940B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-586a user=root] RecordOp ASSIGN: info.cancelable, haTask-25-vim.VirtualMachine.powerOn-130896486. Applied change to temp map.
2021-05-15T13:06:26.552Z verbose hostd[B940B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-586a user=root] RecordOp ASSIGN: info.cancelable, haTask-25-vim.VirtualMachine.powerOn-130896486. Applied change to temp map.
2021-05-15T13:06:26.553Z info hostd[B940B70] [Originator@6876 sub=Libs opID=vim-cmd-8f-586a user=root] DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.
2021-05-15T13:06:26.553Z info hostd[B940B70] [Originator@6876 sub=Libs opID=vim-cmd-8f-586a user=root] Could not expand environment variable HOME.
2021-05-15T13:06:26.553Z info hostd[B940B70] [Originator@6876 sub=Libs opID=vim-cmd-8f-586a user=root] DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.
2021-05-15T13:06:26.556Z info hostd[B940B70] [Originator@6876 sub=Libs opID=vim-cmd-8f-586a user=root] Starting vmx as /bin/vmx
2021-05-15T13:06:26.556Z info hostd[B940B70] [Originator@6876 sub=Libs opID=vim-cmd-8f-586a user=root] Starting vmx as /bin/vmx
2021-05-15T13:06:26.557Z info hostd[B940B70] [Originator@6876 sub=vm:VigorExecVMXExCommon: VM /vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013/archiveteam-warrior-v3-20171013.vmx opID=vim-cmd-8f-586a user=root] in directory /vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013
2021-05-15T13:06:26.557Z info hostd[B940B70] [Originator@6876 sub=vm:VigorExecVMXExCommon: Exec()'ing /bin/vmx /vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013/archiveteam-warrior-v3-20171013.vmx opID=vim-cmd-8f-586a user=root]
2021-05-15T13:06:26.557Z info hostd[B940B70] [Originator@6876 sub=Libs opID=vim-cmd-8f-586a user=root] Vigor: VMKernel_ForkExec(/bin/vmx, detached=1): status=0 pid=181085
2021-05-15T13:06:26.611Z verbose hostd[B081B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: info.cancelable, haTask-25-vim.VirtualMachine.powerOn-130896486. Applied change to temp map.
2021-05-15T13:06:26.625Z verbose hostd[A0E7B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: info.progress, haTask-25-vim.VirtualMachine.powerOn-130896486. Applied change to temp map.
2021-05-15T13:06:26.798Z verbose hostd[BBC2B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:06:26.798Z info hostd[BBC2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 766 : Message on archiveteam-warrior-v3-20171013 on ESXi-X10SRH-CF in ha-datacenter: This virtual machine might have been moved or copied.
--> In order to configure certain management and networking features, VMware ESX needs to know if this virtual machine was moved or copied.
-->
--> If you don't know, answer "I Co_pied It".
-->
-->
2021-05-15T13:06:26.798Z verbose hostd[BBC2B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: runtime.question, 25. Sent notification immediately.
2021-05-15T13:06:26.799Z verbose hostd[BBC2B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: summary.runtime.question, 25. Sent notification immediately.
2021-05-15T13:06:26.799Z verbose hostd[BBC2B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: disabledMethod, 25. Sent notification immediately.

The time out while waiting for the answer to the message immediately leads to the actual power on state of the virtual machine:

2021-05-15T13:10:27.043Z verbose hostd[A0E7B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: info.progress, haTask-25-vim.VirtualMachine.powerOn-130896486. Applied change to temp map.
2021-05-15T13:10:27.053Z verbose hostd[B040B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: runtime.question, 25. Sent notification immediately.
2021-05-15T13:10:27.053Z verbose hostd[B040B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: summary.runtime.question, 25. Sent notification immediately.
2021-05-15T13:10:27.053Z verbose hostd[B040B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: disabledMethod, 25. Sent notification immediately.
2021-05-15T13:10:27.053Z verbose hostd[B040B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:10:27.053Z info hostd[B040B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 769 : Message on archiveteam-warrior-v3-20171013 on ESXi-X10SRH-CF in ha-datacenter: This virtual machine might have been moved or copied.
--> In order to configure certain management and networking features, VMware ESX needs to know if this virtual machine was moved or copied.
-->
--> If you don't know, answer "I Co_pied It".
-->
-->
2021-05-15T13:10:27.062Z info hostd[AE81B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013/archiveteam-warrior-v3-20171013.vmx] Answered question 5970112
2021-05-15T13:10:27.062Z info hostd[AE81B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013/archiveteam-warrior-v3-20171013.vmx] Answered question 5970112
2021-05-15T13:10:27.079Z info hostd[AE81B70] [Originator@6876 sub=Hostsvc] Decremented SIOC Injector Flag2
2021-05-15T13:10:27.080Z verbose hostd[AE81B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:10:27.080Z info hostd[AE81B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 770 : archiveteam-warrior-v3-20171013 on  ESXi-X10SRH-CF in ha-datacenter is powered on
2021-05-15T13:10:27.080Z info hostd[AE81B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013/archiveteam-warrior-v3-20171013.vmx] State Transition (VM_STATE_POWERING_ON -> VM_STATE_ON)

The loggoff of the vim-cmd that tried to power on the moved virtual machine. It ends when the message got the default answer, and the virtual machine fully powered up:

2021-05-15T13:10:27.116Z info hostd[AE81B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/6097a4eb-6c6e4c76-46d3-0cc47aaa9742/base/archiveteam/archiveteam-warrior-v3-20171013/archiveteam-warrior-v3-20171013.vmx] Skip a duplicate transition to: VM_STATE_ON
2021-05-15T13:10:27.116Z verbose hostd[AE81B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: runtime.featureRequirement, 25. Sent notification immediately.
2021-05-15T13:10:27.123Z verbose hostd[AE81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-5d23 user=root] RecordOp ASSIGN: info, haTask--vim.ServiceInstance.retrieveContent-130896490. Applied change to temp map.
2021-05-15T13:10:27.123Z verbose hostd[AE81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-5d23 user=root] RecordOp ASSIGN: info, haTask--vim.ServiceInstance.retrieveContent-130896490. Applied change to temp map.
2021-05-15T13:10:27.125Z verbose hostd[AE81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-5d24 user=root] RecordOp ASSIGN: info, haTask--vim.SessionManager.logout-130896491. Applied change to temp map.
2021-05-15T13:10:27.125Z verbose hostd[AE81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-5d24 user=root] RecordOp REMOVE: sessionList["52165943-fe83-da06-513a-6c2c38b0e470"], ha-sessionmgr. Applied change to temp map.
2021-05-15T13:10:27.125Z verbose hostd[AE81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-5d24 user=root] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-15T13:10:27.125Z info hostd[AE81B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-8f-5d24 user=root] Event 771 : User root@127.0.0.1 logged out (login time: Saturday, 15 May, 2021 13:06:26, number of API invocations: 0, user agent: VMware-client/6.5.0)
2021-05-15T13:10:27.125Z verbose hostd[AE81B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-8f-5d24 user=root] RecordOp ASSIGN: info, haTask--vim.SessionManager.logout-130896491. Applied change to temp map.

Back to answering the question

Let’s first repeat the help:

# vim-cmd help vmsvc/message
Usage: message vmid [messageId] [messageChoice]

Replies to the current pending question on the vm.

Given this message for virtual machine having a vmid of 25 awaiting an answer to messageId number 6233874:

# vim-cmd vmsvc/message 25
Virtual machine message 6233874:
This virtual machine might have been moved or copied. In order to configure certain management and networking features, VMware ESX needs to know if this virtual machine was moved or copied. If you don't know, answer "I Copied It". 
   0. button.uuid.cancel (Cancel)
   1. button.uuid.movedTheVM (I Moved It)
   2. button.uuid.copiedTheVM (I Copied It) [default]

You can answer the question in a separate console/ssh session with either of these:

Cancel: vim-cmd vmsvc/message 25 6233874 0 where 25 is the vmid of the virtual machine and 6233874 is the messageId of the message.
- The Virtual Machine will not power on and the power.on command fails:
```
# vim-cmd vmsvc/power.on 25
Powering on VM:
Power on fail
```
The virtual machine was moved vim-cmd vmsvc/message 25 6233874 1 where 25 is the vmid of the virtual machine and 6233874 is the messageId of the message.
- The Virtual Machine will power on and the power.on command succeeds:
```
# vim-cmd vmsvc/power.on 25
Powering on VM:
```
- The .vmx file of the virtual machine will change these fields:
  - uuid.location
  - cleanShutdown (for running virtual machines, this always has value "FALSE"; for cleanly shutdown machines the value will be "TRUE")
  - sched.swap.derivedName with a new value for the .vswp file.
  - softPowerOff(for running virtual machines, this always has the value "FALSE"; for cleanly shutdown machines the value will be "TRUE")
2, or wait for more than four (4) minutes:
- The Virtual Machine will power on, and the power.on command succeeds::
```
# vim-cmd vmsvc/power.on 25
Powering on VM:
```
- The .vmx file of the virtual machine will change these fields (you can see this below the signature or in this gist [Archive.is]):
  - uuid.location
  - uuid.bios (which for Windows system will often require a new license key)
  - ethernet0.generatedAddress
  - cleanShutdown (for running virtual machines, this always has value "FALSE"; for cleanly shutdown machines the value will be "TRUE")
  - sched.swap.derivedName with a new value for the .vswp file.
  - softPowerOff(for running virtual machines, this always has the value "FALSE"; for cleanly shutdown machines the value will be "TRUE")

Generating the “answer” command

I prefer to answer these questions manually, but use this fragment to generate an answer command that selects the “Moved” answer:

    message=`vim-cmd vmsvc/message ${vmid}`
    echo "message: $message"
    messageId=`echo "$message" | sed -n -E -e 's/^Virtual machine message (.*):$/\1/p'`
    movedAnswerId=`echo "$message" | sed -n -E -e 's/^ +([[:digit:]])\. button.uuid.* Moved.*$/\1/p'`
    if [[ "$movedAnswerId" == 1 ]]; then
        echo "vim-cmd vmsvc/message $vmid $messageId $movedAnswerId"
    fi

It is based on these regular expression experiments:

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: storing an ISO 8601 time-stamped backup tarball locally

Posted by jpluimers on 2022/01/25

In Determining the ESXi installation type (2014558) | VMware KB, I also showed how to backup the configuration and download it.

Sometimes you want an ISO 8601 time-stamped local tarball just in case you want to revert to it at a later stage.

First a small recap on how to get the tarball, download location and temporary location in the first place (it will be automatically deleted from the temporary location):

# vim-cmd hostsvc/firmware/sync_config
# vim-cmd hostsvc/firmware/backup_config
Bundle can be downloaded at : http://*/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz
# find /scratch/downloads/ -name *.tgz
/scratch/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz

Goal is to get the download filename and save it to a different folder and embed the ISO 8601 timestamp in the filename.

Like many scripts, sed and regular expressions come to the rescue once more, just like in ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes (which we will need anyway because of the ISO 8601 time stamp, and a bit of fiddling at regex101.com/r/NyrzKF

# SCRATCH_CONFIG_BUNDLE_NAME=$(vim-cmd hostsvc/firmware/backup_config | sed -n -E -e "s/^(Bundle can be downloaded at : http://*)(/downloads/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}/configBundle-.+?)(.tgz)$//scratch23/p")
# echo "SCRATCH_CONFIG_BUNDLE_NAME: '${SCRATCH_CONFIG_BUNDLE_NAME}'"
SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz'
# date --utc -I'seconds' --reference "${SCRATCH_CONFIG_BUNDLE_NAME}"
2021-05-09T17:44:42UTC

Note:

In the sed regular, expression, [[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12} matches a GUID.
In the date command, the --reference parameter must be last.

If you get the error below, then you ran too many backup_config commands in succession:

(vim.fault.TooManyWrites) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = 
   msg = "Received SOAP response fault from []: syncConfiguration
fault.TooManyWrites.summary"
}

You can see this in the hostd.log, which on my system is in /scratch/log/hostd.log where it says I can retry in 2031 seconds (slightly more than half an hour):

2021-05-09T19:34:13.420Z verbose hostd[A703B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab3] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
2021-05-09T19:34:13.420Z info hostd[A703B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-27-2ab3] Event 196 : User root@127.0.0.1 logged in as VMware-client/6.5.0
2021-05-09T19:34:13.420Z verbose hostd[A703B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab3] RecordOp ADD: sessionList["52f6d64e-0a35-c1d7-de97-624d234bc2a7"], ha-sessionmgr. Applied change to temp map.
2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-27-2ab5 user=root] Task Created : haTask--vim.host.FirmwareSystem.syncConfiguration-114207804
2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ADD: recentTask["haTask--vim.host.FirmwareSystem.syncConfiguration-114207804"], ha-taskmgr. Applied change to temp map.
2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ASSIGN: info, haTask--vim.host.FirmwareSystem.syncConfiguration-114207804. Applied change to temp map.
2021-05-09T19:34:13.423Z error hostd[9AC1B70] [Originator@6876 sub=Hostsvc.FirmwareSystem opID=vim-cmd-27-2ab5 user=root] Failed to sync configuration. Too many writes. Next sync possible in 2031 sec.
2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Default opID=vim-cmd-27-2ab5 user=root] AdapterServer caught exception: vim.fault.TooManyWrites
2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-27-2ab5 user=root] Task Completed : haTask--vim.host.FirmwareSystem.syncConfiguration-114207804 Status error
2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ASSIGN: info, haTask--vim.host.FirmwareSystem.syncConfiguration-114207804. Applied change to temp map.
2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Activation [N5Vmomi10ActivationE:0x0987e6e0] : Invoke done [syncConfiguration] on [vim.host.FirmwareSystem:ha-firmwareSystem]
2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Throw vim.fault.TooManyWrites
2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Result:
--> (vim.fault.TooManyWrites) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = 
-->    msg = ""
--> }

Not few people have bumped into this, the only other I could find through [Wayback] “vim.fault.TooManyWrites” “syncConfiguration” – Google Search is [Archive.is] mal wieder purple Screen – VMware-Forum.

Figuring out the various parts of the SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz' is like at regex101.com/r/J4yU72, regex101.com/r/uID9xs and regex101.com/r/o8a4Am:

CONFIG_BUNDLE_DIRECTORY_NAME=$(echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\1/p")
CONFIG_BUNDLE_FILE_NAME=$(     echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\2/p")
CONFIG_BUNDLE_DOT_EXTENSION=$( echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\3/p")
echo "CONFIG_BUNDLE_DIRECTORY_NAME: '${CONFIG_BUNDLE_DIRECTORY_NAME}'"
echo "CONFIG_BUNDLE_FILE_NAME:      '${CONFIG_BUNDLE_FILE_NAME}'"
echo "CONFIG_BUNDLE_DOT_EXTENSION:  '${CONFIG_BUNDLE_DOT_EXTENSION}'"

Output is like this:

SCRATCH_CONFIG_BUNDLE_NAME:   '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/configBundle-ESXi-X10SRH-CF.tgz'
CONFIG_BUNDLE_DIRECTORY_NAME: '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/'
CONFIG_BUNDLE_FILE_NAME:      'configBundle-ESXi-X10SRH-CF'
CONFIG_BUNDLE_DOT_EXTENSION:  '.tgz'

Full backup-config-to-ESXi_configuration_backup-directory.sh script:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, cron/crontab, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: getting and setting the host name, domain and fqdn

Posted by jpluimers on 2022/01/19

A few links and notes:

[Wayback] Changing the hostname of an ESX or ESXi host (1010821)
Run these commands to change the hostname in ESXi 5.x, ESXi 6.x,ESXi 7.x, using the command line:
- esxcli system hostname set –host=hostname
- esxcli system hostname set –fqdn=fqdn
[Wayback] ESX Host appears as localhost.localdomain in VMware Infrastructure/vSphere client (2009720)

Cause

The name resolution parameters were not properly configured during the installation of the ESX host.
[Wayback] Domain repoint for embedded vCenter Server fails with error: “domain_consolidator Failed to set machine id” (71020)

This issue is caused by a mismatch between the FQDN that was configured as the PNID during the vCenter Server deployment and the hostname that is currently configured.

I had a mismatch happen because of the second entry: a host configured in a different domain than it was deployed to.

Here are the commands to list and change the hosts name, domain and fqdn:

Read the rest of this entry »

Posted in Software Development, Development, Power User, *nix, VMware, Scripting, VMware ESXi, *nix-tools, Virtualization, ESXi6, ESXi6.5, ESXi6.7, ESXi7, ash/dash development, ash/dash | Leave a Comment »

ESXi: persistent files you can edit to apply settings during boot

Posted by jpluimers on 2022/01/18

Since ESXi boots from RAM, most files in /etc are not persisted after modification.

The files that are persisted, are only persisted once every hour by auto-backup.sh, so better run auto-backup.sh by hand if you want to reboot after changing them.

The auto-backup.sh script is ran every hour at 1 minute past the hour as per below crontab.

Default ESXi crontab in /var/spool/cron/crontabs/root:
#min hour day mon dow command
1    1    *   *   *   /sbin/tmpwatch.py
1    *    *   *   *   /sbin/auto-backup.sh
0    *    *   *   *   /usr/lib/vmware/vmksummary/log-heartbeat.py
*/5  *    *   *   *   /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
00   1    *   *   *   localcli storage core device purge
Schedules deciphered via [Wayback] Crontab.guru – The cron schedule expression editor:
[Wayback] Every day at 01:01: “This module removes stale temporary files”
1    1    *   *   *   /sbin/tmpwatch.py
[Wayback] Every hour at *:01: saves backup to /bootbank/state.tgz.
1    *    *   *   *   /sbin/auto-backup.sh
[Wayback] Every hour at *:00 logs heartbeat messages to /var/log/vmksummary.log like 2021-02-23T19:00:02Z heartbeat: up 577d2h37m16s, 9 VMs; [[2802426 vmx 4194304kB] [6176344 vmx 4194304kB] [68997 vmx 8388608kB]] [[2802426 vmx 0%max] [6176344 vmx 0%max] [68997 vmx 0%max]]
0    *    *   *   *   /usr/lib/vmware/vmksummary/log-heartbeat.py
[Wayback] Every 5th minute logs to /var/log/hostd-probe.log.
*/5  *    *   *   *   /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
[Wayback] Every day at 01:00: Removes storage devices which have not been seen in some time interval.
00   1    *   *   *   localcli storage core device purge
Note that localcli commands are the same as esxcli; for esxcli, a running hostd is required; localcli can run without hostd. See:

[Wayback] VCDX #200 Blog of one VMware Infrastructure Designer: ESXi localcli

[Wayback/Archive.is] Overview of vSphere Command-Line Interfaces – Getting Started with vSphere Command-Line Interfaces – VMware {code}

localcli commands

Set of commands for use with VMware Technical Support. localcli commands are equivalent to ESXCLI commands, but bypass the hoste daemon ( hostd). The localcli commands are only for situations when hostd is unavailable and cannot be restarted. After you run a localcli command, you must restart hostd. Run ESXCLI commands after the restart.

Warning: If you use a localcli command, an inconsistent system state and potential failure can result.

[Wayback] VMware ESXi 6.5 – ESXCLI Command Reference | virten.net

–jeroen

Posted in Software Development, Development, Power User, *nix, VMware, Scripting, *nix-tools, Virtualization, ESXi6, ESXi6.5, ESXi6.7, ESXi7, ash/dash development, ash/dash | Leave a Comment »

ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes

Posted by jpluimers on 2022/01/17

On VMware ESXi, with the Busybox ash/dash shell, I wanted to get the current UTC timestamp in ISO 8601 format without dashes (-) or especially colons (:) and plus-signs (+) you have to back-slash escape colons or double quote parameters, which is often can be a pain).

[Wayback] How to Manipulate Filenames Having Spaces and Special Characters in Linux describes the pain handling special characters:

In this article, we will see how to create, copy, move and delete filenames that starts with spaces and special characters (say #, *, &, =, etc.) in Linux.
[Wayback] filenames – What characters are safe in cross-platform file names for Linux, Windows and OS-X – Super User
…
- []()^ #%&!@:+={}'~ and [`] all have special meanings in many shells, and are annoying to work around, and so should be avoided. They also tend to look horrible in URLs.
…

This is why we can’t have good things: Getting the UTC 8610 timestamp was far less easy than I hoped for.

First of all, Busybox only allows for a precision of seconds, not milliseconds, and the specification format needs better documentation as per [Wayback] embedded linux – How to get ISO8601 seconds format from “date” in busybox? – Stack Overflow:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

« Previous Entries

	Mike Verhagen on Productivity tips by Florian H…
	Productivity tips by… on media.ccc.de – No, we wo…
	jpluimers on linux – How can I find a…
	jpluimers on The Windows key has no Unicode…
	sglienke on DUnit with FastMM: Detecting m…

The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

Twitter Updates

My Flickr Stream

Pages

All categories

Email Subscription

Archive for the ‘Scripting’ Category

ESXi: on the console/ssh, when a moved VM pauses during power-on: show which VMs have messages waiting, then answer them

Messages I know either equal “`No message`” or are about “`This virtual machine may have been moved or copied.`“

Intermezzo: `/var/log/hostd.log` extract when not answering the “moved” question

Back to answering the question

Generating the “answer” command

ESXi: storing an ISO 8601 time-stamped backup tarball locally

ESXi: getting and setting the host name, domain and fqdn

Cause

ESXi: persistent files you can edit to apply settings during boot

ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes

The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

Twitter Updates

My Flickr Stream

Pages

All categories

Email Subscription

Archive for the ‘Scripting’ Category

ESXi: on the console/ssh, when a moved VM pauses during power-on: show which VMs have messages waiting, then answer them

Messages I know either equal “No message” or are about “This virtual machine may have been moved or copied.“

Intermezzo: /var/log/hostd.log extract when not answering the “moved” question

Back to answering the question

Generating the “answer” command

Rate this:

Share this:

Like this:

ESXi: storing an ISO 8601 time-stamped backup tarball locally

Rate this:

Share this:

Like this:

ESXi: getting and setting the host name, domain and fqdn

Cause

Rate this:

Share this:

Like this:

ESXi: persistent files you can edit to apply settings during boot

Rate this:

Share this:

Like this:

ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes

Rate this:

Share this:

Like this:

Messages I know either equal “`No message`” or are about “`This virtual machine may have been moved or copied.`“

Intermezzo: `/var/log/hostd.log` extract when not answering the “moved” question