Run ESXi from a USB Flash Drive: A How-To-Guide
Posted by jpluimers on 2022/04/13
Guessing the [Wayback] Run ESXi from a USB Flash Drive: A How-To-Guide by just the abstract does not show the value enough:
A USB flash drive can be used not only for installation media – you can also run ESXi from USB flash drives or SD flash cards and boot from these devices.
In fact, the article shows way more, including:
- how the partitions on USB/SD devices are built as compared to HDD devices, and how they even differ depending on USB/SD sizes
- how to backup/restore the USB/SD boot devices (so you can stock them in case of failure)
This is very important, because every now and then, these USB and SD devices fail (see for instance [Wayback] Solved: Remount boot filesystem on a running system. – VMware Technology Network VMTN), so knowing what to do then is key and helps handling errors like this one:
Lost connectivity to the device mpx.vmhba32:C0:T0:L0 backing the boot filesystem /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0. As a result, host configuration changes will not be saved to persistent storage.
A every useful article for my link archive!
Related: ESXi: storing an ISO 8601 time-stamped backup tarball locally
Similar errors via [Wayback] satp_local_updatepath – Google Search and [Wayback] save configuration of vmhba32 – Google Search:
- [Wayback] ESXI 6.5 management services hangs after a couple … – VMware Technology Network VMTN
We have used a USB key to install the OS of the hypervisor (instead of the Dual SD Card Module – we don’t have it installed) in each of the servers.
After a couple of days, the same hypervisor is hanging partially. We detect the problem because when we are going to restart a VM, make a VM shutdown or start, or make some operations (as VM Snapshot creation), the task does not complete. – Vsphere HTML5 interfaces shown the task (for example, register VM, at 0%, 10% or a very slow process that completely hangs at 99%). We can’t cancel the task, it appears random error message like “operation can not be completed” or directly we can’t make click in the cancel icon.
We login in the hypervisor web client directly, and when we try to get the VM inventory (only in that tab), we get a very slow response (normally clients disconnects with “connection lost”) We have in that hypervisor Veeam B&R in 9.5 U4b version to make the backups of the VMs. For example, today after 35 days we have received alerts that says us backup process can not be completed. The VM that failed was powered on (the guest os worked perfectly) but the vm appears as invalid in the VM list. We have shutdown it, we have remove it from inventory, but when we tried to register again, it has been impossible. More than 60 minutes and the task hangs at 99%.
We have connected by SSH to investigate the problem, and we have shown this in logs.
2020-03-14T05:41:41.184Z cpu4:611239)VMW_SATP_LOCAL: satp_local_updatePath:836: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry 2020-03-14T05:41:46.356Z cpu2:65587)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update... 2020-03-14T05:41:46.356Z cpu2:65587)ScsiDeviceIO: 3015: Cmd(0x439510b358c0) 0x28, CmdSN 0x1 from world 67214 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x 2020-03-14T05:42:03.185Z cpu10:65885)ScsiPath: 5149: Command 0x0 (cmdSN 0x0, World 0) to path vmhba32:C0:T0:L0 timed out: expiry time occurs 1002ms in the past 2020-03-14T05:42:03.185Z cpu10:65885)VMW_SATP_LOCAL: satp_local_updatePath:836: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retryesxcli storage core device list -d mpx.vmhba32:C0:T0:L0times out and fails.Similar log extracts from:
vmkwarning.log
VMkernel.log
syslog.log
vmksummary.log
vmkeventd.log
- [Wayback/Archive.is] SD card issues after updating to 6.7 update 1 : vmware
Yes i also had problems. Had to use the legacy usb drivers instead of the vmkusb drivers when going to 6.7 . Used a kernel boot parameter to make it work. Check last page of this thread https://communities.vmware.com/thread/607208
- [Wayback/Archive.is] SSD Device presented as a SAN LUN is claimed by an incorrect SATP VMW_SATP_LOCAL instead of the correct SATP VMW_SATP_ALUA in ESXi 5.x (2041905)
- [Wayback/Archive.is] Bootbank cannot be found at path ‘/bootbank’ and boot device is in APD state (70788)
This indicates that ESXi vmkernel is unable to communicate with the USB device due to hardware(adapter/device).
Contact your server platform vendor to investigate the hardware(adapter/device).
As a workaround reboot the ESXi host so that it will use the RAM but configuration will not be saved as ESXi host does not have its bootbank.
NOTE: Please have a backup of ESXi configurations. In case if ESXi host doesn’t come up after reboot.
- [Wayback] Lost connectivity to the device mpx.vmhba32 ERROR –
I shut it down and took out the SD card riser board and reseated the SD cards and reseated them, Dell have even made a video of how to do this task!
- [Wayback] Lost connectivity to the device backing the boot filesystem – Virtually Boring
We evacuated the host and reseated the SD Card. It mounted without issue but to be safe we went ahead and installed a replacement SD Card. Issue reoccurred. This is caused by the version of iLo running on the server.
- [Wayback] [SOLVED] VMware ESXi error – Spiceworks
Actually, when ESXi initially boots, it loads itself into the RAM on the server. This is where it performs most of it’s tasks regarding resource allocation and management. When something changes, it then writes those changes back to the ‘persistent’ memory (boot device like SD or USB) using a cron job.
So you can make changes and even though the server itself will run, if you reboot then you may very well lose those changes that you made.
It is better to address an issue like this as soon as you see it to insure that you have system integrity after a reboot or change.
–jeroen
The Wiert Corner - irregular stream of stuff 
Leave a Reply