The Wiert Corner – irregular stream of stuff

Whenever I get time to dig into the actual cause (a reboot fixed it; I do not like problems disappearing by themselves).

I got these errors:

1. cannot be live installed
2. No space left on device

A reboot resolved the first, which might be caused by the /bootbank pointing to /tmp:

[root@ESXi-X9SRI-F:~] ls -la /bootbank
lrwxrwxrwx    1 root     root             4 May 20 08:44 /bootbank -> /tmp
[root@ESXi-X9SRI-F:~] ls -la /tmp
total 36
drwxrwxrwt    1 root     root           512 Jun 23 06:13 .
drwxr-xr-x    1 root     root           512 Jun 17 13:13 ..
-rw-------    1 root     root            40 Jun 23 06:15 probe.session
-rw-r--r--    1 root     root         14188 Jun 23 06:01 state.tgz
drwx------    1 root     root           512 Jun 23 04:44 vmware-root
[root@ESXi-X9SRI-F:~] ls -la /tmp/vmware-root/
total 8
drwx------    1 root     root           512 Jun 23 04:44 .
drwxrwxrwt    1 root     root           512 Jun 23 06:13 ..
[root@ESXi-X9SRI-F:~] cd ..

Normally, this points to a valid filesystem:

[root@ESXi-X9SRI-F:~] ls -la /bootbank
lrwxrwxrwx    1 root     root            49 Jun 23 06:29 /bootbank -> /vmfs/volumes/69ab34ee-f2e2bd4a-e013-a62e6975b9e7
[root@ESXi-X9SRI-F:~] du -h `readlink /bootbank`
146.7M  /vmfs/volumes/69ab34ee-f2e2bd4a-e013-a62e6975b9e7

Research links

• Cannot be live installed: finding boot device (in practice a df -h is easier if you just have one USB stick to boot from)
  • [WayBack] Virtual Admin: How to Get ESXi Boot Device
  • [WayBack] Solved: Detect if ESXi installed on internal SD… |VMware Communities
• No space left on device (combined in the above search with “ESXi” and “profile update”)
  • [WayBack] ESXI 6.7 profile update: No space left on device 🌐:aligrant.com
  • [WayBack] Fresh ESXi 6.7 install; Error 28 No space left issue during latest update | ServeTheHome and ServeThe.Biz Forums
    • [WayBack] No Space Left On Device vmware_locker_tools error when upgrading ESXI 6.7 – Ganon.Club
  • [WayBack] ESXi/ESX error: No free space left on device
  • [WayBack] Solved: Vmware Esxi Update No Space Left On Dev… |VMware Communities
  • [WayBack] Solved: Failing while doing Esx 6.5 patch updat… |VMware Communities
    • [WayBack] VMWARE: Cannot update host with ESXCLI due to error OSError: [Errno 28] No space left on device – martinsblog.dk (choose a non-USB SWAP device)

This was while applying patches using the [WayBack] VMware ESXi 6.7.0 Patch History pop-ups:

• ESXi-6.7.0-20190604001-standard
  

  # Cut and paste these commands into an ESXi shell to update your host with this Imageprofile
  # See the Help page for more instructions
  #
  esxcli network firewall ruleset set -e true -r httpClient
  esxcli software profile update -p ESXi-6.7.0-20190604001-standard \
  -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
  esxcli network firewall ruleset set -e false -r httpClient
  #
  # Reboot to complete the upgrade

• ESXi-6.7.0-20190504001-standard
  

  # Cut and paste these commands into an ESXi shell to update your host with this Imageprofile
  # See the Help page for more instructions
  #
  esxcli network firewall ruleset set -e true -r httpClient
  esxcli software profile update -p ESXi-6.7.0-20190504001-standard \
  -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
  esxcli network firewall ruleset set -e false -r httpClient
  #
  # Reboot to complete the upgrade

The swap settings were exactly the same:

• Working
  

  

• Failing
  

  

The USB devices were the same too:

• Working
  

  

• Failing
  

  

The failing system had even more partition space than the working system:

Working	Failing

So was quite baffled that pointing the swap space pointing to a datastore solved the problem:

• Working
  

  

• Working too
  

  

You can also request this from the shell (via [WayBack] VMware vSphere 5.1):

[root@ESXi-X9SRI-F:~] esxcli sched swap system get
Datastore Active: true
Datastore Enabled: true
Datastore Name: QVO860_960GB
Datastore Order: 1
Hostcache Active: false
Hostcache Enabled: true
Hostcache Order: 0
Hostlocalswap Active: false
Hostlocalswap Enabled: true
Hostlocalswap Order: 2

Empty bootbank

Quoted in full as it cannot be archived in the WayBack machine nor Archive.is: Bootbanks mounts to /tmp folder and configuration changes are lost after rebooting of ESXi 6.0 Server (2148321)

Last Updated: 1/31/2019Categories: Troubleshooting

 Symptoms

• Configuration changes on the ESXi is lost after rebooting the server.
• Bootbank and altbootbank points to /tmp folder instead of pointing to the ESXi installed partition (this can be confirmed by running ls -l on root (“/”) folder)

 Cause

This issue happens if there is a delay in Fabric Login during the ESXi Boot process. By default, there is 3 Second timeout for Fabric login.

If Fabric Login takes more than 3 seconds, claiming the devices and discovering the LUNs happens parallelly and some LUNs will not be claimed on time during the boot process. If the boot LUN is not claimed before mounting the bootbank, ESXi boot process will mount bootbank and altbootbank to /tmp.

 Resolution

VMware introduced a new boot configuration parameter on ESXi 6.0 Patch 4 to handle the delay in Fabric logins.

To resolve this issue, follow the steps:

1. To mount the bootbank to proper LUN, run this command on ESXi:
   localcli –plugin-dir=/usr/lib/vmware/esxcli/int/ boot system restore –bootbanks
2. Upgrade ESXi to Patch 4 (Build number 4600944) available VMware Downloads.
3. Connect to the Server KVM Console.
4. During the ESXi boot process, press Shift+O and append the boot parameter devListStabilityCount=10 (its 10 Seconds, can be different value depending on flogi delay in the environment).
5. After the boot process, append the devListStabilityCount=10 entry in the /bootbank/boot.cfg file.Note: This step is to ensure that change is persistent.

 Related Information

Following log from ESXi boot, HBA Link is UP and it took more than 3 seconds for the Fabric login completion (flogi succeeded) and Claim rules are enabled after 3rd Second :

2016-05-13T11:07:45.610Z cpu24:33363)<7>fnic : 1 :: link up

2016-05-13T11:07:45.610Z cpu24:33363)<6>host1: libfc: Link up on port ( 0)

2016-05-13T11:07:45.752Z cpu15:33365)<7>fnic : 2 :: link up

2016-05-13T11:07:45.752Z cpu15:33365)<6>host2: libfc: Link up on port ( 0)

2016-05-13T11:07:46.754Z cpu0:33369)<6>usb 1-1: suspended

2016-05-13T11:07:46.754Z cpu0:33369)<6>usb 1-1: suspended

2016-05-13T11:07:48.019Z cpu0:33315)ScsiClaimrule: 2463: Enabling claimrules for MPplugins.

2016-05-13T11:07:49.465Z cpu0:33351)<6>usb usb1: suspended

2016-05-13T11:07:49.465Z cpu0:33351)<6>usb usb1: suspended

2016-05-13T11:07:49.620Z cpu26:33353)<6>host1: Assigned Port ID 16369

2016-05-13T11:07:49.620Z cpu26:33353)<7>fnic : 1 :: set port_id 16369 fp 0x439e41ccebc8

2016-05-13T11:07:49.620Z cpu26:33353)<6>host1: fip: received FLOGI LS_ACC using non-FIP mode

2016-05-13T11:07:49.620Z cpu26:33353)<7>fnic : 1 :: update_mac 0x439e41ccec41M

2016-05-13T11:07:49.620Z cpu26:33353)<7>fnic : 1 :: FLOGI reg issued fcid 16369 map 0 dest 0x43915249bcfaM

2016-05-13T11:07:49.622Z cpu11:33158)<7>fnic : 1 :: flog reg succeeded

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Build numbers and versions of VMware ESXi/ESX

—jeroen

https://gist.github.com/jpluimers/7ff32a56e3463d8e63cc05c76680977d