Alternatives to VMware ESXi: working around “[Errno 28] No space left on device” when updating (especially when booting from USB-stick)
Posted by jpluimers on 2021/05/07
Yesterday I talked about VMware ESXi: working around “[Errno 28] No space left on device” when updating (especially when booting from USB-stick).
There are some alternative workarounds mentioned on the interwebz. Below are a the ones I found. I discuss which ones won’t work, and why I dislike others.
Alternative workarounds that failed
Configuring host-swap
This was suggested by:
- [Wayback] VMWARE: Cannot update host with ESXCLI due to error OSError: [Errno 28] No space left on device – martinsblog.dk (which – besides showing a screenshot of an older web-client than the mentioned ESXi 6.5 – does not format code as
code, so all quoting is wrong; I bitched about this in VMware ESXi 6 and 7: checking and setting/clearing maintenance mode from the console as it makes me distrust sites like this as you cannot rely on their code to work)It looks like I am bitching a lot, but that is for a reason: you need to present accurate information that works, otherwise readers like your future self pay the burden. ESXi 6.5 has the same swap settings web-ui as ESXI 6.7, but it is not configured to use a datastore by default:
ESXi 6.5 swap configuration
- Two posts from eknori.de which combined indicate changing the swap space to use a datastore does work for ESXi 6.5, but fails for ESXi 6.7:
- [Wayback] ESXI 6.7 update: No space left on device | eknori.de (which nicely mentions the proposed solution fails, see below where I discuss workarounds that do work but I still don’t like)
- [Wayback] [VMWARE ESXi] – [Errno 28] No space left on device #ibmchampion | eknori.de (which mentions that changing swap space to use a data store does work on ESXi 6.5; it also shows an actual ESXi 6.5 web-ui, but posts the code as screenshots, so you need not-so-reliable OCR to extract the actual code)
Host swap was already configured, and it still failed.
Just in case you ever want to configure host swap, it is under an URL like https://esxi67.example.org/ui/#/host/manage/system/swap and looks like this:
ESXi 6.7: configuring host swap
You get there by:
- logging on to the web UI
- clicking
Host - clicking
ManageunderHost - clicking
Swapunder theSystemtab - clicking
Editsettings when you want to change them
ESXi 6.7: edit host swap settings
More information about host swap:
- ESXi 6.7: [Wayback] About System Swap
- ESXi 6.7: [Wayback] Configure System Swap
- ESXi 7.0: [Wayback] About System Swap
- ESXi 7.0: [Wayback] Configure System Swap
Alternative workarounds I like less
Below are a few alternative workarounds. I will include them as they gained me more knowledge, but I will also describe why I like them less.
- [Wayback] ESXI 6.7 update: No space left on device | eknori.de after explaining that directing the swap space to a datastore fails, also mentions alternative this:
Unfortunately, in this situation, host swap already was enabled.
There is though, a workaround. You can use an image that doesn’t have the tools vib included with this command:
esxcli software profile update -p ESXi-6.7.0-20190802001-no-tools -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xmlYou can then manually install the troublesome vib (if you have a need for tools) with this command:
esxcli software vib install -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_10.3.10.12406962-14141615.vibI had to edit it as the post itself shows the filename as
vmw-depot -index.xml
Yup: bitching again, as markup issues make code unreliable. It also allows me to explain why I do not like the solution, which is because of two reasons:
-
It doesn’t explain why this solution works and if it is future proof. Does a future upgrade that includes changed
VMware_locker_tools-lightalso fail? If it does not fail, does it update theVMware_locker_tools-light? - It does not explain how to get the path of
https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_10.3.10.12406962-14141615.vib. I did some mor research on this, and it is actually pretty straightforward: the [Wayback] VMware ESXi 6.7 Patch History has it in the table
ESXi-6.7.0-20210304001-standard patch table
The “Version” link for “tools-light” [Wayback] 11.2.5.17337674-17700514 actually links to the
VMware_locker_tools-light_11.1.1.16303738-16701467.vibfile.
-
- [Wayback] ESXi 6.7.0 – [Errno 28] No space left on devicevibs = VMware_locker_tools-light_11.1.1.16303738-16701467 does not explain where to get the
VMware_locker_tools-light_11.1.1.16303738-16701467.viblink from, does not have the code formatted as such (so I did that below), but does actually answers part of the above questions, but not if a future upgrade will also fail. In short: re-running the upgrade after manually installing theVMware_locker_tools-light_11.1.1.16303738-16701467.vibwill succeed:
Unfortunately swap was already enabled to I had to manually install the tools-light with this command:
esxcli software vib install -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_11.1.1.16303738-16701467.vibThen re-ran the upgrade and it was successful.
- [Wayback] ESXi 6 Update error – No Space left on device /locker which suggests to delete “find all big files in
/lockerand remove it”. I think that is a bad idea, as the/lockerdirectory is maintained by your ESXi system and you should not remove any big file without knowing if it is relied upon by ESXi. - While updating VMware ESXi servers, VMware vSphere users may encounter the “No space left on device” error that pops up while executing “esxcli software vib update” command. Interestingly, the problem occurs even though disks are doing well and have enough free space and df -h command proves that.[Wayback] No Space Left on Device? Updating VMware ESXi | StarWind Blog has bad code markup, but explains
- how to get disk usage with
df -hwhere thevfatvolumes usually indicate the ones on USB or SD-card media. - that hardly the number of inodes is a problem, and that
stat -f /can help you figure out if that is the case on the volume where the upgrade files are stored - how to find large files not in data stores; I have changed added
-hto thelscommand so it becomes human readable:
find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -lh '{}' \; - suggests how to put the swap space on a data store (which doesn’t work on ESXi 6.7 systems any more)
- how to get disk usage with
-
[Wayback] Intel NUC Kit NUC5i3RYH met ESXi 6.0 updaten naar 6.7 – Gahan Zwart’s Blog
- I like the upgrade copying the ISO to an USB stick with Rufus
- I do not like the
VMware_locker_tools-light...intermediate step, as the last step (download the full depot to a datastore, then update from there) - ESXi 7.0 has the same
ErrNo 28update problem as ESXi 6.7 and 6.5, so I will default to the depot download.
–jeroen
The Wiert Corner - irregular stream of stuff 
Leave a Reply