The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,392 other followers

Alternatives to VMware ESXi: working around “[Errno 28] No space left on device” when updating (especially when booting from USB-stick)

Posted by jpluimers on 2021/05/07

Yesterday I talked about VMware ESXi: working around “[Errno 28] No space left on device” when updating (especially when booting from USB-stick).

There are some alternative workarounds mentioned on the interwebz. Below are a the ones I found. I discuss which ones won’t work, and why I dislike others.

Alternative workarounds that failed

Configuring host-swap

This was suggested by:

Host swap was already configured, and it still failed.

Just in case you ever want to configure host swap, it is under an URL like https://esxi67.example.org/ui/#/host/manage/system/swap and looks like this:

ESXi 6.7: configuring host swap

ESXi 6.7: configuring host swap

You get there by:

  1. logging on to the web UI
  2. clicking Host
  3. clicking Manage under Host
  4. clicking Swap under the System tab
  5. clicking Edit settings when you want to change them
    ESXi 6.7: edit host swap settings

    ESXi 6.7: edit host swap settings

More information about host swap:

Alternative workarounds I like less

Below are a few alternative workarounds. I will include them as they gained me more knowledge, but I will also describe why I like them less.

  • [Wayback] ESXI 6.7 update: No space left on device | eknori.de after explaining that directing the swap space to a datastore fails, also mentions alternative this:

    Unfortunately, in this situation, host swap already was enabled.

    There is though, a workaround. You can use an image that doesn’t have the tools vib included with this command:

    esxcli software profile update -p ESXi-6.7.0-20190802001-no-tools -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

    You can then manually install the troublesome vib (if you have a need for tools) with this command:

    esxcli software vib install -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_10.3.10.12406962-14141615.vib

    I had to edit it as the post itself shows the filename as

    vmw-depot
     -index.xml

    Yup: bitching again, as markup issues make code unreliable. It also allows me to explain why I do not like the solution, which is because of two reasons:

    1. It doesn’t explain why this solution works and if it is future proof. Does a future upgrade that includes changed VMware_locker_tools-light also fail? If it does not fail, does it update the VMware_locker_tools-light?
    2. It does not explain how to get the path of https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_10.3.10.12406962-14141615.vib.  I did some mor research on this, and it is actually pretty straightforward: the [Wayback] VMware ESXi 6.7 Patch History has it in the table
      ESXi-6.7.0-20210304001-standard patch table

      ESXi-6.7.0-20210304001-standard patch table

      The “Version” link for “tools-light” [Wayback] 11.2.5.17337674-17700514 actually links to the VMware_locker_tools-light_11.1.1.16303738-16701467.vib file.

  • [Wayback] ESXi 6.7.0 – [Errno 28] No space left on devicevibs = VMware_locker_tools-light_11.1.1.16303738-16701467 does not explain where to get the VMware_locker_tools-light_11.1.1.16303738-16701467.vib link from, does not have the code formatted as such (so I did that below), but does actually answers part of the above questions, but not if a future upgrade will also fail. In short: re-running the upgrade after manually installing the VMware_locker_tools-light_11.1.1.16303738-16701467.vib will succeed:

    Unfortunately swap was already enabled to I had to manually install the tools-light with this command:

    esxcli software vib install -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_11.1.1.16303738-16701467.vib

    Then re-ran the upgrade and it was successful.

  • [Wayback] ESXi 6 Update error – No Space left on device /locker which suggests to delete “find all big files in /locker and remove it”. I think that is a bad idea, as the /locker directory is maintained by your ESXi system and you should not remove any big file without knowing if it is relied upon by ESXi.
  • While updating VMware ESXi servers, VMware vSphere users may encounter the “No space left on device” error that pops up while executing “esxcli software vib update” command. Interestingly, the problem occurs even though disks are doing well and have enough free space and df -h command proves that.[Wayback] No Space Left on Device? Updating VMware ESXi | StarWind Blog has bad code markup, but explains
    • how to get disk usage with df -h where the vfat volumes usually indicate the ones on USB or SD-card media.
    • that hardly the number of inodes is a problem, and that stat -f / can help you figure out if that is the case on the volume where the upgrade files are stored
    • how to find large files not in data stores; I have changed added -h to the ls command so it becomes human readable:
      find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -lh '{}' \;
    • suggests how to put the swap space on a data store (which doesn’t work on ESXi 6.7 systems any more)
  •  

    [Wayback] Intel NUC Kit NUC5i3RYH met ESXi 6.0 updaten naar 6.7 – Gahan Zwart’s Blog

    • I like the upgrade copying the ISO to an USB stick with Rufus
    • I do not like the VMware_locker_tools-light... intermediate step, as the last step (download the full depot to a datastore, then update from there)
    • ESXi 7.0 has the same ErrNo 28 update problem as ESXi 6.7 and 6.5, so I will default to the depot download.

–jeroen

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

 
%d bloggers like this: