The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 4,262 other subscribers

Archive for May 7th, 2021

Alternatives to VMware ESXi: working around “[Errno 28] No space left on device” when updating (especially when booting from USB-stick)

Posted by jpluimers on 2021/05/07

Yesterday I talked about VMware ESXi: working around “[Errno 28] No space left on device” when updating (especially when booting from USB-stick).

There are some alternative workarounds mentioned on the interwebz. Below are a the ones I found. I discuss which ones won’t work, and why I dislike others.

Alternative workarounds that failed

Configuring host-swap

This was suggested by:

Host swap was already configured, and it still failed.

Just in case you ever want to configure host swap, it is under an URL like https://esxi67.example.org/ui/#/host/manage/system/swap and looks like this:

ESXi 6.7: configuring host swap

ESXi 6.7: configuring host swap

You get there by:

  1. logging on to the web UI
  2. clicking Host
  3. clicking Manage under Host
  4. clicking Swap under the System tab
  5. clicking Edit settings when you want to change them
    ESXi 6.7: edit host swap settings

    ESXi 6.7: edit host swap settings

More information about host swap:

Alternative workarounds I like less

Below are a few alternative workarounds. I will include them as they gained me more knowledge, but I will also describe why I like them less.

  • [Wayback] ESXI 6.7 update: No space left on device | eknori.de after explaining that directing the swap space to a datastore fails, also mentions alternative this:

    Unfortunately, in this situation, host swap already was enabled.

    There is though, a workaround. You can use an image that doesn’t have the tools vib included with this command:

    esxcli software profile update -p ESXi-6.7.0-20190802001-no-tools -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

    You can then manually install the troublesome vib (if you have a need for tools) with this command:

    esxcli software vib install -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_10.3.10.12406962-14141615.vib

    I had to edit it as the post itself shows the filename as

    vmw-depot
 -index.xml

    Yup: bitching again, as markup issues make code unreliable. It also allows me to explain why I do not like the solution, which is because of two reasons:

    1. It doesn’t explain why this solution works and if it is future proof. Does a future upgrade that includes changed VMware_locker_tools-light also fail? If it does not fail, does it update the VMware_locker_tools-light?
    2. It does not explain how to get the path of https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_10.3.10.12406962-14141615.vib.  I did some mor research on this, and it is actually pretty straightforward: the [Wayback] VMware ESXi 6.7 Patch History has it in the table
      ESXi-6.7.0-20210304001-standard patch table

      ESXi-6.7.0-20210304001-standard patch table

      The “Version” link for “tools-light” [Wayback] 11.2.5.17337674-17700514 actually links to the VMware_locker_tools-light_11.1.1.16303738-16701467.vib file.

  • [Wayback] ESXi 6.7.0 – [Errno 28] No space left on devicevibs = VMware_locker_tools-light_11.1.1.16303738-16701467 does not explain where to get the VMware_locker_tools-light_11.1.1.16303738-16701467.vib link from, does not have the code formatted as such (so I did that below), but does actually answers part of the above questions, but not if a future upgrade will also fail. In short: re-running the upgrade after manually installing the VMware_locker_tools-light_11.1.1.16303738-16701467.vib will succeed:

    Unfortunately swap was already enabled to I had to manually install the tools-light with this command:

    esxcli software vib install -v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_tools-light_11.1.1.16303738-16701467.vib

    Then re-ran the upgrade and it was successful.

  • [Wayback] ESXi 6 Update error – No Space left on device /locker which suggests to delete “find all big files in /locker and remove it”. I think that is a bad idea, as the /locker directory is maintained by your ESXi system and you should not remove any big file without knowing if it is relied upon by ESXi.
  • While updating VMware ESXi servers, VMware vSphere users may encounter the “No space left on device” error that pops up while executing “esxcli software vib update” command. Interestingly, the problem occurs even though disks are doing well and have enough free space and df -h command proves that.[Wayback] No Space Left on Device? Updating VMware ESXi | StarWind Blog has bad code markup, but explains
    • how to get disk usage with df -h where the vfat volumes usually indicate the ones on USB or SD-card media.
    • that hardly the number of inodes is a problem, and that stat -f / can help you figure out if that is the case on the volume where the upgrade files are stored
    • how to find large files not in data stores; I have changed added -h to the ls command so it becomes human readable:
      find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -lh '{}' \;
    • suggests how to put the swap space on a data store (which doesn’t work on ESXi 6.7 systems any more)
  •  

    [Wayback] Intel NUC Kit NUC5i3RYH met ESXi 6.0 updaten naar 6.7 – Gahan Zwart’s Blog

    • I like the upgrade copying the ISO to an USB stick with Rufus
    • I do not like the VMware_locker_tools-light... intermediate step, as the last step (download the full depot to a datastore, then update from there)
    • ESXi 7.0 has the same ErrNo 28 update problem as ESXi 6.7 and 6.5, so I will default to the depot download.

–jeroen

Posted in ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

MacOS IPMIView and Supermicro X9SRi-F web interface cannot mount ISOs; on Windows, the web started Java iKVM console can

Posted by jpluimers on 2021/05/07

Screenshots so I do not forget.

Works from Windows Java iKVM Viewer v1.69.21 started via the web interface:

C:\Users\jeroenp>java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

Fails from web interface:

It does not matter if the share is iso$ or iso: both fail in the same way.

Note this was after working around this very confusion error message from the web interface:

The image path is invalid or contain /:*?"<>|

The error message is just telling that the path should start with a back-slash:

[WayBack] Supermicro IPMI and Windows Share – Server Fault

With HTML5 iKVM this seems to work fine, but that requires more recent motherboards:

[WayBack] How to map a network share to boot from ISO, moving Supermicro Java iKVM to browser-only HTML5 iKVM | TinkerTry IT @ Home

Fails on MacOs with Java iKVM Viewer v1.69 r14.

# java -version
java version "1.8.0_74"
Java(TM) SE Runtime Environment (build 1.8.0_74-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)

Gets you the rainbow circle of death and an “application not responding”

The success and failure are the same no matter the firmware; the same results were obtained both before and after upgrading the IPMI firmware:

–jeroen

PS: from Twitter

 

Posted in Hardware, IPMI, Mainboards, Power User, SuperMicro, X9SRi-F | Leave a Comment »

Need to research alternative IPMI tooling

Posted by jpluimers on 2021/05/07

Since IPMIView cannot mount ISOs and some tasks are not that convenient in GUI tools, I want to look more deeply in console based IPMI tooling.

Though my machine to remotely manage stuff is a MacOS based machine, most of my sites have Raspberry Pi devices, so some compatibility there would be nice. imputil seems to work there given IPMI firmware is recent:

Here are some names for future research:

  • impmiutil
    • [WayBack] ipmiutil – IPMI Management Utilities

      IPMI Management Utilities Project provides a series of common utilities for IPMI server management locally or via LAN. The utilities provide source and scriptable command binaries for automating server management functions.

      IPMIUTIL performs a series of common IPMI server management functions to allow administrators to perform management functions without a learning curve. It can gather FRU inventory data, SEL firmware log, sensors, watchdog, power control, health, monitoring, and has an SOL console. It can write sensor thresholds, FRU asset tags, and supports a full IPMI configuration save/restore. Ease-of-use examples with IPMIUTIL:

      • Native builds on many OSs: Linux, Windows, BSD, Solaris, MacOSX, HPUX
      • The only IPMI project natively supporting Windows drivers
      • IPMIUTIL supports any IPMI-compliant vendor firmware
      • IPMIUTIL detects and handles OEM-specific IPMI firmware variants
      • IPMIUTIL interprets various vendor OEM-specific sensor values automatically.
      • Any IPMI values not yet recognized at least return the values, rather than just ‘na’ or Unknown.
      • Shared library for custom applications, sample source included
      • IPMIUTIL BSD license is compatible with open-source or commercial use
      • Linux driverless support is ideal for boot media or embedded
      • Configuring the IPMI LAN requires just a single command
      • The decoding of IPMI SEL events includes a severity.
      • A soft-reboot can be easily performed instead of a hard reboot by using ipmiutil_asy, even if the platform does not support ACPI.
      • IPMI SEL management is automated via checksel.
      • Watchdog management is automated via ipmiutil_wdt.
    • [WayBack] homebrew-core/ipmiutil.rb at master · Homebrew/homebrew-core · GitHub
    • [WayBack] ipmiutil — Homebrew Formulae
    • [WayBack] Install ipmiutil on Mac with Brew | BrewInstall
  • ipmitool
  • freeipmi
    • [WayBack] FreeIPMI – Home

      FreeIPMI provides in-band and out-of-band IPMI software based on the IPMI v1.5/2.0 specification. The IPMI specification defines a set of interfaces for platform management and is implemented by a number vendors for system management. The features of IPMI that most users will be interested in are sensor monitoring, system event monitoring, power control, and serial-over-LAN (SOL). The FreeIPMI tools and libraries listed below should provide users with the ability to access and utilize these and many other features. A number of useful features for large HPC or cluster environments have also been implemented into FreeIPMI. See the README or FAQ for more info.

       

    • [WayBack] homebrew-core/freeipmi.rb at master · Homebrew/homebrew-core · GitHub
    • [WayBack] freeipmi — Homebrew Formulae

Things I am interested in:

  • discovering IPMI capable devices in a network
  • reading sensors
  • setting power state
  • mounting/unmounting ISO images

–jeroen

Posted in Hardware, IPMI, Mainboards, Power User, SuperMicro | 3 Comments »