The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘ash/dash’ Category

Some insights on how readlink approached canonicalisation of a filename having symlinks

Posted by jpluimers on 2022/03/03

Cool, I didn’t realise how readlink operated, but found out a bit more in the answers to [Wayback] symlink – How to get full path of original file of a soft symbolic link? – Unix & Linux Stack Exchange, thanks to [Wayback] daisy, [Wayback] Peter.O and [Wayback] Gilles ‘SO- stop being evil’:

  • Try this line:
    readlink -f `which command`
    

    If command is in your $PATH variable , otherwise you need to specify the path you know.

    -f will return a path to a non-existent final target, so long as the intermediate link targets exist… Use -e to avoid this, ie. -e will return null if the final target does not exist. – Peter.O

  • Under Linux, readlink reads the contents of a symlink, and readlink -f follows symlinks to symlinks to symlinks, etc., until it finds something that isn’t a symlink.

–jeroen

Posted in *nix, *nix-tools, ash/dash, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

Too bad: ESXi busybox has `diff`, but not `patch`

Posted by jpluimers on 2022/03/02

On my ESXi boxes, I have a directory with local scripts that in part depend on the machine.

So I contemplated patching the dending parts with patch.

Then I found out that the BusyBox that VMware built for ESXi does have diff, but not patch:

# $(readlink -f "`which diff`")
BusyBox v1.29.3 (2021-01-17 01:25:00 PST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list
   or: function [arguments]...

    BusyBox is a multi-call binary that combines many common Unix
    utilities into a single executable.  Most people will create a
    link to busybox for each function they wish to use and BusyBox
    will act like whatever it was invoked as.

Currently defined functions:
    addgroup, adduser, arch, ash, awk, basename, bunzip2, bzcat, bzip2, cat, chgrp, chmod, chown, chvt, cksum, clear, cp, crond,
    cut, date, dd, delgroup, deluser, diff, dirname, dnsdomainname, du, echo, egrep, eject, env, expr, false, fdisk, fgrep, find,
    fstrim, getty, grep, groups, gunzip, gzip, halt, head, hexdump, hostname, inetd, init, kill, ln, logger, login, ls, lzop,
    lzopcat, md5sum, mkdir, mkfifo, mknod, mktemp, more, mv, nohup, nslookup, od, passwd, poweroff, printf, readlink, reboot,
    reset, resize, rm, rmdir, sed, seq, setsid, sh, sha1sum, sha256sum, sha3sum, sha512sum, sleep, sort, ssl_client, stat, stty,
    sum, sync, tail, tar, taskset, tee, test, time, timeout, touch, true, uname, uniq, unlink, unlzop, unzip, usleep, vi, watch,
    wc, wget, which, who, xargs, zcat

This list is much shorter than the applets that are supported in [Wayback] BusyBox – The Swiss Army Knife of Embedded Linux, so VMware did cut out quite a few.

Generating the above output

The command-line trick above first expands diff using the output of which diff, then finds out where it links to through the readlink -f wrapper there the back-quotes “`” get this output:

# readlink -f "`which diff`"
/usr/lib/vmware/busybox/bin/busybox

Finally the $(...) executes the output of readlink.

It is based on [Wayback] bash – How to resolve symbolic links in a shell script – Stack Overflow

readlink -f "$path"

Editor’s note: The above works with GNU readlink and FreeBSD/PC-BSD/OpenBSD readlink, but not on OS X as of 10.11.GNU readlink offers additional, related options…

Need to devise a way to apply patches

Given there is no patch, I need to think about a good way to apply patches, for instance to snip this into /etc/rc.local.d/local.sh in a reliable way:

## BEGIN-PATCH-PATH

# local binaries are in /vmfs/volumes/NVMe980PRO_1TB/local-bin/
# link that directory from /opt/bin
# then add /opt/bin to the PATH in /etc/profile so that on each logon it becomes available
# this means you need to logon twice after reboot:
# - first to patch /etc/profile
# - second to have the correct PATH loaded from /etc/profile
# direcory exist trick from https://stackoverflow.com/questions/59838/how-can-i-check-if-a-directory-exists-in-a-bash-shell-script

patch_etc_profile_PATH() {
    if [ -d "$1" ]; then
      ln -s "$1" "/opt/bin"
      sed -i -e 's!PATH=/bin:/sbin!PATH=/bin:/sbin:/opt/bin/!' /etc/profile
    fi
}

patch_etc_profile_PATH /vmfs/volumes/NVMe980PRO_1TB/local-bin/

## END-PATCH-PATH

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: on the console/ssh, when a moved VM pauses during power-on: show which VMs have messages waiting, then answer them

Posted by jpluimers on 2022/01/27

First the script that display messages for all virtual machines, vim-cmd-display-messages-for-all-VMs.sh:

#!/bin/sh
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    echo "Messages for VM with id ${vmid} which has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    vim-cmd vmsvc/message ${vmid}
done
exit 0

It is very similar to vim-cmd-reload-all-VM-vmx-configurations.sh from Source: ESXi: reloading all virtual machines from their (potentially) vmx files.

Messages I know either equal “No message” or are about “This virtual machine may have been moved or copied.

If there is no available message, then you always get the stock message No message., so this is something you can use as a check in scripts.

Read the rest of this entry »

Posted in *nix, *nix-tools, ArchiveTeamWarrior, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: storing an ISO 8601 time-stamped backup tarball locally

Posted by jpluimers on 2022/01/25

In Determining the ESXi installation type (2014558) | VMware KB, I also showed how to backup the configuration and download it.

Sometimes you want an ISO 8601 time-stamped local tarball just in case you want to revert to it at a later stage.

First a small recap on how to get the tarball, download location and temporary location in the first place (it will be automatically deleted from the temporary location):

# vim-cmd hostsvc/firmware/sync_config
# vim-cmd hostsvc/firmware/backup_config
Bundle can be downloaded at : http://*/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz
# find /scratch/downloads/ -name *.tgz
/scratch/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz

Goal is to get the download filename and save it to a different folder and embed the ISO 8601 timestamp in the filename.

Like many scripts, sed and regular expressions come to the rescue once more, just like in ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes (which we will need anyway because of the ISO 8601 time stamp, and a bit of fiddling at regex101.com/r/NyrzKF

# SCRATCH_CONFIG_BUNDLE_NAME=$(vim-cmd hostsvc/firmware/backup_config | sed -n -E -e "s/^(Bundle can be downloaded at : http://*)(/downloads/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}/configBundle-.+?)(.tgz)$//scratch23/p")
# echo "SCRATCH_CONFIG_BUNDLE_NAME: '${SCRATCH_CONFIG_BUNDLE_NAME}'"
SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz'
# date --utc -I'seconds' --reference "${SCRATCH_CONFIG_BUNDLE_NAME}"
2021-05-09T17:44:42UTC

Note:

  • In the sed regular, expression, [[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12} matches a GUID.
  • In the date command, the --reference parameter must be last.
  • If you get the error below, then you ran too many backup_config commands in succession:
    (vim.fault.TooManyWrites) {
       faultCause = (vmodl.MethodFault) null, 
       faultMessage = 
       msg = "Received SOAP response fault from []: syncConfiguration
    fault.TooManyWrites.summary"
    }

    You can see this in the hostd.log, which on my system is in /scratch/log/hostd.log where it says I can retry in 2031 seconds (slightly more than half an hour):

    2021-05-09T19:34:13.420Z verbose hostd[A703B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab3] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
    2021-05-09T19:34:13.420Z info hostd[A703B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-27-2ab3] Event 196 : User root@127.0.0.1 logged in as VMware-client/6.5.0
    2021-05-09T19:34:13.420Z verbose hostd[A703B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab3] RecordOp ADD: sessionList["52f6d64e-0a35-c1d7-de97-624d234bc2a7"], ha-sessionmgr. Applied change to temp map.
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-27-2ab5 user=root] Task Created : haTask--vim.host.FirmwareSystem.syncConfiguration-114207804
    2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ADD: recentTask["haTask--vim.host.FirmwareSystem.syncConfiguration-114207804"], ha-taskmgr. Applied change to temp map.
    2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ASSIGN: info, haTask--vim.host.FirmwareSystem.syncConfiguration-114207804. Applied change to temp map.
    2021-05-09T19:34:13.423Z error hostd[9AC1B70] [Originator@6876 sub=Hostsvc.FirmwareSystem opID=vim-cmd-27-2ab5 user=root] Failed to sync configuration. Too many writes. Next sync possible in 2031 sec.
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Default opID=vim-cmd-27-2ab5 user=root] AdapterServer caught exception: vim.fault.TooManyWrites
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-27-2ab5 user=root] Task Completed : haTask--vim.host.FirmwareSystem.syncConfiguration-114207804 Status error
    2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ASSIGN: info, haTask--vim.host.FirmwareSystem.syncConfiguration-114207804. Applied change to temp map.
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Activation [N5Vmomi10ActivationE:0x0987e6e0] : Invoke done [syncConfiguration] on [vim.host.FirmwareSystem:ha-firmwareSystem]
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Throw vim.fault.TooManyWrites
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Result:
    --> (vim.fault.TooManyWrites) {
    -->    faultCause = (vmodl.MethodFault) null,
    -->    faultMessage = 
    -->    msg = ""
    --> }
    

Not few people have bumped into this, the only other I could find through [Wayback] “vim.fault.TooManyWrites” “syncConfiguration” – Google Search is [Archive.is] mal wieder purple Screen – VMware-Forum.

Figuring out the various parts of the SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz' is like at regex101.com/r/J4yU72, regex101.com/r/uID9xs and regex101.com/r/o8a4Am:

CONFIG_BUNDLE_DIRECTORY_NAME=$(echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\1/p")
CONFIG_BUNDLE_FILE_NAME=$(     echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\2/p")
CONFIG_BUNDLE_DOT_EXTENSION=$( echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\3/p")
echo "CONFIG_BUNDLE_DIRECTORY_NAME: '${CONFIG_BUNDLE_DIRECTORY_NAME}'"
echo "CONFIG_BUNDLE_FILE_NAME:      '${CONFIG_BUNDLE_FILE_NAME}'"
echo "CONFIG_BUNDLE_DOT_EXTENSION:  '${CONFIG_BUNDLE_DOT_EXTENSION}'"

Output is like this:

SCRATCH_CONFIG_BUNDLE_NAME:   '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/configBundle-ESXi-X10SRH-CF.tgz'
CONFIG_BUNDLE_DIRECTORY_NAME: '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/'
CONFIG_BUNDLE_FILE_NAME:      'configBundle-ESXi-X10SRH-CF'
CONFIG_BUNDLE_DOT_EXTENSION:  '.tgz'

Full backup-config-to-ESXi_configuration_backup-directory.sh script:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, cron/crontab, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: getting and setting the host name, domain and fqdn

Posted by jpluimers on 2022/01/19

A few links and notes:

  1. [Wayback] Changing the hostname of an ESX or ESXi host (1010821)

    Run these commands to change the hostname in ESXi 5.x, ESXi 6.x,ESXi 7.x, using the command line:

    • esxcli system hostname set –host=hostname
    • esxcli system hostname set –fqdn=fqdn
  2. [Wayback] ESX Host appears as localhost.localdomain in VMware Infrastructure/vSphere client (2009720)

    Cause

    The name resolution parameters were not properly configured during the installation of the ESX host.
  3. [Wayback] Domain repoint for embedded vCenter Server fails with error: “domain_consolidator Failed to set machine id” (71020)

    This issue is caused by a mismatch between the FQDN that was configured as the PNID during the vCenter Server deployment and the hostname that is currently configured.

I had a mismatch happen because of the second entry: a host configured in a different domain than it was deployed to.

Here are the commands to list and change the hosts name, domain and fqdn:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: persistent files you can edit to apply settings during boot

Posted by jpluimers on 2022/01/18

Since ESXi boots from RAM, most files in /etc are not persisted after modification.

The files that are persisted, are only persisted once every hour by auto-backup.sh, so better run auto-backup.sh by hand if you want to reboot after changing them.

The auto-backup.sh script is ran every hour at 1 minute past the hour as per below crontab.

Default ESXi crontab in /var/spool/cron/crontabs/root:

#min hour day mon dow command
1    1    *   *   *   /sbin/tmpwatch.py
1    *    *   *   *   /sbin/auto-backup.sh
0    *    *   *   *   /usr/lib/vmware/vmksummary/log-heartbeat.py
*/5  *    *   *   *   /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
00   1    *   *   *   localcli storage core device purge

Schedules deciphered via [Wayback] Crontab.guru – The cron schedule expression editor:

  • [Wayback] Every day at 01:01: “This module removes stale temporary files”
    1    1    *   *   *   /sbin/tmpwatch.py
  • [Wayback] Every hour at *:01: saves backup to /bootbank/state.tgz.
    1    *    *   *   *   /sbin/auto-backup.sh
  • [Wayback] Every hour at *:00 logs heartbeat messages to /var/log/vmksummary.log like 2021-02-23T19:00:02Z heartbeat: up 577d2h37m16s, 9 VMs; [[2802426 vmx 4194304kB] [6176344 vmx 4194304kB] [68997 vmx 8388608kB]] [[2802426 vmx 0%max] [6176344 vmx 0%max] [68997 vmx 0%max]]
    0    *    *   *   *   /usr/lib/vmware/vmksummary/log-heartbeat.py
  • [Wayback] Every 5th minute logs to /var/log/hostd-probe.log.
    */5  *    *   *   *   /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
  • [Wayback] Every day at 01:00: Removes storage devices which have not been seen in some time interval.
    00   1    *   *   *   localcli storage core device purge

Note that localcli commands are the same as esxcli; for esxcli, a running hostd is required; localcli can run without hostd. See:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware | Leave a Comment »

ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes

Posted by jpluimers on 2022/01/17

On VMware ESXi, with the  Busybox ash/dash shell, I wanted to get the current UTC timestamp in ISO 8601 format without dashes (-) or especially colons (:) and plus-signs (+) you have to back-slash escape colons or double quote parameters, which is often can be a pain).

This is why we can’t have good things: Getting the UTC 8610 timestamp was far less easy than I hoped for.

First of all, Busybox only allows for a precision of seconds, not milliseconds, and the specification format needs better documentation as per [Wayback] embedded linux – How to get ISO8601 seconds format from “date” in busybox? – Stack Overflow:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Busybox ash/dash – Hexadecimal To Decimal in Shell Script (via Stack Overflow)

Posted by jpluimers on 2021/12/21

This works fine on “BusyBox v1.29.3 (2019-05-21 15:22:06 PDT) multi-call binary.” that is included with VMware ESXi 6.5 update 3:

[Wayback] bash – Hexadecimal To Decimal in Shell Script – Stack Overflow

Dealing with a very lightweight embedded version of busybox on Linux means many of the traditional commands are not available (bc, printf, dc, perl, python)

echo $((0x2f))
47

hexNum=2f
echo $((0x${hexNum}))
47

Credit to [Wayback] Peter Leung for this solution.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, Power User, Scripting, Software Development | Leave a Comment »

On my list of things to try: Python with ESXi

Posted by jpluimers on 2021/10/28

After doing a lot of – historically grown – dash scripting for ESXi, I found out there is Python available on ESXi:

  • Python 3.5.10 on VMware ESXi 6.7.0 build-17700523 (VMware ESXi 6.7.0 Update 3)
  • Python 3.5.6 on VMware ESXi 6.5.0 build-13932383 (VMware ESXi 6.5.0 Update 3)
  • VMware 7: to be determined.

Yes I know that Python 3.5 is end-of-life (and 3.5.10 was the latest version), but it is a lot better than shell scripts.

So now some links for my list of things to try in order to use Python for scripting ESXi operations:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, Power User, Python, Scripting, Software Development | Leave a Comment »

Some bash parameter propagation links that hopefully will work with ash/dash too

Posted by jpluimers on 2021/10/27

For my link archive; I started with [Wayback] dash get all parameters quoted – Google Search:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »