The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Email Subscription

    Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 716 other subscribers

Archive for the ‘BusyBox’ Category

ESXi: rsync over ssh between two ESXi machines (and overcoming a potential “sh: rsync: not found”)

Posted by jpluimers on 2026/10/20

I have been using rsync for years now to sync directories around on VMware ESXi systems with great success.

Usually when upgrading to new physical machines, I synced the things to migrate to a big disk, then physically mounted that disk in a new machine.

Not this time: because of an unscheduled event I had to bring up the target machine in a different place before I could assemble the data on the big disk.

But since there was a network tunnel between the networks of both ESXi machines, I wanted to transfer over ssh: a thing I had done numerous times between various Linux and unix machines.

The cool thing is that rsync supports remote connections over ssh out of the box. Between ESXi machines, I bumped into this error though:

sh: rsync: not found
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.2]
Command exited with non-zero status 12

The last three lines are collateral of the first line, which I queried [Wayback/Archive] “sh: rsync: not found” – Google Search. Further below I will go deeper into the rabbit-hole I took, but the actual solution was to Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, man/manual pages, OpenSSH, Power User, rsync, Scripting, Software Development, SSH, ssh/sshd, Virtualization, VMware, VMware ESXi | Leave a Comment »

Some notes on replacing parts of a text file with template text using sed on a Busybox system

Posted by jpluimers on 2022/03/17

Note before you think about putting stuff in /etc/rc.local.d/local.sh: that script will not be executed when UEFI booting.

In a very lightweight Busybox system, I wanted to modify some configuration files automatically using fragments stored in template files.

The system has diff, but no patch.

The basic idea is to use sed to insert the template files into certain spots of the configuration file when certain marker texts are not present. So I want the opposite of [Wayback] Hey Stephen Wood: Try patch instead of sed in shell scripts.

Basically the idea is a poor-man’s patch, described in Too bad: ESXi busybox has diff, but not patch « The Wiert Corner – irregular stream of stuff.

Some links that might help me with this:

One alternative would have been to use ed (which is part of the normal Busybox), but ESXi Busybox omits ed like it omits patch.

Too bad that sed commands are too different from ed commands, as I could have used diff -e on another system based on ideas here:

I might give it one more go, as vi is sort of derived from ed via ex (see vi: Creation – Wikipedia), which means that vi “colon mode” (officially command mode: [Wayback] Vim documentation: cmdline) is very similar to ed.

Another alternative would be awk, but I have done so little work with it awk, that I’m hesitating to use a new tool. Some links:

And finally, ash could be used:

The kind of modifications I am after

Below are a few links with examples of the kind of modifications I want to make. Most patch just /etc/rc.local.d/local.sh, but some others introduce other changes as well.

Note that especially with networking settings, local.sh commands might not have any effect (for instance when having slow DHCP or other network issues), see for instance [Wayback/Archive.is] I’m running ESXi 5.5 and my persistent route in local.sh is not taking effect after boot. : vmware.

There is a very convoluted way around using local.sh by using the VIB authoring tool as described in [Wayback] How to create persistent firewall rules on ESXi. It requires lowering the software acceptance level to Community Supported (esxcli software acceptance set --level=CommunitySupported), which gives you a hard time installing ESXi updates.

I got that VIB idea from [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN, as:

The local.sh file gets overwritten often with upgrades so it would mean another step during the process.

From the same thread comes [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN

set the sticky bit on your separate xml-file – then it will be backed up and persist through reboot: chmod +t

run backup manually before the first reboot: /sbin/auto-backup.sh  because backup runs only once per hour

Within vSphere, one could use [Wayback] Configure ESXi Hosts with Host Profiles, but a standalone ESXi box is not part of vSphere, so that won’t work.

ESXi 7 and up

ESXi 7 makes the above harder as for instance user root cannot change file rights any more, so eventually I might revert to a VM that auto-boots when ESXi comes up, then patches the right files in place over PowerCLI (read-only) or SSH.

Need to give this some thought later:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Awk, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, PowerCLI, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Too bad: ESXi busybox has `diff`, but not `patch`

Posted by jpluimers on 2022/03/02

On my ESXi boxes, I have a directory with local scripts that in part depend on the machine.

So I contemplated patching the dending parts with patch.

Then I found out that the BusyBox that VMware built for ESXi does have diff, but not patch:

# $(readlink -f "`which diff`")
BusyBox v1.29.3 (2021-01-17 01:25:00 PST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list
   or: function [arguments]...

    BusyBox is a multi-call binary that combines many common Unix
    utilities into a single executable.  Most people will create a
    link to busybox for each function they wish to use and BusyBox
    will act like whatever it was invoked as.

Currently defined functions:
    addgroup, adduser, arch, ash, awk, basename, bunzip2, bzcat, bzip2, cat, chgrp, chmod, chown, chvt, cksum, clear, cp, crond,
    cut, date, dd, delgroup, deluser, diff, dirname, dnsdomainname, du, echo, egrep, eject, env, expr, false, fdisk, fgrep, find,
    fstrim, getty, grep, groups, gunzip, gzip, halt, head, hexdump, hostname, inetd, init, kill, ln, logger, login, ls, lzop,
    lzopcat, md5sum, mkdir, mkfifo, mknod, mktemp, more, mv, nohup, nslookup, od, passwd, poweroff, printf, readlink, reboot,
    reset, resize, rm, rmdir, sed, seq, setsid, sh, sha1sum, sha256sum, sha3sum, sha512sum, sleep, sort, ssl_client, stat, stty,
    sum, sync, tail, tar, taskset, tee, test, time, timeout, touch, true, uname, uniq, unlink, unlzop, unzip, usleep, vi, watch,
    wc, wget, which, who, xargs, zcat

This list is much shorter than the applets that are supported in [Wayback] BusyBox – The Swiss Army Knife of Embedded Linux, so VMware did cut out quite a few.

Generating the above output

The command-line trick above first expands diff using the output of which diff, then finds out where it links to through the readlink -f wrapper there the back-quotes “`” get this output:

# readlink -f "`which diff`"
/usr/lib/vmware/busybox/bin/busybox

Finally the $(...) executes the output of readlink.

It is based on [Wayback] bash – How to resolve symbolic links in a shell script – Stack Overflow

readlink -f "$path"

Editor’s note: The above works with GNU readlink and FreeBSD/PC-BSD/OpenBSD readlink, but not on OS X as of 10.11.GNU readlink offers additional, related options…

Need to devise a way to apply patches

Given there is no patch, I need to think about a good way to apply patches, for instance to snip this into /etc/rc.local.d/local.sh in a reliable way:

## BEGIN-PATCH-PATH

# local binaries are in /vmfs/volumes/NVMe980PRO_1TB/local-bin/
# link that directory from /opt/bin
# then add /opt/bin to the PATH in /etc/profile so that on each logon it becomes available
# this means you need to logon twice after reboot:
# - first to patch /etc/profile
# - second to have the correct PATH loaded from /etc/profile
# direcory exist trick from https://stackoverflow.com/questions/59838/how-can-i-check-if-a-directory-exists-in-a-bash-shell-script

patch_etc_profile_PATH() {
    if [ -d "$1" ]; then
      ln -s "$1" "/opt/bin"
      sed -i -e 's!PATH=/bin:/sbin!PATH=/bin:/sbin:/opt/bin/!' /etc/profile
    fi
}

patch_etc_profile_PATH /vmfs/volumes/NVMe980PRO_1TB/local-bin/

## END-PATCH-PATH

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

How to rename a VM through SSH on ESXi ? |VMware Communities

Posted by jpluimers on 2021/06/28

From [WayBack] How to rename a VM through SSH on ESXi ? |VMware Communities (numbering and code highlighting mine):

Kindly find the below:

  1. Backup the virtual machine
  2. Power down the virtual machine
  3. Remove the virtual machine from the vSphere host inventory
  4. Open an SSH console session to the vSphere host
  5. Navigate to the storage directory containing the virtual machine: For example: cd /vmfs/volumes/<datastore_name>/<original_vmname>
  6. Rename the primary .vmdk configuration files: vmkfstools -E "<original_vmname>.vmdk" "<new_vmname>.vmdk"
  7. Rename the .vmx configuration file: mv "original_vmname.vmx" "new_vmname.vmx"
  8. Edit the virtual machine .vmx configuration file (Be sure to properly update the directory and file name of the .vswp swap file reference): vi "new_vmname.vmx"
  9. Rename any remaining files in the virtual machine’s folder as needed:
    1. Rename the .vmxf configuration file: mv "original_vmname.vmxf" "new_vmname.vmxf"
    2. Rename the .nvram configuration file: mv "original_vmname.nvram" "new_vmname.nvram"
    3. Rename the .vsd configuration file: mv "original_vmname.vsd" "new_vmname.vmsd"
  10. Rename the virtual machine folder: Move up one directory level to the parent folder ( cd .. ) then rename the virtual machine directory: mv "original_directory" "new_directory"
  11. Add the newly-named virtual machine to the host’s inventory (the newly renamed .vmx configuration file)
  12. Power on the newly renamed virtual machine
  13. Answer “I moved it” to the virtual machine question prompt (not “I copied it”)
  14. Review the virtual machine and all files/folders to make sure it is named as desired and functioning properly

Note: There are other methods to allow for renaming, but this method is fairly quick and easy. It should work on all editions of vSphere from free to Enterprise Plus.

The “Answer question” prompt where you should selected “I moved it”:

->

Prompt with symlink names in the path

On a site note, I need to figure uit how to set the ESXi shell prompt to show the current path like pwd does (with symlink names in it instead of the followed symlink targets):

[root@ESXi-X9SRI-3F:~] cd /vmfs/volumes/EVO860_250GB/
[root@ESXi-X9SRI-3F:/vmfs/volumes/5c9bd516-ef1f6d4c-f1b1-0025907d9d5c] pwd
/vmfs/volumes/EVO860_250GB

The ESXi shell is based on busybox, in fact it uses the ash variety:

[root@ESXi-X9SRI-3F:/vmfs/volumes/5c9bd516-ef1f6d4c-f1b1-0025907d9d5c] `readlink -f \`which readlink\`` | grep ^BusyBox
BusyBox v1.29.3 (2018-11-02 15:37:50 PDT) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
[root@ESXi-X9SRI-3F:/vmfs/volumes/5c9bd516-ef1f6d4c-f1b1-0025907d9d5c] type chdir
chdir is a shell builtin

This seemed to work fine:

[root@ESXi-X9SRI-3F:/vmfs/volumes/5c9bd516-ef1f6d4c-f1b1-0025907d9d5c] PS1="[\u@\h:`pwd`] "
[root@ESXi-X9SRI-3F:/vmfs/volumes/EVO860_250GB]

But in faxt fails, as it only takes a pwd value once, and not every time the prompt is evaluated:

[root@ESXi-X9SRI-3F:/vmfs/volumes/EVO860_250GB] cd ..
[root@ESXi-X9SRI-3F:/vmfs/volumes/EVO860_250GB] pwd
/vmfs/volumes
[root@ESXi-X9SRI-3F:/vmfs/volumes/EVO860_250GB]

So I need to re-visit these links:

–jeroen

Posted in *nix, *nix-tools, BusyBox, ESXi6, ESXi6.5, ESXi6.7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: where are my log files actually stored? Actually, most of them are in `/scratch/log` which points to a hidden `.locker` directory in a datastore.

Posted by jpluimers on 2021/05/05

A summary of the full gist at [Wayback/Archive.is] ESXi-where-are-my-log-files-stored.txt:

# ls -al / /var/ /var/log/ /var/run/ /scratch/ /scratch/log/ | grep "/\|log\|-\>"
/:
...
lrwxrwxrwx    1 root     root            57 Apr  4 18:16 scratch -> /vmfs/volumes/5ce2d440-72311161-75c5-0025907d9d5c/.locker
...
/scratch/:
drwxr-xr-x    1 root     root        106496 Apr 10 08:40 log
/scratch/log/:
-rw-------    1 root     root           411 Apr  4 18:20 Xorg.log
...
-rw-------    1 root     root         78835 Apr  4 10:30 syslog.0.gz
-rw-------    1 root     root         61136 Mar 18 15:05 syslog.1.gz
-rw-------    1 root     root         60589 Feb 24 00:30 syslog.2.gz
-rw-------    1 root     root         60373 Feb  1 08:01 syslog.3.gz
-rw-------    1 root     root         60203 Jan  9 15:50 syslog.4.gz
-rw-------    1 root     root         59889 Dec 17 23:20 syslog.5.gz
-rw-------    1 root     root         60398 Nov 25 06:50 syslog.6.gz
-rw-------    1 root     root         60563 Nov  2 14:25 syslog.7.gz
-rw-------    1 root     root        531794 Apr 10 09:35 syslog.log
...
-rw-------    1 root     root        157255 Apr  4 18:17 vvold.log
/var/:
...
drwxr-xr-x    1 root     root           512 Apr  5 19:19 log
...
/var/log/:
-rw-r--r--    1 root     root           416 Apr  4 18:16 .vmsyslogd.err
...
-rw-r--r--    1 root     root         38069 Apr  4 18:20 configRP.log
-rw-r--r--    1 root     root             0 Apr  4 18:16 cryptoloader.log
...
-rw-r--r--    1 root     root            87 Apr  5 21:57 esxcli.log
...
-rw-------    1 root     root          3350 Apr  4 18:16 init.log
-rw-r--r--    1 root     root           966 Apr  4 18:16 iofilter-init.log
...
-rw-r--r--    1 root     root         21769 Apr  4 18:16 jumpstart-esxcli-stdout.log
-rw-r--r--    1 root     root         18857 Apr  4 18:16 jumpstart-native-stdout.log
-rw-r--r--    1 root     root         10837 Apr  4 18:16 jumpstart-stdout.log
-rw-r--r--    1 root     root             0 Apr  4 18:16 kickstart.log
...
-rw-------    1 root     root         10916 Apr  4 18:16 sysboot.log
...
-rw-------    1 root     root            64 Apr 10 09:13 tallylog
...
/var/run/:
lrwxrwxrwx    1 root     root            12 Apr  4 18:16 log -> /scratch/log
  • Almost all log files (most from /var/log and all from /var/run/log) are actually persistently stored in /scratch/log and survive reboots. Just a few are non-persistent.
  • /var/log/syslog is being archived as .gz files (compressed by gzip).
  • syslog is special: the location can be configured, and even be external: [Wayback] Configuring syslog on ESXi (2003322)

    VMware vSphere ESXi 5.0 and higher hosts run a Syslog service (vmsyslogd) that provides a standard mechanism for logging messages from the VMkernel and other system components. By default in ESXi, these logs are placed on a local scratch volume or a ramdisk. To preserve the logs further, ESXi can be configured to place these logs to an alternate storage location on disk and to send the logs across the network to a Syslog server.

  • A summary of some of the above log files is at [Wayback] ESXi Log File Locations

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, gzip, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

VMware ESXi 6 and 7: checking and setting/clearing maintenance mode from the console

Posted by jpluimers on 2021/04/21

Every now and then it is useful to be able to do maintenance work from the ESXi console addition to the ESXi web-user interface.

I know there are many sites having this information, but many of them forgot to format the statements with code markup, so parameters with two dashes -- (each a Wayback Unicode Character ‘HYPHEN-MINUS’ (U+002D)) now have become an [Wayback] Unicode Character ‘EN DASH’ (U+2013) which is incompatible with most console programs, especially the ESXi ones (as they are Busybox based to minimise footprint).

Note you can use this small site (which runs in-browser, so does not phone home) to get the unicode code points for any string: [Wayback] What Unicode character is this ?.

Links like below (most on the vmware.com domain) have this EN DASH and make me document things on my blog instead of trying code directly from blogs or forum posts:

So below are three commands I use that have to do with the maintenance mode (the mode that for instance you can use to update an ESXi host to the latest patch level).

    1. Check the maintenance mode (which returns Enabled or Disabled):
      esxcli system maintenanceMode get
    2. Enable maintenance mode (which returns nothing when succeeded, and Maintenance mode is already enabled. when failed):
      esxcli system maintenanceMode set --enable true
    3. Disable maintenance mode (which returns nothing when succeeded, and Maintenance mode is already disabled. when failed):
      esxcli system maintenanceMode get

Some examples, especially an the various output possibilities (commands in bold, output in italic):

# esxcli system maintenanceMode get
Disabled
# esxcli system maintenanceMode set --enable false
Maintenance mode is already disabled.
# esxcli system maintenanceMode set --enable true 
# esxcli system maintenanceMode get
Enabled
# esxcli system maintenanceMode set --enable true
Maintenance mode is already enabled.
# esxcli system maintenanceMode set --enable false
# esxcli system maintenanceMode get
Disabled

I made these scripts for this:

  • esxcli-maintenanceMode-show.sh:
    #!/bin/sh
esxcli system maintenanceMode get
  • esxcli-maintenanceMode-enter.sh:
    #!/bin/sh
esxcli system maintenanceMode set --enable true
  • esxcli-maintenanceMode-exit.sh:
    #!/bin/sh
esxcli system maintenanceMode set --enable false

Note I have not checked the exit codes for these esxcli commands yet, but did blog about how to do that: Busybox sh (actually ash derivative dash): checking exit codes.

–jeroen

Posted in BusyBox, Development, Encoding, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Software Development, Unicode, Virtualization, VMware, VMware ESXi | Leave a Comment »

Busybox sh (actually ash derivative dash): checking exit codes

Posted by jpluimers on 2021/04/20

Even if you include a double quotes "sh" in a Google search to force only sh (in the early days this was the Thompson shell, but nowadays usually a Bourne shell or derivative) results, almost all unix like scripting examples you find are based on bash (the Bourne again shell), so I was glad I dug a bit deeper into what the actual Busybox shell is.

I wanted to know which shell Busybox uses and what capabilities it has, as ESXi ships with this very slimmed down set of tools (called applets in Busybox speak).

It does not even include ssh: that gap is often filled by [Wayback] Dropbear SSH, which was used by ESXi and named dbclient (I think with ESXi 6.0 it was replaced with a more regular ssh implementation): [Wayback] How to compile a statically linked rsync binary for ESXi.

Busybox shell source code is at [Wayback] ash.c\shell – busybox – BusyBox: The Swiss Army Knife of Embedded Linux and indicates the shell is the ash (the Almquist shell) derivative dash (yes, you guessed it right: the Debian Almquist shell), ported from NetBSD and debianized:

 * Copyright (c) 1997-2005 Herbert Xu <herbert@gondor.apana.org.au>
 * was re-ported from NetBSD and debianized.
...
//config:   The most complete and most pedantically correct shell included with
//config:   busybox. This shell is actually a derivative of the Debian 'dash'
//config:   shell (by Herbert Xu), which was created by porting the 'ash' shell
//config:   (written by Kenneth Almquist) from NetBSD.

nx like systems have a shell hell similar to Windows DLL hell: there are too many, and their differences and be both subtle and frustrating. To get a feel, browse through Source: Comparison of command shells – Wikipedia (yes, some shells from other operating environments like DOS, OS/2, VMS and Windows, but the majority is nx).

Since ash is sufficiently different from bash (for example [Wayback] ash – exit code for a piped process), I always want to know what shell code (which often comes from bash as it is so ubiquitous) will work.

There is hardly any shell documentation at the Busybox site. There is [Wayback] BusyBox – The Swiss Army Knife of Embedded Linux, the source code at [Wayback] ash.c\shell – busybox – BusyBox: The Swiss Army Knife of Embedded Linux does not offer much either,

A manual page of it is at [Archive.is] ash(1) [minix man page]. There you see the age: back then, “exit status” is used where nowadays many people would use “exit code”. It does not explain how to check for specific exit codes.

Because ash is derived from the Bourne shell, this page was of great help for me to grasp exit code handing: [Wayback] Exit Codes – Shell Scripting Tutorial

A Bourne Shell Programming / Scripting Tutorial for learning about using the Unix shell.

Here two examples from that page to get me going:

#!/bin/sh
# Second attempt at checking return codes
grep "^${1}:" /etc/passwd > /dev/null 2>&1
if [ "$?" -ne "0" ]; then
  echo "Sorry, cannot find user ${1} in /etc/passwd"
  exit 1
fi
USERNAME=`grep "^${1}:" /etc/passwd|cut -d":" -f1`
NAME=`grep "^${1}:" /etc/passwd|cut -d":" -f5`
HOMEDIR=`grep "^${1}:" /etc/passwd|cut -d":" -f6`

echo "USERNAME: $USERNAME"
echo "NAME: $NAME"
echo "HOMEDIR: $HOMEDIR"

and

#!/bin/sh
# A Tidier approach

check_errs()
{
  # Function. Parameter 1 is the return code
  # Para. 2 is text to display on failure.
  if [ "${1}" -ne "0" ]; then
    echo "ERROR # ${1} : ${2}"
    # as a bonus, make our script exit with the right error code.
    exit ${1}
  fi
}

### main script starts here ###

grep "^${1}:" /etc/passwd > /dev/null 2>&1
check_errs $? "User ${1} not found in /etc/passwd"
USERNAME=`grep "^${1}:" /etc/passwd|cut -d":" -f1`
check_errs $? "Cut returned an error"
echo "USERNAME: $USERNAME"
check_errs $? "echo returned an error - very strange!"

This basically means that status code handling is the same as in bash, so constructs can be used like [Wayback] bash – How to check the exit status using an if statement – Stack Overflow:

$? is a parameter like any other. You can save its value to use before ultimately calling exit.

exit_status=$?
if [ $exit_status -eq 1 ]; then
    echo "blah blah blah"
fi
exit $exit_status

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, BusyBox, Development, Power User, Scripting, Software Development, ssh/sshd | 1 Comment »