The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,854 other subscribers

Archive for the ‘*nix-tools’ Category

Too bad: ESXi busybox has `diff`, but not `patch`

Posted by jpluimers on 2022/03/02

On my ESXi boxes, I have a directory with local scripts that in part depend on the machine.

So I contemplated patching the dending parts with patch.

Then I found out that the BusyBox that VMware built for ESXi does have diff, but not patch:

# $(readlink -f "`which diff`")
BusyBox v1.29.3 (2021-01-17 01:25:00 PST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list
   or: function [arguments]...

    BusyBox is a multi-call binary that combines many common Unix
    utilities into a single executable.  Most people will create a
    link to busybox for each function they wish to use and BusyBox
    will act like whatever it was invoked as.

Currently defined functions:
    addgroup, adduser, arch, ash, awk, basename, bunzip2, bzcat, bzip2, cat, chgrp, chmod, chown, chvt, cksum, clear, cp, crond,
    cut, date, dd, delgroup, deluser, diff, dirname, dnsdomainname, du, echo, egrep, eject, env, expr, false, fdisk, fgrep, find,
    fstrim, getty, grep, groups, gunzip, gzip, halt, head, hexdump, hostname, inetd, init, kill, ln, logger, login, ls, lzop,
    lzopcat, md5sum, mkdir, mkfifo, mknod, mktemp, more, mv, nohup, nslookup, od, passwd, poweroff, printf, readlink, reboot,
    reset, resize, rm, rmdir, sed, seq, setsid, sh, sha1sum, sha256sum, sha3sum, sha512sum, sleep, sort, ssl_client, stat, stty,
    sum, sync, tail, tar, taskset, tee, test, time, timeout, touch, true, uname, uniq, unlink, unlzop, unzip, usleep, vi, watch,
    wc, wget, which, who, xargs, zcat

This list is much shorter than the applets that are supported in [Wayback] BusyBox – The Swiss Army Knife of Embedded Linux, so VMware did cut out quite a few.

Generating the above output

The command-line trick above first expands diff using the output of which diff, then finds out where it links to through the readlink -f wrapper there the back-quotes “`” get this output:

# readlink -f "`which diff`"
/usr/lib/vmware/busybox/bin/busybox

Finally the $(...) executes the output of readlink.

It is based on [Wayback] bash – How to resolve symbolic links in a shell script – Stack Overflow

readlink -f "$path"

Editor’s note: The above works with GNU readlink and FreeBSD/PC-BSD/OpenBSD readlink, but not on OS X as of 10.11.GNU readlink offers additional, related options…

Need to devise a way to apply patches

Given there is no patch, I need to think about a good way to apply patches, for instance to snip this into /etc/rc.local.d/local.sh in a reliable way:

## BEGIN-PATCH-PATH

# local binaries are in /vmfs/volumes/NVMe980PRO_1TB/local-bin/
# link that directory from /opt/bin
# then add /opt/bin to the PATH in /etc/profile so that on each logon it becomes available
# this means you need to logon twice after reboot:
# - first to patch /etc/profile
# - second to have the correct PATH loaded from /etc/profile
# direcory exist trick from https://stackoverflow.com/questions/59838/how-can-i-check-if-a-directory-exists-in-a-bash-shell-script

patch_etc_profile_PATH() {
    if [ -d "$1" ]; then
      ln -s "$1" "/opt/bin"
      sed -i -e 's!PATH=/bin:/sbin!PATH=/bin:/sbin:/opt/bin/!' /etc/profile
    fi
}

patch_etc_profile_PATH /vmfs/volumes/NVMe980PRO_1TB/local-bin/

## END-PATCH-PATH

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Bash functions to encode and decode the ‘Basic’ HTTP Authentication Scheme

Posted by jpluimers on 2022/02/24

IoT devices still often use the ‘Basic’ HTTP Authentication Scheme for authorisation, see [Wayback] RFC7617: The ‘Basic’ HTTP Authentication Scheme (RFC ) and [Wayback] RFC2617: HTTP Authentication: Basic and Digest Access Authentication (RFC ).

Often this authentication is used even over http instead of over https, for instance the Egardia/Woonveilig alarm devices I wrote about yesterday at  Egardia/Woonveilig: some notes about logging on a local gateway to see more detailed information on the security system. This is contrary to guidance in:

  • RFC7617:
       This scheme is not considered to be a secure method of user
       authentication unless used in conjunction with some external secure
       system such as TLS (Transport Layer Security, [RFC5246]), as the
       user-id and password are passed over the network as cleartext.
  • RFC2617:
       "HTTP/1.0", includes the specification for a Basic Access
       Authentication scheme. This scheme is not considered to be a secure
       method of user authentication (unless used in conjunction with some
       external secure system such as SSL [5]), as the user name and
       password are passed over the network as cleartext.

Fiddling with those alarm devices, I wrote these two little bash functions (with a few notes) that work both on MacOS and in Linux:

# `base64 --decode` is platform neutral (as MacOS uses `-D` and Linux uses `-d`)
# `$1` is the encoded username:password
function decode_http_Basic_Authorization(){
  echo $1 | base64 --decode
  echo
}

# `base64` without parameters encodes
# `echo -n` does not output a new-line
# `$1` is the username; `$2` is the password
function encode_http_Basic_Authorization(){
  echo $1:$2 | base64
}

The first decodes the <credentials> from a Authorization: Basic <credentials> header into a username:password clean text followed by a newline.

The second one encodes a pair of username and password parameters into such a <credentials> string.

They are based on these initial posts that were not cross platform or explanatory:

  1. [Wayback] Decode HTTP Basic Access Authentication – Stack Pointer
  2. [Wayback] Create Authorization Basic Header | MJ’s Web Log

–jeroen

Posted in *nix, *nix-tools, Apple, Authentication, bash, bash, Communications Development, Development, HTTP, Internet protocol suite, Linux, Mac OS X / OS X / MacOS, Power User, Scripting, Security, Software Development, TCP, Web Development | Leave a Comment »

Viewing the last lines of the postfix log with journalctl (with help from Unix & Linux Stack Exchange)

Posted by jpluimers on 2022/02/24

Two command-lines I use to view my Postfix logs:

  1. journalctl --unit postfix --since "2 days ago"
  2. journalctl --unit postfix --pager-end

Note that neither of these work well with the --follow (or equivalent -f) option, as this will effectively disable the pager (which by default is less).

The second is via [Wayback] systemd – How to see the latest x lines from systemctl service log – Unix & Linux Stack Exchange (which got the--pagerend bit wrong, as it misses a dash and should be --pager-end, but still thanks [Wayback] Daniel Kmak):

Just:

journalctl -u SERVICE_NAME -e

Parameter -e stands for:

-e –pagerend

That’s the one ! Other answers will go through the whole log to get to its end, which can be veeeeery long for large syslogs.

The last bit (by [Wayback] Léo Germond, thanks!) is why I like it most.

Similarly, specifying --since in the first example will not go through the whole log.

Some background information:

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, Development, journalctl and journald, Linux, postfix, Power User, Scripting, Software Development, systemd | Leave a Comment »

Sendmail queue files: “Mhost map: lookup” deferred

Posted by jpluimers on 2022/02/22

I had forgotten about sendmail mail queue entries that start with Mhost map: lookup.

It it happens for all email then you have a DNS problem.

When it happens only for a few emails, then usually that domain is gone (often because of SPAM reasons).

via: [Wayback] “Mhost map: lookup” – Google Search

–jeroen

Posted in *nix, *nix-tools, Power User, sendmail | Leave a Comment »

I always forget how simple it is to show the definition of bash function or alias (via: Stack Overflow)

Posted by jpluimers on 2022/02/17

I always facepalm myself after looking up this: [Wayback] Can bash show a function’s definition? – Stack Overflow (thanks [Wayback] Benjamin Bannier!):

Use type. If foobar is e.g. defined in your ~/.profile:

$ type foobar
foobar is a function
foobar {
    echo "I'm foobar"
}

type will also expand aliases, which is a nice bonus :) – [Wayback] Esdras Lopez

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

rsync backup of your ESXi box: How to make a statically linked rsync binary

Posted by jpluimers on 2022/02/09

As promised mid last year in “fixing” ESXi “rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]”, I would follow up on building a static rsync for ESXi one day.

So below a few links on how to do this, roughly in the order I found them (most via [Wayback] vmware rsync “3.1.2” static – Google Search):

Especially the last link has a great set of steps on how to build manually.

Boy I forgot how long ago CentOS 3.9 was: [Wayback] [CentOS-announce] CentOS 3.9 is released for i386 and x86_64 Read the rest of this entry »

Posted in *nix, *nix-tools, CentOS, Development, Linux, Power User, RedHat, Software Development | 2 Comments »

ESXi: on the console/ssh, when a moved VM pauses during power-on: show which VMs have messages waiting, then answer them

Posted by jpluimers on 2022/01/27

First the script that display messages for all virtual machines, vim-cmd-display-messages-for-all-VMs.sh:

#!/bin/sh
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    echo "Messages for VM with id ${vmid} which has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    vim-cmd vmsvc/message ${vmid}
done
exit 0

It is very similar to vim-cmd-reload-all-VM-vmx-configurations.sh from Source: ESXi: reloading all virtual machines from their (potentially) vmx files.

Messages I know either equal “No message” or are about “This virtual machine may have been moved or copied.

If there is no available message, then you always get the stock message No message., so this is something you can use as a check in scripts.

Read the rest of this entry »

Posted in *nix, *nix-tools, ArchiveTeamWarrior, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Overriding some DNS entries for internal networks

Posted by jpluimers on 2022/01/27

Based on [Wayback] domain name system – Overriding some DNS entries in BIND for internal networks – Server Fault and some further reading, there seem to be two ways used in these scenarios:

I wonder how that would interact best with Pi-Hole based solutions. Would it be best to have your local network use the Pi-Hole server, then have the Pi-Hole server obtain the DNS information it cannot resolve through one of the above solutions? Or would other solutions work better?

So here are a few links:

Pi-Hole seems not interested in RPZ: [Wayback] Implement Response Zone Policies (NXDOMAIN) for end-user performance increase – Feature Requests / Implemented – Pi-hole Userspace

Pi-Hole default blacklist is mentioned in [Wayback/Archive.is] pi-hole/basic-install.sh at master · pi-hole/pi-hole (look for adlistFile which defaults to [Wayback/Archive.is] StevenBlack/hosts: 🔒 Consolidating and extending hosts files from several well-curated sources. Optionally pick extensions for porn, social media, and other categories.).

Since I need this for ESXi: [Wayback/Archive.is] Let’s Encrypt SSL for ESXi

–jeroen

Posted in *nix, *nix-tools, bind-named, DNS, Internet, Linux, Power User | Leave a Comment »

ESXi: storing an ISO 8601 time-stamped backup tarball locally

Posted by jpluimers on 2022/01/25

In Determining the ESXi installation type (2014558) | VMware KB, I also showed how to backup the configuration and download it.

Sometimes you want an ISO 8601 time-stamped local tarball just in case you want to revert to it at a later stage.

First a small recap on how to get the tarball, download location and temporary location in the first place (it will be automatically deleted from the temporary location):

# vim-cmd hostsvc/firmware/sync_config
# vim-cmd hostsvc/firmware/backup_config
Bundle can be downloaded at : http://*/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz
# find /scratch/downloads/ -name *.tgz
/scratch/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz

Goal is to get the download filename and save it to a different folder and embed the ISO 8601 timestamp in the filename.

Like many scripts, sed and regular expressions come to the rescue once more, just like in ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes (which we will need anyway because of the ISO 8601 time stamp, and a bit of fiddling at regex101.com/r/NyrzKF

# SCRATCH_CONFIG_BUNDLE_NAME=$(vim-cmd hostsvc/firmware/backup_config | sed -n -E -e "s/^(Bundle can be downloaded at : http://*)(/downloads/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}/configBundle-.+?)(.tgz)$//scratch23/p")
# echo "SCRATCH_CONFIG_BUNDLE_NAME: '${SCRATCH_CONFIG_BUNDLE_NAME}'"
SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz'
# date --utc -I'seconds' --reference "${SCRATCH_CONFIG_BUNDLE_NAME}"
2021-05-09T17:44:42UTC

Note:

  • In the sed regular, expression, [[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12} matches a GUID.
  • In the date command, the --reference parameter must be last.
  • If you get the error below, then you ran too many backup_config commands in succession:
    (vim.fault.TooManyWrites) {
       faultCause = (vmodl.MethodFault) null, 
       faultMessage = 
       msg = "Received SOAP response fault from []: syncConfiguration
    fault.TooManyWrites.summary"
    }

    You can see this in the hostd.log, which on my system is in /scratch/log/hostd.log where it says I can retry in 2031 seconds (slightly more than half an hour):

    2021-05-09T19:34:13.420Z verbose hostd[A703B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab3] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map.
    2021-05-09T19:34:13.420Z info hostd[A703B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-27-2ab3] Event 196 : User root@127.0.0.1 logged in as VMware-client/6.5.0
    2021-05-09T19:34:13.420Z verbose hostd[A703B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab3] RecordOp ADD: sessionList["52f6d64e-0a35-c1d7-de97-624d234bc2a7"], ha-sessionmgr. Applied change to temp map.
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-27-2ab5 user=root] Task Created : haTask--vim.host.FirmwareSystem.syncConfiguration-114207804
    2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ADD: recentTask["haTask--vim.host.FirmwareSystem.syncConfiguration-114207804"], ha-taskmgr. Applied change to temp map.
    2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ASSIGN: info, haTask--vim.host.FirmwareSystem.syncConfiguration-114207804. Applied change to temp map.
    2021-05-09T19:34:13.423Z error hostd[9AC1B70] [Originator@6876 sub=Hostsvc.FirmwareSystem opID=vim-cmd-27-2ab5 user=root] Failed to sync configuration. Too many writes. Next sync possible in 2031 sec.
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Default opID=vim-cmd-27-2ab5 user=root] AdapterServer caught exception: vim.fault.TooManyWrites
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-27-2ab5 user=root] Task Completed : haTask--vim.host.FirmwareSystem.syncConfiguration-114207804 Status error
    2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ASSIGN: info, haTask--vim.host.FirmwareSystem.syncConfiguration-114207804. Applied change to temp map.
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Activation [N5Vmomi10ActivationE:0x0987e6e0] : Invoke done [syncConfiguration] on [vim.host.FirmwareSystem:ha-firmwareSystem]
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Throw vim.fault.TooManyWrites
    2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Result:
    --> (vim.fault.TooManyWrites) {
    -->    faultCause = (vmodl.MethodFault) null,
    -->    faultMessage = 
    -->    msg = ""
    --> }
    

Not few people have bumped into this, the only other I could find through [Wayback] “vim.fault.TooManyWrites” “syncConfiguration” – Google Search is [Archive.is] mal wieder purple Screen – VMware-Forum.

Figuring out the various parts of the SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz' is like at regex101.com/r/J4yU72, regex101.com/r/uID9xs and regex101.com/r/o8a4Am:

CONFIG_BUNDLE_DIRECTORY_NAME=$(echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\1/p")
CONFIG_BUNDLE_FILE_NAME=$(     echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\2/p")
CONFIG_BUNDLE_DOT_EXTENSION=$( echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\3/p")
echo "CONFIG_BUNDLE_DIRECTORY_NAME: '${CONFIG_BUNDLE_DIRECTORY_NAME}'"
echo "CONFIG_BUNDLE_FILE_NAME:      '${CONFIG_BUNDLE_FILE_NAME}'"
echo "CONFIG_BUNDLE_DOT_EXTENSION:  '${CONFIG_BUNDLE_DOT_EXTENSION}'"

Output is like this:

SCRATCH_CONFIG_BUNDLE_NAME:   '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/configBundle-ESXi-X10SRH-CF.tgz'
CONFIG_BUNDLE_DIRECTORY_NAME: '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/'
CONFIG_BUNDLE_FILE_NAME:      'configBundle-ESXi-X10SRH-CF'
CONFIG_BUNDLE_DOT_EXTENSION:  '.tgz'

Full backup-config-to-ESXi_configuration_backup-directory.sh script:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, cron/crontab, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

dd on MacOS / OS X with progress report

Posted by jpluimers on 2022/01/21

Since dd on Apple does not support progress parameters via [Wayback] macos dd progress – Google Search:

  • [Wayback] Quick: dd with progress indication on macOS

    A nice way I found to get progress indication whilst still being able to benefit from the huge speed increase in using /dev/rdiskX is to install a tool called pv, also known as [WayBack] Pipe Viewer.

    Example:

    sudo dd if=/dev/rdiskX bs=1m | pv -s 64G | sudo dd of=/dev/rdiskY bs=1m

    Another way to achieve something similar would be to use brew to install coreutils, which will come with a newer version of dd that supports the status option.

    Example:

    gdd if=/dev/diskX of=/dev/diskY bs=1m status=progress

  • [Wayback] el capitan – How can I track progress of dd – Ask Different

    You just need to enter a controlT character from the keyboard while the dd command is executing.

    By pressing the controlT character, you are sending the same SIGINFO signal to the dd command that the command pkill -INFO -x dd sends.

     

    dd itself doesn’t provide a progress bar. You may estimate the progress of the dd copy process by adding a pkill -INFO command though.

    Example:

    dd if=/dev/zero of=/dev/null bs=64m count=1000 & while pkill -INFO -x dd; do sleep 1; done
  • [Wayback] dd progress indicator on OSX

    signal siginfo is coupled to key-combination CTRL-T. No need to use kill, you can just type CTRL-T in the terminal window where dd is running.

–jeroen

Posted in *nix, *nix-tools, Apple, iMac, Mac, Mac OS X / OS X / MacOS, MacBook, MacBook Retina, MacBook-Air, MacBook-Pro, MacMini, Power User | Leave a Comment »