The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,862 other subscribers

Archive for the ‘*nix’ Category

VMware ESXi console: viewing all VMs, suspending and waking them up: part 2

Posted by jpluimers on 2021/04/27

Last week ended up to be a kind of VMware ESXi heavey, and this week will be similar. So it is time for following up on VMware ESXi console: viewing all VMs, suspending and waking them up: part 1.

That one ended with

Then we get this to print some of the capture groups:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/Vmid:\1 Guest:\6 Version:\7 Name:\3 Datastore:\4 File:\5 Annotation:\8/p"

With this output:

Vmid:10 Guest:windows9_64Guest Version:vmx-14 Name:X9SRI-3F-W10P-EN-MEDIA Datastore:[EVO860_500GB] File:VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx Annotation:
Vmid:5 Guest:centos64Guest Version:vmx-11 Name:PPB Local_Virtual Machine_v4.0 Datastore:[EVO860_500GB] File:VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx Annotation:PowerPanel Business software(Local) provides the service which communicates

Figuring out power state for each VM

This will be in the next installment, as by now this already has become a big blog-post (:

I basically went the vim-cmd vmsvc way instead of the esxcli vm way. My motivation was the easier to understand ID values. They are the basis of virtually all vim-cmd vmsvc based commands:

# vim-cmd vmsvc --help
Commands available under vmsvc/:
acquiremksticket                 get.snapshotinfo                 
acquireticket                    get.spaceNeededForConsolidation  
createdummyvm                    get.summary                      
destroy                          get.tasklist                     
device.connection                getallvms                        
device.connusbdev                gethostconstraints               
device.ctlradd                   message                          
device.ctlrremove                power.getstate                   
device.disconnusbdev             power.hibernate                  
device.diskadd                   power.off                        
device.diskaddexisting           power.on                         
device.diskextend                power.reboot                     
device.diskremove                power.reset                      
device.getdevices                power.shutdown                   
device.nvdimmadd                 power.suspend                    
device.nvdimmremove              power.suspendResume              
device.toolsSyncSet              queryftcompat                    
devices.createnic                reload                           
get.capability                   setscreenres                     
get.config                       snapshot.create                  
get.config.cpuidmask             snapshot.dumpoption              
get.configoption                 snapshot.get                     
get.datastores                   snapshot.remove                  
get.disabledmethods              snapshot.removeall               
get.environment                  snapshot.revert                  
get.filelayout                   snapshot.setoption               
get.filelayoutex                 tools.cancelinstall              
get.guest                        tools.install                    
get.guestheartbeatStatus         tools.upgrade                    
get.managedentitystatus          unregister                       
get.networks                     upgrade                          
get.runtime                      

My “goto” for getting information is [Wayback] “vim-cmd vmsvc” site:vmware.com – Google Search, and a few sample pages are here:

  1. [Wayback] Performing common virtual machine-related tasks with command-line utilities (2012964) (showing that there are many tasks only vim-cmd vmsvc can do, but esxcli vm cannot)
  2. [Wayback] Powering on a virtual machine from the command line when the host cannot be managed using vSphere Client (1038043) (showing how to combine vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and vim-cmd vmsvc/power.on)
  3. [Wayback] Determine the power status of a virtual machine on an ESX or ESXi host (1003737) (showing vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and ps –auxwww | grep –i VM_NAME)
  4. [Wayback] Collecting information about tasks in VMware ESXi/ESX (1013003) (showing the relation between VMs and tasks using  vim-cmd vimsvc/task_list, vim-cmd vmsvc/getallvms and vim-cmd vimsvc/task_info)
  5. [Wayback] Unable to Power off a Virtual Machine in an ESXi host (1014165) (focussing on vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate, vim-cmd vmsvc/power.shutdown and vim-cmd vmsvc/power.off)
  6. [Wayback] Reloading a vmx file without removing the virtual machine from inventory (1026043) (showing vim-cmd vmsvc/getallvms and vim-cmd vmsvc/reload)
  7. [Wayback] Investigating virtual machine file locks on ESXi hosts (10051) (trying to show how to combine vim-cmd vmsvc/getallvms, grep, awk, find and xargs to find vmdk files, but fails because of parsing errors)

The pattern above is that most of the vim-cmd vmsvc examples are for power state and tasks. Not fully sure why, but my guess is it is what most people use it for. That kind of use what this series of posts also focuses on too, but certainly not the only use. Read the first numbered entry above to get a full grasp of what is possible. I hope to find time in the future to show some more examples outside the power and task realms.

Basically the only time you need to check out esxcli with VMs is when you cannot shut down a VM in a normal way. These links explain what to do in that case:

So let’s go back to basics, and start with getting info on all vim-cmd vmsvc commands.

Help on all vim-cmd vmsvc commands

Executing vim-cmd help vmsvc (preferred) or vim-cmd help vmsvc --help gives you all commands prepended with the line Commands available under vmsvc/:.

Executing vim-cmd help vmsvc/command prints the help for a single command (but vim-cmd help vmsvc/command -help first prints an error, then the help).

Here are the steps how I got the help help for all commands.

First I needed a list of all commands. This is already a multi-stage process, so below the full command I will explain the bits.

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort
  1. vim-cmd help vmsvc gives all the commands (two per line!) prepended by the line Commands available under vmsvc/:.
  2. sed '1d' stripts that line.
  3. xargs -n 1 -r echo does a lot of things:
    1. It parses the sed '1d' input line by line, splits each line into parts, combines all the parts, then executes echo with the combined parts
    2. The -n 1 ensures each invocation of echo takes only a single one of the combined parts
    3. -r is just a protection: if there is no input, then echo is never executed, resulting in empty output
  4. sort will sort all the combined output of all the echo invocations to undo the horizontal combination of parts that xargs did

Now getting the help is doing more of the above, with some more bits to explain:

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} vim-cmd help vmsvc/{}
  1. Normally, xargs will execute each command by appending the parameter inserting a space in front of each parameter
  2. -I {} will force xargs to put each argument just as is in the place where {} is used in the argument
  3. This executes vim-cmd help vmsvc/command in stead of vim-cmd help vmsvc/ command

The result is a long blob of text that is very hard to read as there are no separators between the commands. I saved it as a [Wayback] vim-cmd help for each vmsvc command.txt gist.

With a sh -c shell trick, you can add some more information and separation to the output by embedding :

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} sh -c 'echo "-----" ; echo "help for vim-cmd help vmsvc/{}" ; echo ; vim-cmd help vmsvc/{}'

I have added the output to the [Wayback] delimited vim-cmd help for each vmsvc command.txt gist.

Commands taking a vmid parameter

Now that we know how to output all help, we can filter on it.

An interesting one is to filder only commands taking a vmid parameter:

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} vim-cmd help vmsvc/{} | grep -iw vmid

On VMware ESXi 6.7, this gets you the list:

Usage: acquiremksticket vmid
Usage: acquireticket vmid ticketType
Usage: destroy vmid
Usage: device.connection vmid deviceKey connect
Usage: device.connusbdev vmid usbid
Usage: device.ctlradd vmid ctlr_type bus_number
Usage: device.ctlrremove vmid ctlr_type bus_number
Usage: device.disconnusbdev vmid usbid
Usage: device.diskadd vmid size controller_numer unit_number datastore [ctlr_type]
Usage: device.diskaddexisting vmid disk_file controller_number unit_number [ctlr_type]
Usage: device.diskextend vmid new_size controller_numer unit_number [ctlr_type]
Usage: device.diskremove vmid controller_number unit_number delete_file [controller_type]
Usage: device.getdevices vmid
Usage: device.nvdimmadd vmid size
Usage: device.nvdimmremove vmid deviceKey
Usage: device.toolsSyncSet vmid new state
Usage: devices.createnic vmid adapter-type network-id [network-type]
Usage: get.capability vmid
Usage: get.config vmid
Usage: get.config.cpuidmask vmid
Usage: get.configoption vmid
Usage: get.datastores vmid
Usage: get.disabledmethods vmid
Usage: get.environment vmid
Usage: get.filelayout vmid
Usage: get.filelayoutex vmid
Usage: get.guest vmid
Usage: get.guestheartbeatStatus vmid
Usage: get.managedentitystatus vmid
Usage: get.networks vmid
Usage: get.runtime vmid
Usage: get.snapshotinfo vmid
Usage: get.spaceNeededForConsolidation vmid
Usage: get.summary vmid
Usage: get.tasklist vmid
Usage: message vmid [messageId] [messageChoice]
Usage: power.getstate vmid
Usage: power.hibernate vmid
Usage: power.off vmid
Usage: power.on vmid
Usage: power.reboot vmid
Usage: power.reset vmid
Usage: power.shutdown vmid
Usage: power.suspend vmid
Usage: power.suspendResume vmid
Usage: queryftcompat vmid [faultToleranceType]
Usage: reload vmid
Usage: setscreenres vmid width height
Usage: snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory] [quiesced]
Usage: snapshot.get vmid
Usage: snapshot.remove vmid snapshotId [removeChildren]
Usage: snapshot.removeall vmid
Usage: snapshot.revert vmid snapshotId suppressPowerOn
Usage: snapshot.setoption [OPTIONS] vmid
Usage: tools.cancelinstall vmid
Usage: tools.install vmid
Usage: tools.upgrade vmid [args]
Usage: unregister vmid
Usage: upgrade vmid [vm_hwversion]

In the above list, the bold entries have to do with power, that is what this series is supposed to center around, so more on that tomorrow.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Awk, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, fgrep, Power User, Scripting, sed, sed script, sh, Sh Shell, Software Development, sort, Virtualization, VMware, VMware ESXi, xargs | Leave a Comment »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 1

Posted by jpluimers on 2021/04/22

I think the easiest way to list all VMs is the vim-cmd vmsvc/getallvms command, but it has a big downside: the output is a mess.

The reason is that the output:

  • has a lot of columns (Vmid, Name, Datastore, File, Guest OS, Version, Annotation),
  • more than 500 characters per line (eat that 1080p monitor!),
  • and potentially more than one line per VM as the Annotation is a free-text field that can have newlines.

Example output on one of my machines:


Vmid Name File Guest OS Version Annotation
10 X9SRI-3F-W10P-EN-MEDIA [EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx windows9_64Guest vmx-14
5 PPB Local_Virtual Machine_v4.0 [EVO860_500GB] VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx centos64Guest vmx-11 PowerPanel Business software(Local) provides the service which communicates
with the UPS through USB or Serial cable and relays the UPS state to each Remote on other computers
via a network.
It also monitors and logs the UPS status. The computer which has been installed the Local provides
graceful,
unattended shutdown in the event of the power outage to protect the hosted computer.

As an alternative, you could use esxcli vm process list, but that gives IDs that are way harder to remember:


PPB Local_Virtual Machine_v4.0
World ID: 2099719
Process ID: 0
VMX Cartel ID: 2099713
UUID: 56 4d 74 f8 c8 22 41 27-a3 88 49 df 8b dc d6 63
Display Name: PPB Local_Virtual Machine_v4.0
Config File: /vmfs/volumes/5d35e7d8-e8df636f-46b9-0025907d9d5c/VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx
X9SRI-3F-W10P-EN-MEDIA
World ID: 2099728
Process ID: 0
VMX Cartel ID: 2099717
UUID: 56 4d 51 ac f6 cf e4 0b-b6 86 2f 53 a2 8a 4b ea
Display Name: X9SRI-3F-W10P-EN-MEDIA
Config File: /vmfs/volumes/5d35e7d8-e8df636f-46b9-0025907d9d5c/VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx

I got both of the above commands from [Wayback] VMware Knowledge Base: Performing common virtual machine-related tasks with command-line utilities (2012964).

Back to the columns that vim-cmd vmsvc/getallvms returns:

  • Vmid is an unsigned integer
  • Name can have spaces
  • Datastore has square brackets [ and ] around it
  • File can contain spaces
  • Guest OS is an identifier without spaces (it is a value from [Wayback] the vSphere API VcVirtualMachineGuestOsIdentifier
  • Version looks like vmx-# where # is an unsigned integer
  • Annotation is multi-line free-form so potentially can have lines starting like being Vmid, but the chance that a line looks exactly like a non-annotated one is very low

So let’s find a grep or  sed filter to get just the lines without annotation continuations. Though in general I try to avoid regular expressions as they are hard to both write and read, but with Busybox there is no much choice.

I choose sed, just in case I wanted to do some manipulation in addition to matching.

Busybox sed

Though the source code [Wayback] sed.c\editors – busybox – BusyBox: The Swiss Army Knife of Embedded Linux indicates sed.c - very minimalist version of sed, the implementation actually is reasonably feature rich, just not feature complete. That’s OK given the aim of Busybox to be small.

Luckily, deep in the busybox sed code, it indicates that extended regular expressions are supported (support is in [Wayback] /uClibc/plain/libc/misc/regex/regcomp.c (look for regcomp, do not get confused by xregcomp on call sites as that is [Wayback] just a tiny wrapper to call regcomp).

The support has become better over time, like [Wayback] gnu – sed Command on BusyBox expects different syntax? – Super User shows.

This means far less escaping than basic regular expressions, capture groups are supported as well as character classes (so [[:digit:]] is more readable than [0-9]), and the + is supported to match once or more (so [0-9]+ means one or more digits, as does [[:digit:]]+, but [d]+ or \d+ don’t ). Unfortunately named capture groups are not supported (so documenting parts of the regular expression like (?<Vmid>^[[:digit:]]+) is not possible, it will give you an error [Wayback] Invalid preceding regular expression).

But first a few of the sed commandline options and their order:

vim-cmd vmsvc/getallvms | sed -n -E -e '/(^[[:digit:]]+)/p'
  1. -n outputs only matching lines that have a p print command.
  2. -E allows extended regular expressions (you can also use -r for that)
  3. -e adds a (in this case extended) regular expression
  4. '/(^[[:digit:]]+)/p' is the extended regular expression embedded in quotes
    1. / at the start indicates that sed should match the regular expression on each line it parses
    2. /p at the end indicates the matching line should be printed
    3. Parentheses ( and ) surround a capture group
    4. ^[[:digit:]]+ matches 1 or more digits at the start of the line

The grep command is indeed much shorter, but does not allow post-editing:

vim-cmd vmsvc/getallvms | grep -E '(^[[:digit:]]+)'

Building a sed filter

I came up with the below sed regular expression to filter out lines:

  1. starting with a Vmid unsigned integer
  2. having a [Datastore] before the File
  3. have a Guest OS identifier after File
  4. have a Version matching vmx-# after File where # is an unsigned integer
  5. optionally has an Annotation after Version
vim-cmd vmsvc/getallvms | sed -n -E -e  "/^([[:digit:]]+)(\s+)((\S.+\S)?)(\s+)(\[\S+\])(\s+)(.+\.vmx)(\s+)(\S+)(\s+)(vmx-[[:digit:]]
+)(\s*?)((\S.+)?)$/p"

A longer expression that I used to fiddle around with is at regex101.com/r/A7MfKu and contains named capture groups. I had to nest a few groups and use the ? non-greedy (or lazy) operator a few times to ensure the fields would not include the spaces between the columns.

Others use different expressions as for instance explained in [Wayback] Get all VMs with “vmware-vim-cmd vmsvc/getallvms” – VMware Technology Network VMTN:

Output from “vim-cmd vmsvc/getallvms” is really challenging to process. Our normal approaches such as awk column indexes, character index, and regular expression are all error prone here. The character index of each column varies depending on maximum field length of, for example, VM name. And the presence of spaces in VM names throws off processing as awk columns. And VM name could contain almost any character, foiling regex’s.

Printing capture groups

The cool thing is that it is straightforward to modify the expression to print any of the capture groups in the order you wish: you convert the match expression (/match/p) into a replacement expression (s/match/replace/p) and print the required capture groups in the replace part. A short example is at [Wayback] regex – How to output only captured groups with sed? – Stack Overflow.

There is one gotcha though: Busybox sed only allows single-digit capture group numbers, and we have far more than 9 capture groups. This fails and prints 0 after the output of capture group 1 instead of printing capture group 10, similar for 2 after group 1 instead of printing group 12:

vim-cmd vmsvc/getallvms | sed -n -E -e  "s/^([[:digit:]]+)(\s+)((\S.+\S)?)(\s+)(\[\S+\])(\s+)(.+\.vmx)(\s+)(\S+)(\s+)(vmx-[[:digit:]]+)(\s*?)((\S.+)?)$/Vmid:\1 Guest:\10 Version:\12 Name:\3 Datastore:\7 File:\8/p"

So we need to cut down on capture groups first by removing all capture groups around the \s white-space matching:

vim-cmd vmsvc/getallvms | sed -n -E -e  "/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/p"

Then we get this to print some of the capture groups:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/Vmid:\1 Guest:\6 Version:\7 Name:\3 Datastore:\4 File:\5 Annotation:\8/p"

With this output:

Vmid:10 Guest:windows9_64Guest Version:vmx-14 Name:X9SRI-3F-W10P-EN-MEDIA Datastore:[EVO860_500GB] File:VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx Annotation:
Vmid:5 Guest:centos64Guest Version:vmx-11 Name:PPB Local_Virtual Machine_v4.0 Datastore:[EVO860_500GB] File:VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx Annotation:PowerPanel Business software(Local) provides the service which communicates

Figuring out power state for each VM

This will be in the next installment, as by now this already has become a big blog-post (:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, RegEx, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

VMware ESXi 6 and 7: checking and setting/clearing maintenance mode from the console

Posted by jpluimers on 2021/04/21

Every now and then it is useful to be able to do maintenance work from the ESXi console addition to the ESXi web-user interface.

I know there are many sites having this information, but many of them forgot to format the statements with code markup, so parameters with two dashes -- (each a Wayback Unicode Character ‘HYPHEN-MINUS’ (U+002D)) now have become an [Wayback] Unicode Character ‘EN DASH’ (U+2013) which is incompatible with most console programs, especially the ESXi ones (as they are Busybox based to minimise footprint).

Note you can use this small site (which runs in-browser, so does not phone home) to get the unicode code points for any string: [Wayback] What Unicode character is this ?.

Links like below (most on the vmware.com domain) have this EN DASH and make me document things on my blog instead of trying code directly from blogs or forum posts:

So below are three commands I use that have to do with the maintenance mode (the mode that for instance you can use to update an ESXi host to the latest patch level).

    1. Check the maintenance mode (which returns Enabled or Disabled):
      esxcli system maintenanceMode get
    2. Enable maintenance mode (which returns nothing when succeeded, and Maintenance mode is already enabled. when failed):
      esxcli system maintenanceMode set --enable true
    3. Disable maintenance mode (which returns nothing when succeeded, and Maintenance mode is already disabled. when failed):
      esxcli system maintenanceMode get

Some examples, especially an the various output possibilities (commands in bold, output in italic):

# esxcli system maintenanceMode get
Disabled
# esxcli system maintenanceMode set --enable false
Maintenance mode is already disabled.
# esxcli system maintenanceMode set --enable true 
# esxcli system maintenanceMode get
Enabled
# esxcli system maintenanceMode set --enable true
Maintenance mode is already enabled.
# esxcli system maintenanceMode set --enable false
# esxcli system maintenanceMode get
Disabled

I made these scripts for this:

  • esxcli-maintenanceMode-show.sh:
    #!/bin/sh
    esxcli system maintenanceMode get
  • esxcli-maintenanceMode-enter.sh:
    #!/bin/sh
    esxcli system maintenanceMode set --enable true
  • esxcli-maintenanceMode-exit.sh:
    #!/bin/sh
    esxcli system maintenanceMode set --enable false

Note I have not checked the exit codes for these esxcli commands yet, but did blog about how to do that: Busybox sh (actually ash derivative dash): checking exit codes.

–jeroen

Posted in BusyBox, Development, Encoding, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Software Development, Unicode, Virtualization, VMware, VMware ESXi | Leave a Comment »

Busybox sh (actually ash derivative dash): checking exit codes

Posted by jpluimers on 2021/04/20

Even if you include a double quotes "sh" in a Google search to force only sh (in the early days this was the Thompson shell, but nowadays usually a Bourne shell or derivative) results, almost all unix like scripting examples you find are based on bash (the Bourne again shell), so I was glad I dug a bit deeper into what the actual Busybox shell is.

I wanted to know which shell Busybox uses and what capabilities it has, as ESXi ships with this very slimmed down set of tools (called applets in Busybox speak).

It does not even include ssh: that gap is often filled by [Wayback] Dropbear SSH, which was used by ESXi and named dbclient (I think with ESXi 6.0 it was replaced with a more regular ssh implementation): [Wayback] How to compile a statically linked rsync binary for ESXi.

Busybox shell source code is at [Wayback] ash.c\shell – busybox – BusyBox: The Swiss Army Knife of Embedded Linux and indicates the shell is the ash (the Almquist shell) derivative dash (yes, you guessed it right: the Debian Almquist shell), ported from NetBSD and debianized:

 * Copyright (c) 1997-2005 Herbert Xu <herbert@gondor.apana.org.au>
 * was re-ported from NetBSD and debianized.
...
//config:   The most complete and most pedantically correct shell included with
//config:   busybox. This shell is actually a derivative of the Debian 'dash'
//config:   shell (by Herbert Xu), which was created by porting the 'ash' shell
//config:   (written by Kenneth Almquist) from NetBSD.

nx like systems have a shell hell similar to Windows DLL hell: there are too many, and their differences and be both subtle and frustrating. To get a feel, browse through Source: Comparison of command shells – Wikipedia (yes, some shells from other operating environments like DOS, OS/2, VMS and Windows, but the majority is nx).

Since ash is sufficiently different from bash (for example [Wayback] ash – exit code for a piped process), I always want to know what shell code (which often comes from bash as it is so ubiquitous) will work.

There is hardly any shell documentation at the Busybox site. There is [Wayback] BusyBox – The Swiss Army Knife of Embedded Linux, the source code at [Wayback] ash.c\shell – busybox – BusyBox: The Swiss Army Knife of Embedded Linux does not offer much either,

A manual page of it is at [Archive.is] ash(1) [minix man page]. There you see the age: back then, “exit status” is used where nowadays many people would use “exit code”. It does not explain how to check for specific exit codes.

Because ash is derived from the Bourne shell, this page was of great help for me to grasp exit code handing: [Wayback] Exit Codes – Shell Scripting Tutorial

A Bourne Shell Programming / Scripting Tutorial for learning about using the Unix shell.

Here two examples from that page to get me going:

#!/bin/sh
# Second attempt at checking return codes
grep "^${1}:" /etc/passwd > /dev/null 2>&1
if [ "$?" -ne "0" ]; then
  echo "Sorry, cannot find user ${1} in /etc/passwd"
  exit 1
fi
USERNAME=`grep "^${1}:" /etc/passwd|cut -d":" -f1`
NAME=`grep "^${1}:" /etc/passwd|cut -d":" -f5`
HOMEDIR=`grep "^${1}:" /etc/passwd|cut -d":" -f6`

echo "USERNAME: $USERNAME"
echo "NAME: $NAME"
echo "HOMEDIR: $HOMEDIR"

and

#!/bin/sh
# A Tidier approach

check_errs()
{
  # Function. Parameter 1 is the return code
  # Para. 2 is text to display on failure.
  if [ "${1}" -ne "0" ]; then
    echo "ERROR # ${1} : ${2}"
    # as a bonus, make our script exit with the right error code.
    exit ${1}
  fi
}

### main script starts here ###

grep "^${1}:" /etc/passwd > /dev/null 2>&1
check_errs $? "User ${1} not found in /etc/passwd"
USERNAME=`grep "^${1}:" /etc/passwd|cut -d":" -f1`
check_errs $? "Cut returned an error"
echo "USERNAME: $USERNAME"
check_errs $? "echo returned an error - very strange!"

This basically means that status code handling is the same as in bash, so constructs can be used like [Wayback] bash – How to check the exit status using an if statement – Stack Overflow:

$? is a parameter like any other. You can save its value to use before ultimately calling exit.

exit_status=$?
if [ $exit_status -eq 1 ]; then
    echo "blah blah blah"
fi
exit $exit_status

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, BusyBox, Development, Power User, Scripting, Software Development, ssh/sshd | 1 Comment »

Recovering files with scalpel.

Posted by jpluimers on 2021/04/19

I missed this 2014 article [WayBack] Recovering Deleted Files with Scalpel » Linux Magazine:

The Scalpel file carver helps users restore what they thought were lost files.

Via the now defunct G+ link: https://plus.google.com/+Doortodoorgeek/posts/eskyp8PH57a?_utm_source=1-2-2 from which I saved this quote:

+honkey Magoo recovering with Photorec can be hard, I had a touch more luck with this one

Scalpel File Carver: http://www.linux-magazine.com/Online/Features/Recovering-Deleted-Files-with-Scalpel

I wish it had been maintained longer, as the most recent changes are indeed from 2014: [WayBack] GitHub – sleuthkit/scalpel: Scalpel is an open source data carving tool. (it is now indeed part of Sleuthkit, see [WayBack] Scalpel – ForensicsWiki)

So basically this was a short revival: WayBack: Digital Forensics Solutions: Announcing Scalpel 2.0.

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, Apple, Mac OS X / OS X / MacOS, Power User | Leave a Comment »

Dockerfile with Bite Size Networking tools from b0rk

Posted by jpluimers on 2021/04/16

[WayBack] Ryan James Spencer on Twitter:

“I wrote a quick Dockerfile so people who purchase Bite Size Networking from  @b0rk can quickly have access to the tools. You can slim the image down to do debugging on docker networks once you get comfortable with which tools are most useful to you.”

He is planning to do more, so maybe a few of [WayBack] wizard zines get encapsulated into dockerfiles as well.

–jeroen

PS: [WayBack] Dockerfile | Docker Documentation


# N.B. The only tool missing here that is mentioned in the document is `zenmap`
# purely because this image is intended to be run via a CLI and `zenmap` is a GUI
# to `nmap` i.e. one can play around with the tools by running:
#
# $ docker build –name bite_size_networking:latest .
# $ docker run –rm -d –name bsn_test bite_size_networking:latest
# $ docker exec -it bsn_test bash
#
# Alternatively, one can change the `ENTRYPOINT` to `["bash"]` and run:
#
# $ docker run -it –name bsn_test bite_size_networking:latest
#
# then later (after exiting the shell):
#
# $ docker start bsn_test
# $ docker attach bsn_test
#
# One can also run this image on a docker network to capture packets and so
# forth for debugging purposes. Once you've found the tooling that best suits
# your needs, it may make sense to make a slimmed down version of this
# Dockerfile and, if wireguard isn't needed, base this image off
# `debian:stable` instead.
#
# Lastly, you can purchase Bite Size Networking or Julia's other fantastic
# zines over at https://wizardzines.com/
# We use `unstable` here since we install `wireguard` below
FROM debian:unstable
RUN apt update && \
DEBIAN_FRONTEND=noninteractive apt install -y \
dnsutils \
curl \
nmap \
tcpdump \
ngrep \
mitmproxy \
iptables \
ethtool \
stunnel \
hping3 \
lsof \
ipcalc \
p0f \
iperf \
apache2-utils \
wget \
python3 \
iftop \
nethogs \
iptraf \
httpie \
nload \
aria2 \
nftables \
tcpflow \
telnet \
openvpn \
links \
wireguard \
tshark
ENTRYPOINT ["sh", "-c", "while true; do sleep $(( 60 * 60 * 24 )); done"]

Posted in *nix, *nix-tools, Cloud, Containers, Docker, Infrastructure, Power User | Leave a Comment »

The Architecture of Open Source Applications: Sendmail

Posted by jpluimers on 2021/04/12

Cool historic article: [WayBack] The Architecture of Open Source Applications: Sendmail by Eric Allman.

It is Chapter 17 of this book [WayBack]:

The Architecture of
Open Source Applications

Amy Brown and Greg Wilson (eds.)
ISBN 978-1-257-63801-7

I totally missed that book being published in 2014.

Great historic read!

–jeroen

Posted in *nix, *nix-tools, History, Power User, sendmail | Leave a Comment »

linux – How can I execute a series of commands in a bash subshell as another user using sudo? – Stack Overflow

Posted by jpluimers on 2021/03/31

Based on [WayBack] linux – How can I execute a series of commands in a bash subshell as another user using sudo? – Stack Overflow:

alias restart-spotlight-service-as-root="sudo bash -c 'echo stop;launchctl stop com.apple.metadata.mds;echo start;launchctl start com.apple.metadata.mds;echo started'"

The bold bits above sudo bash -c 'echo stop;launchctl stop com.apple.metadata.mds;echo start;launchctl start com.apple.metadata.mds;echo started' allow the commands between single quotes to executed in one new bash shell under sudo.

–jeroen

Posted in *nix, *nix-tools, Apple, bash, bash, Development, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »

Disable ESXi Password Complexity – Perfect Cloud

Posted by jpluimers on 2021/03/29

Sometimes you have a long enough password, that matches with the confirmation, but pressing “Enter” to continue gives “Password does not have enough character types”:

From [WayBack] Disable ESXi Password Complexity – Perfect Cloud:

A part of my job as a VMware Certified Instructor is to update our lab systems whenever new vSphere versions come out.   After upgrading from 5.5 to 6.0 I decided we should change passwords, h…

This is the workflow:

  1. Make a backup of /etc/pam.d/passwd.
  2. Use vi to edit /etc/pam.d/passwd, and:
    1. Put a # in front of the lines starting with password requisite
    2. Remove the use_authtok bit of the line starting with password sufficient
    3. Put a # in front of the line starting with password required
    4. Quit vi while saving (press Esc, then enter :wq on the prompt)
  3. Change the password to a less secure one
  4. Restore the original /etc/pam.d/passwd.

Via: esxi 6 force short password – Google Search

Working around this on during ESXi installation fails

I tried this:

  1. Press Alt-F1 to go from the installation screen to the console screen
  2. Logon as root, with no password at all to get to the command-prompt:

  3. Perform the /etc/pam.d/passwd editing steps above
  4. Press Alt-F2 to go back to the install screen
  5. Enter root password

The password requirements stayed.

(more screenshots at [WayBack] ESXi 6.7 installation Guide – Let We-i Go)

Related

On my ESXI 6.5 system where the italic bit is removed, besides the two lines being commented out:

  1. original /etc/pam.d/passwd:
    #%PAM-1.0
    
    # Change only through host advanced option "Security.PasswordQualityControl".
    password   requisite    /lib/security/$ISA/pam_passwdqc.so retry=3 min=disabled,disabled,disabled,7,7
    password   sufficient   /lib/security/$ISA/pam_unix.so use_authtok nullok shadow sha512
    password   required     /lib/security/$ISA/pam_deny.so
    
  2. modified /etc/pam.d/passwd:
    #%PAM-1.0
    
    # Change only through host advanced option "Security.PasswordQualityControl".
    #password   requisite    /lib/security/$ISA/pam_passwdqc.so retry=3 min=disabled,disabled,disabled,7,7
    password   sufficient   /lib/security/$ISA/pam_unix.so nullok shadow sha512
    #password   required     /lib/security/$ISA/pam_deny.so
    

On my ESXI 6.7 system (which adds the bold lines below):

  1. original /etc/pam.d/passwd:
    #%PAM-1.0
    
    # Change only through host advanced option "Security.PasswordQualityControl".
    password   requisite    /lib/security/$ISA/pam_passwdqc.so retry=3 min=disabled,disabled,disabled,7,7
    
    # Change only through host advanced option "Security.PasswordHistory"
    password   requisite    /lib/security/$ISA/pam_pwhistory.so use_authtok enforce_for_root retry=2 remember=0
    
    password   sufficient   /lib/security/$ISA/pam_unix.so use_authtok nullok shadow sha512
    password   required     /lib/security/$ISA/pam_deny.so
    
  2. modified /etc/pam.d/passwd:
    #%PAM-1.0
    
    # Change only through host advanced option "Security.PasswordQualityControl".
    #password   requisite    /lib/security/$ISA/pam_passwdqc.so retry=3 min=disabled,disabled,disabled,7,7
    
    # Change only through host advanced option "Security.PasswordHistory"
    #password   requisite    /lib/security/$ISA/pam_pwhistory.so use_authtok enforce_for_root retry=2 remember=0
    
    password   sufficient   /lib/security/$ISA/pam_unix.so nullok shadow sha512
    #password   required     /lib/security/$ISA/pam_deny.so
    

–jeroen

Posted in *nix, ESXi6, ESXi6.5, ESXi6.7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

email file decoding: Encode/Decode Quoted Printable – Webatic

Posted by jpluimers on 2021/03/26

For my link archive: [WayBack] Encode/Decode Quoted Printable – Webatic.

It did a splendid job at decoding email files in MIME format Quoted-printable.

–jeroen

Posted in *nix, *nix-tools, Communications Development, Development, eMail, Encoding, Internet, Internet protocol suite, Power User, sendmail, SMTP, SocialMedia, Software Development | Leave a Comment »