Archive for the ‘ash/dash development’ Category
Posted by jpluimers on 2022/01/25
In Determining the ESXi installation type (2014558) | VMware KB, I also showed how to backup the configuration and download it.
Sometimes you want an ISO 8601 time-stamped local tarball just in case you want to revert to it at a later stage.
First a small recap on how to get the tarball, download location and temporary location in the first place (it will be automatically deleted from the temporary location):
# vim-cmd hostsvc/firmware/sync_config
# vim-cmd hostsvc/firmware/backup_config
Bundle can be downloaded at : http://*/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz
# find /scratch/downloads/ -name *.tgz
/scratch/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz
Goal is to get the download filename and save it to a different folder and embed the ISO 8601 timestamp in the filename.
Like many scripts, sed and regular expressions come to the rescue once more, just like in ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes (which we will need anyway because of the ISO 8601 time stamp, and a bit of fiddling at regex101.com/r/NyrzKF
# SCRATCH_CONFIG_BUNDLE_NAME=$(vim-cmd hostsvc/firmware/backup_config | sed -n -E -e "s/^(Bundle can be downloaded at : http://*)(/downloads/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}/configBundle-.+?)(.tgz)$//scratch23/p")
# echo "SCRATCH_CONFIG_BUNDLE_NAME: '${SCRATCH_CONFIG_BUNDLE_NAME}'"
SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz'
# date --utc -I'seconds' --reference "${SCRATCH_CONFIG_BUNDLE_NAME}"
2021-05-09T17:44:42UTC
Note:
Not few people have bumped into this, the only other I could find through [Wayback] “vim.fault.TooManyWrites” “syncConfiguration” – Google Search is [Archive.is] mal wieder purple Screen – VMware-Forum.
Figuring out the various parts of the SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz' is like at regex101.com/r/J4yU72, regex101.com/r/uID9xs and regex101.com/r/o8a4Am:
CONFIG_BUNDLE_DIRECTORY_NAME=$(echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\1/p")
CONFIG_BUNDLE_FILE_NAME=$( echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\2/p")
CONFIG_BUNDLE_DOT_EXTENSION=$( echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\3/p")
echo "CONFIG_BUNDLE_DIRECTORY_NAME: '${CONFIG_BUNDLE_DIRECTORY_NAME}'"
echo "CONFIG_BUNDLE_FILE_NAME: '${CONFIG_BUNDLE_FILE_NAME}'"
echo "CONFIG_BUNDLE_DOT_EXTENSION: '${CONFIG_BUNDLE_DOT_EXTENSION}'"
Output is like this:
SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/configBundle-ESXi-X10SRH-CF.tgz'
CONFIG_BUNDLE_DIRECTORY_NAME: '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/'
CONFIG_BUNDLE_FILE_NAME: 'configBundle-ESXi-X10SRH-CF'
CONFIG_BUNDLE_DOT_EXTENSION: '.tgz'
Full backup-config-to-ESXi_configuration_backup-directory.sh script:
Read the rest of this entry »
Posted in *nix, *nix-tools, ash/dash, ash/dash development, cron/crontab, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/01/19
A few links and notes:
- [Wayback] Changing the hostname of an ESX or ESXi host (1010821)
Run these commands to change the hostname in ESXi 5.x, ESXi 6.x,ESXi 7.x, using the command line:
- esxcli system hostname set –host=hostname
- esxcli system hostname set –fqdn=fqdn
- [Wayback] ESX Host appears as localhost.localdomain in VMware Infrastructure/vSphere client (2009720)
The name resolution parameters were not properly configured during the installation of the ESX host.
- [Wayback] Domain repoint for embedded vCenter Server fails with error: “domain_consolidator Failed to set machine id” (71020)
This issue is caused by a mismatch between the FQDN that was configured as the PNID during the vCenter Server deployment and the hostname that is currently configured.
I had a mismatch happen because of the second entry: a host configured in a different domain than it was deployed to.
Here are the commands to list and change the hosts name, domain and fqdn:
Read the rest of this entry »
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/01/18
Since ESXi boots from RAM, most files in /etc are not persisted after modification.
The files that are persisted, are only persisted once every hour by auto-backup.sh, so better run auto-backup.sh by hand if you want to reboot after changing them.
The auto-backup.sh script is ran every hour at 1 minute past the hour as per below crontab.
Default ESXi crontab in /var/spool/cron/crontabs/root:
#min hour day mon dow command
1 1 * * * /sbin/tmpwatch.py
1 * * * * /sbin/auto-backup.sh
0 * * * * /usr/lib/vmware/vmksummary/log-heartbeat.py
*/5 * * * * /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
00 1 * * * localcli storage core device purge
Schedules deciphered via [Wayback] Crontab.guru – The cron schedule expression editor:
- [Wayback] Every day at
01:01: “This module removes stale temporary files”
1 1 * * * /sbin/tmpwatch.py
- [Wayback] Every hour at
*:01: saves backup to /bootbank/state.tgz.
1 * * * * /sbin/auto-backup.sh
- [Wayback] Every hour at
*:00 logs heartbeat messages to /var/log/vmksummary.log like 2021-02-23T19:00:02Z heartbeat: up 577d2h37m16s, 9 VMs; [[2802426 vmx 4194304kB] [6176344 vmx 4194304kB] [68997 vmx 8388608kB]] [[2802426 vmx 0%max] [6176344 vmx 0%max] [68997 vmx 0%max]]
0 * * * * /usr/lib/vmware/vmksummary/log-heartbeat.py
- [Wayback] Every
5th minute logs to /var/log/hostd-probe.log.
*/5 * * * * /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
- [Wayback] Every day at
01:00: Removes storage devices which have not been seen in some time interval.
00 1 * * * localcli storage core device purge
Note that localcli commands are the same as esxcli; for esxcli, a running hostd is required; localcli can run without hostd. See:
–jeroen
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware | Leave a Comment »
Posted by jpluimers on 2022/01/17
On VMware ESXi, with the Busybox ash/dash shell, I wanted to get the current UTC timestamp in ISO 8601 format without dashes (-) or especially colons (:) and plus-signs (+) you have to back-slash escape colons or double quote parameters, which is often can be a pain).
- [Wayback] How to Manipulate Filenames Having Spaces and Special Characters in Linux describes the pain handling special characters:
In this article, we will see how to create, copy, move and delete filenames that starts with spaces and special characters (say #, *, &, =, etc.) in Linux.
- [Wayback] filenames – What characters are safe in cross-platform file names for Linux, Windows and OS-X – Super User
…
[]()^ #%&!@:+={}'~ and [`] all have special meanings in many shells, and are annoying to work around, and so should be avoided. They also tend to look horrible in URLs.
…
This is why we can’t have good things: Getting the UTC 8610 timestamp was far less easy than I hoped for.
First of all, Busybox only allows for a precision of seconds, not milliseconds, and the specification format needs better documentation as per [Wayback] embedded linux – How to get ISO8601 seconds format from “date” in busybox? – Stack Overflow:
Read the rest of this entry »
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2021/12/21
This works fine on “BusyBox v1.29.3 (2019-05-21 15:22:06 PDT) multi-call binary.” that is included with VMware ESXi 6.5 update 3:
[Wayback] bash – Hexadecimal To Decimal in Shell Script – Stack Overflow
Dealing with a very lightweight embedded version of busybox on Linux means many of the traditional commands are not available (bc, printf, dc, perl, python)
echo $((0x2f))
47
hexNum=2f
echo $((0x${hexNum}))
47
Credit to [Wayback] Peter Leung for this solution.
–jeroen
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, Power User, Scripting, Software Development | Leave a Comment »
Posted by jpluimers on 2021/10/28
After doing a lot of – historically grown – dash scripting for ESXi, I found out there is Python available on ESXi:
- Python 3.5.10 on VMware ESXi 6.7.0 build-17700523 (VMware ESXi 6.7.0 Update 3)
- Python 3.5.6 on VMware ESXi 6.5.0 build-13932383 (VMware ESXi 6.5.0 Update 3)
- VMware 7: to be determined.
Yes I know that Python 3.5 is end-of-life (and 3.5.10 was the latest version), but it is a lot better than shell scripts.
So now some links for my list of things to try in order to use Python for scripting ESXi operations:
–jeroen
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, Power User, Python, Scripting, Software Development | Leave a Comment »
Posted by jpluimers on 2021/10/27
Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2021/10/26
This is sort of a follow-up on VMware ESXi console: viewing all VMs, suspending and waking them up: part 4 which already gave part of the configuration details of all the configured VMs.
Back then, we ended with this:
List the vmid values, power status and name of all VMs
Back to the listing script vim-cmd-list-all-VMs.sh:
#!/bin/sh
# https://wiert.me/2021/04/29/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-4/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
done
It uses vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and vim-cmd vmsvc/get.config with some sed and a for loop from dash to generate a nice list of information.
A long time ago, I already figured out that vim-cmd vmsvc/get.guest # gives all guest information including network information for a running VM that has either VMware Tools or open-vm-tools running (see VMware ESXi console: viewing all VMs, suspending and waking them up: part 3 for the difference between these two tools).
A full output of a sample VM is below the signature.
There are a few places that have the LAN ipAddress. For now, I choose to use only the IPv4 main address from ipAddress, which is in between (vim.vm.GuestInfo) { and net = (vim.vm.GuestInfo.NicInfo) [.
I modified the above script to become this:
Read the rest of this entry »
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, find, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2021/10/26
I needed a way to append the directory of a script to the path as all my tool scripts are in there, and I did not want to modify any profile scripts as these might be modified during ESXi upgrade.
First you need the full script filename through readlink then toe parent directory name through dirname:
Note there might be dragons with more symlinks or different shells:
I created the script below. It is not perfect, but for my situation it gets the job done.
If you do not start a new shell, then the export is lost as a new dash shell process is started for each script that runs from the terminal or console.
# cat /opt/bin/append-script-directory-to-path-and-start-new-shell.sh
#!/bin/sh
# Absolute path to this script, e.g. /home/user/bin/foo.sh
# echo "'$0'"
SCRIPT=$(readlink -f "$0")
# Absolute path this script is in, thus /home/user/bin
SCRIPTPATH=$(dirname "$SCRIPT")
# echo Appending to $PATH: $SCRIPTPATH
export PATH=$PATH:$SCRIPTPATH
sh
–jeroen
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2021/08/30
Reminder to self: create a static ESXi binary for a recent rsync release.
Quite a few people have bumped into rsync erroring out with “large” sets of files (where large can be as low as ~1000), like for instance Tj commenting on my post “ESXi 5.1 and rsync – damiendebin.net.”:
ERROR: out of memory in receive_sums [sender] │······
rsync error: error allocating core memory buffers (code 22) at util2.c(102) [sender=3.1.1] │······
rsync: [generator] write error: Broken pipe (32) │······
I bumped into this myself as well, even when updating from rsync 3.1.0 to 3.1.2.
There are various static rsync for ESXi around. Just a few of them for completeness:
There is also 3.0.9 (via [Wayback] VMware ESXi 5.1 rsync 3.0.9 statically linked binary erstellen – bachmann-lan.de), but it has a VMFS bug ([Wayback] 8177 – Problems with big sparsed files) as per [Wayback] ESXi 5.1 and rsync – damiendebin.net.)
The good news is that it is fixed in 3.2.2 as a user-configurable setting, but since there is no ESXi build yet (see reminder above)…
Anyway: [Wayback] 12769 – error allocating core memory buffers (code 22) depending on source file system
This is what happens with 3.1.2 and 3.1.3:
time rsync -aiv --info=progress2 --progress --partial --existing --inplace /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
sending incremental file list
0 0% 0.00kB/s 0:00:00 (xfr#0, ir-chk=1000/1259)
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]
Command exited with non-zero status 22
real 0m 0.87s
user 0m 0.10s
sys 0m 0.00s
time rsync -aiv --info=progress2 --progress --partial --ignore-existing --sparse /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
sending incremental file list
0 0% 0.00kB/s 0:00:00 (xfr#0, ir-chk=1000/1259)
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]
Command exited with non-zero status 22
real 0m 0.28s
user 0m 0.12s
sys 0m 0.00s
Finished
I was lucky that [Wayback] “rsync error: error allocating core memory buffers” protocol version “3.1.2” – Google Search got me a result so quickly: add a --protocol-29 and you are set.
The first result (Wayback has the results reversed from what got) didn’t fix it. The second did.
- [Wayback] 225761 – net/rsync long path causes buffer overflow (update to 3.1.3)
- [Wayback/Archive.is] AIX Open Source – IBM Power Systems Community: rsync out of memory
As a work around, I added “--protocol=29” to one of our servers that was consistently failing with “ERROR: out of memory in flist_expand [receiver]” “rsync error: error allocating core memory buffers (code 22) at util2.c(105) [receiver=3.1.3]” in rsync-3.1.3-2.ppc
I read the man page and started experimenting with the protocol version until I lowered it enough to get it to work consistently.
The problem might be that running on the ESXi gives you limited memory, but then some 10k files should not use more than like half a megabyte of memory.
Sometime I will dig deeper into the protocol version differences, for now a list of files I think will be relevant for that (mainly look for protocol_version):
Some web pages mentioning the --protocol option and might give me more insight in the protocol differences:
With --protocol=29, time estimation is way off, but there are no errors:
time rsync -aiv --info=progress2 --progress --partial --existing --inplace --protocol=29 /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
building file list ...
9059 files to consider
.d..t...... isos/
27,593 0% 0.00kB/s 0:00:06 (xfr#1, to-chk=0/9059)
sent 212,594 bytes received 268 bytes 20,272.57 bytes/sec
total size is 3,055,677,645,398 speedup is 14,355,204.99
real 0m 13.31s
user 0m 1.35s
sys 0m 0.00s
time /vmfs/volumes/5791a3e1-0b9368de-4965-0cc47aaa9742/local-bin/rsync -aiv --info=progress2 --progress --partial --ignore-existing --sparse --protocol=29 /vmfs/volumes/Samsung850-2TB-S3D4NX0HA01043L/ Samsung850-2TB-S3D4NX0HA01043L/
building file list ...
9059 files to consider
>f+++++++++ isos/EN-Windows-XP-SP3-VL.iso
...
cd+++++++++ ESXi65.filesystem-root/usr/share/
216,868,164,639 7% 40.64MB/s 1:24:48 (xfr#2571, to-chk=0/9059)
sent 216,894,938,870 bytes received 57,858 bytes 42,582,702.80 bytes/sec
total size is 3,055,677,645,398 speedup is 14.09
real 1h 24m 58s
user 34m 5.59s
sys 0m 0.00s
Finished
Even not on ESXi, there were just a few people bumping into this, so I wonder why there are so few matches on [Wayback] “ERROR: out of memory in flist_expand [sender]” “sender=3.1” – Google Search:
–jeroen
Posted in *nix, *nix-tools, ash/dash development, Development, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, rsync, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
The Wiert Corner - irregular stream of stuff 