December 2025
M	T	W	T	F	S	S
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31

Archive for the ‘sed’ Category

XKCD to ExplainXKCD Bookmarklet

Posted by jpluimers on 2025/01/21

Sometimes the XKCD episodes are hard to get, so I wanted a Bookmarklet to navigate to [Wayback/Archive] explain xkcd.

Here is the regular expression to change the domain name part if it matches: [Wayback/Archive] regex101: build, test, and debug regex: VI34VW with this generated sed code:

Read the rest of this entry »

Posted in *nix, *nix-tools, Bookmarklet, Development, JavaScript/ECMAScript, Power User, Scripting, sed, Software Development, Web Browsers, Web Development | Leave a Comment »

Zypper: list info on all patterns, so you can find out which pattern provides a package

Posted by jpluimers on 2022/05/11

I wanted to know which pattern provides [WayBack] etckeeper which is in the [WayBack] openSUSE Software package etckeeper.

It seems no built-in search query can do that, so I built one my own.

Since the result takes quite a while to produce, the output is a pattern.txt that you can manually search.

This is the command:

zypper search -t pattern | grep "|" | tail -n +2 | perl -pe 's/^.*? \| //' | perl -pe 's/ *\| .*$//' | xargs -I {} sh -c "zypper info -t pattern {}" > patterns.txt

The content is like this (the 2017 date shows I wrote this a long time ago):

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, etckeeper, Linux, Perl, Power User, Scripting, sed, Software Development | Leave a Comment »

Some notes on replacing parts of a text file with template text using sed on a Busybox system

Posted by jpluimers on 2022/03/17

Note before you think about putting stuff in /etc/rc.local.d/local.sh: that script will not be executed when UEFI booting.

In a very lightweight Busybox system, I wanted to modify some configuration files automatically using fragments stored in template files.

The system has diff, but no patch.

The basic idea is to use sed to insert the template files into certain spots of the configuration file when certain marker texts are not present. So I want the opposite of [Wayback] Hey Stephen Wood: Try patch instead of sed in shell scripts.

Basically the idea is a poor-man’s patch, described in Too bad: ESXi busybox has diff, but not patch « The Wiert Corner – irregular stream of stuff.

Some links that might help me with this:

[Wayback] regex – Using sed to delete all lines between two matching patterns – Stack Overflow (thanks users [Wayback] Lri and [Wayback] Akito)
- [Wayback/Archive.is] akito-libbash/setup.bash at dd91364083f13d1132d68489172bbce664b9c9c0 · theAkito/akito-libbash an actual example close to what I am after (it appends, I want to keep the end of the file in tact).
[Wayback] bash – removing lines between two patterns (not inclusive) with sed – Stack Overflow
- [Wayback] sed, a stream editor: sed script overview
- [Wayback] sed, a stream editor: Multiple commands syntax
[Wayback] shell – How to append multiple lines to a file – Unix & Linux Stack Exchange
- [Wayback] Sed Command in Linux – Append and Insert Lines to a File ~ Your Own Linux..!
[Wayback] text processing – How to insert the content of a file into another file before a pattern (marker)? – Unix & Linux Stack Exchange

One alternative would have been to use ed (which is part of the normal Busybox), but ESXi Busybox omits ed like it omits patch.

Too bad that sed commands are too different from ed commands, as I could have used diff -e on another system based on ideas here:

I might give it one more go, as vi is sort of derived from ed via ex (see vi: Creation – Wikipedia), which means that vi “colon mode” (officially command mode: [Wayback] Vim documentation: cmdline) is very similar to ed.

Another alternative would be awk, but I have done so little work with it awk, that I’m hesitating to use a new tool. Some links:

[Wayback] awk – Check for multi-line content in a file – Stack Overflow

And finally, ash could be used:

[Wayback] text processing – Script matching literal pattern over multiple lines? – Unix & Linux Stack Exchange

The kind of modifications I am after

Below are a few links with examples of the kind of modifications I want to make. Most patch just /etc/rc.local.d/local.sh, but some others introduce other changes as well.

Note that especially with networking settings, local.sh commands might not have any effect (for instance when having slow DHCP or other network issues), see for instance [Wayback/Archive.is] I’m running ESXi 5.5 and my persistent route in local.sh is not taking effect after boot. : vmware.

There is a very convoluted way around using local.sh by using the VIB authoring tool as described in [Wayback] How to create persistent firewall rules on ESXi. It requires lowering the software acceptance level to Community Supported (esxcli software acceptance set --level=CommunitySupported), which gives you a hard time installing ESXi updates.

I got that VIB idea from [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN, as:

The local.sh file gets overwritten often with upgrades so it would mean another step during the process.

From the same thread comes [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN

set the sticky bit on your separate xml-file – then it will be backed up and persist through reboot: chmod +t

run backup manually before the first reboot: /sbin/auto-backup.sh because backup runs only once per hour

Within vSphere, one could use [Wayback] Configure ESXi Hosts with Host Profiles, but a standalone ESXi box is not part of vSphere, so that won’t work.

ESXi 7 and up

ESXi 7 makes the above harder as for instance user root cannot change file rights any more, so eventually I might revert to a VM that auto-boots when ESXi comes up, then patches the right files in place over PowerCLI (read-only) or SSH.

Need to give this some thought later:

[Wayback] virtualization – ESXi 7.X file permissions – how to buypass new security measures – Server Fault
[Archive.is] The root account can no longer change permissions or executable files in ESXi 7.0.x (78689)
[Wayback/Archive.is] PowerCLI, ESXi 6.5 free and Start-VM: Current license or ESXi version prohibits execution of the requested operation : homelab

PowerCLI commands on the free version are limited to commands that are “read-only”, so you can only find out information rather than perform actions, if that makes sense.
So you can find out if a VM is turned on, find it’s uptime, but you can not turn it on or reboot it
This presumably is to prevent automation without a license

…

Having a quick look at the VMware KB, it looks like PowerCLI was limited on the free version to “read-only” operations from 5.0 (I.e. when it went from ESX to ESXI) along with vCLI and vSphere-Perl, so for some time from the looks of it

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Awk, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, PowerCLI, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Jilles posted a small script to show offline/online status based on ping

Posted by jpluimers on 2022/03/16

Jilles posted a small script to show offline/online status based on ping in [Archive.is] Jilles on Twitter: “#!/bin/bash HOST={1ST HOP HERE} while true;do p=$(ping -c1 $HOST) if [ $? -ne 0 ];then s=offline else s=online\ fi echo $(date +%F\ %T) $s – $(echo $p | sed -e ‘s/^PING.*— 1/1/g’) sleep 10 done”

#!/bin/bash
HOST={1ST HOP HERE}
while true;do
  p=$(ping -c1 $HOST)
  if [ $? -ne 0 ];then
    s=offline
  else
    s=online\ 
  fi
  echo $(date +%F\ %T) $s - $(echo $p | sed -e 's/^PING.*--- 1/1/g')
  sleep 10
done

The reason was that his ISP had connection problems for the block of homes where Jilles lives.

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, grep, Power User, Scripting, sed, Software Development | Leave a Comment »

ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes

Posted by jpluimers on 2022/01/17

On VMware ESXi, with the Busybox ash/dash shell, I wanted to get the current UTC timestamp in ISO 8601 format without dashes (-) or especially colons (:) and plus-signs (+) you have to back-slash escape colons or double quote parameters, which is often can be a pain).

[Wayback] How to Manipulate Filenames Having Spaces and Special Characters in Linux describes the pain handling special characters:

In this article, we will see how to create, copy, move and delete filenames that starts with spaces and special characters (say #, *, &, =, etc.) in Linux.
[Wayback] filenames – What characters are safe in cross-platform file names for Linux, Windows and OS-X – Super User
…
- []()^ #%&!@:+={}'~ and [`] all have special meanings in many shells, and are annoying to work around, and so should be avoided. They also tend to look horrible in URLs.
…

This is why we can’t have good things: Getting the UTC 8610 timestamp was far less easy than I hoped for.

First of all, Busybox only allows for a precision of seconds, not milliseconds, and the specification format needs better documentation as per [Wayback] embedded linux – How to get ISO8601 seconds format from “date” in busybox? – Stack Overflow:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

ESXi: listing virtual machines with their IP addresses

Posted by jpluimers on 2021/10/26

This is sort of a follow-up on VMware ESXi console: viewing all VMs, suspending and waking them up: part 4 which already gave part of the configuration details of all the configured VMs.

Back then, we ended with this:

List the `vmid` values, power status and name of all VMs

Back to the listing script vim-cmd-list-all-VMs.sh:

#!/bin/sh
# https://wiert.me/2021/04/29/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-4/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
done

It uses vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and vim-cmd vmsvc/get.config with some sed and a for loop from dash to generate a nice list of information.

A long time ago, I already figured out that vim-cmd vmsvc/get.guest # gives all guest information including network information for a running VM that has either VMware Tools or open-vm-tools running (see VMware ESXi console: viewing all VMs, suspending and waking them up: part 3 for the difference between these two tools).

A full output of a sample VM is below the signature.

There are a few places that have the LAN ipAddress. For now, I choose to use only the IPv4 main address from ipAddress, which is in between (vim.vm.GuestInfo) { and net = (vim.vm.GuestInfo.NicInfo) [.

I modified the above script to become this:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, find, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

windows – Is there any sed like utility for cmd.exe? – Stack Overflow

Posted by jpluimers on 2021/07/19

[WayBack] windows – Is there any sed like utility for cmd.exe? – Stack Overflow

TL;DR: many people suggest to use PowerShell, but there is GNU sed in Chocolatey

The chocolatey part:

[WayBack] Chocolatey Gallery | GNU sed 4.5
Based on [WayBack] GitHub – mbuilov/sed-windows: Instructions for building sed.exe as a native windows application

Instructions for building [WayBack] Gnu Sed as a native windows application

All patches under the same license as sources of [WayBack] Gnu Sed: [WayBack] GPLv3 or later
- The 4.5 version of sed.exe was exactly the binary [WayBack] sed-windows/sed-4.5-x64.exe at cafe68124fb8f01db3fb1d9ea586f8f6a72d6917 · mbuilov/sed-windows · GitHub

The PowerShell part: read the other answers from the above question.

–jeroen

Posted in *nix, *nix-tools, CommandLine, Power User, PowerShell, RegEx, sed, Windows | Leave a Comment »

Hopefully datendomina (@sys_adm_ama) has followed up with some cool vi tips…

Posted by jpluimers on 2021/06/28

I hope that datendomina (@sys_adm_ama) has followed up with some cool vi/vim tips.

Though I can do basic editing (far more than quit-without-saving) and know about he various mode, I still feel not proficient.

[WayBack] Jeroen Pluimers on Twitter: “LOL! Boy was I glad that after finding my way in Ed and sed on SunOS in the 1980s, I discovered vi. Still not proficient in it (and I probably never will). However, knowing some of the basics allowed me to visually edit any file on any Unix like system. That’s still gold to me.…”

It also made me discover [WayBack] ed(1) Conference (@ed1conf) | Twitter.

One important tip:

[WayBack] Kristian Köhntopp on Twitter: “vi movement Kommandos haben System. Erkenne und lerne das System. hjkl + prefix+hjkl, Marken, prefix+jump to mark und so weiter. Und bleibe von den verblödeten Plugins weg. Die braucht kein Mensch und machen vim nur langsam im Start und kompliziert.”

The original thread, which I hoped would get longer: [WayBack] Thread by @sys_adm_ama: “Ich lerne jetzt vi(m). Klingt beknackt, oder?se […]”

Ich lerne jetzt vi(m).

Klingt beknackt, oder? Aber ich hab überlegt: ich möchte effizienter werden, meinen Kram stressfreier bewältigen. Und ich finde, es bietet sich an da an Baustellen anzusetzen, die täglich relevant sind. Und vi(m) nutze ich in der Tat täglich.

1/

Aber auch wenn ich ihn nutze und über den »Hilfe, wie komm ich aus dem Editor wieder raus?!«-Witz nur sehr müde die Augen rolle gehe ich davon aus, dass ich nicht mal einen Bruchteil der Möglichkeiten ausschöpfe, die er bietet (1. Release 1976, älter als ich!). Das ist spannend.

Das ist jetzt meine Mini-Challenge, auf die ich jeden Tag eine Viertelstunde verwenden will: wenn ich eine Funktion brauche mich nicht mehr drum herum zu hacken, sondern recherchieren wie es richtig geht und das dann gefälligst auch verinnerlichen. Mal sehen, ob das so klappt

3/

In dem Zuge will ich auch wieder verstärkt (neo)mutt in Verbindung mit vim nutzen – das wäre ein wunderbares tägliches Training 😎 Mails schreiben muss man irgendwie immer.

neomutt bietet leider auch einen Eimer voll Funktionen, die ich noch nicht ordentlich nutze. Gnah.

4/

An euren Replies erkenne ich, dass das mit dem »sich die Kürzel merken« echt heikel zu sein scheint 🤔 Wie handhabt ihr das im Alltag? Einfach ein paar Basics wie :u und CTRL+r und gut ist? Ich bin neugierig. Oder nutzt ihr alle nano? (Ich glaub, dann muss ich entfolgen) 😂

Nur als kleinen Zwischenstand: nach dem Lesen eurer Antworten schließe ich, dass ich mich mit meinem Kenntnisstand nicht verstecken muss 😂 Da hat mir der virtuelle Schulterblick schon weiter geholfen.

/5

vi(m), weil ich Admin bin und dieses Tool auf jedem System und ohne X-Geraffels üblicherweise vorfinde (wenn auch bei neueren Installationen dieser absurde „visual mode“ der Default ist 😳). Auch unter (Open)Solaris, IRIX, was weiß ich.

/6

Eben hab ich das Buch von @MasteringVim aus der Packstation gezogen (extrem vielversprechend!) – und klar, ich werde berichten 😎 Ich bin sehr gespannt.

/7ed

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, Development, ed, Power User, Scripting, sed, sed script, Software Development, vi/vim | Leave a Comment »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 5

Posted by jpluimers on 2021/04/30

Yesterday’s post got a bit longer than anticipated as there were most steps than I hoped for to create the listing script vim-cmd-list-all-VMs.sh:

#!/bin/sh
# https://wiert.me/2021/04/29/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-4/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
done

This means that today there is a new try to cover the rest of these lists I started yesterday with:

Available commands

vim-cmd vmsvc/power.getstate vmid

vim-cmd vmsvc/power.hibernate vmid

vim-cmd vmsvc/power.off vmid

vim-cmd vmsvc/power.on vmid

vim-cmd vmsvc/power.reboot vmid

vim-cmd vmsvc/power.reset vmid

vim-cmd vmsvc/power.shutdown vmid

vim-cmd vmsvc/power.suspend vmid

vim-cmd vmsvc/power.suspendResume vmid

Unavailable commands

vim-cmd vmsvc/power.startup vmid

vim-cmd vmsvc/power.resume vmid

vim-cmd vmsvc/power.wakeup vmid

So here we go:

`vim-cmd-hibernate-running-VMs.sh` and `vim-cmd-suspend-running-VMs.sh`

These files are almost the same: the echo and command are different (Hibernating versus Suspending and power.hibernate versus suspend)

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on  VM with id ${vmid} and name: $name"
        echo "Hibernating VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.hibernate ${vmid}
    fi
done

This hibernates all VMs with power state Powered on and waits for each VM to complete hibernation.

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on VM with id ${vmid} and name: $name"
        echo "Suspending VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.suspend ${vmid}
    fi
done

This suspends all VMs with power state Powered on and waits for each VM to complete suspending.

`vim-cmd-resume-suspended-VMs.sh` and `vim-cmd-wakeup-suspended-VMs.sh`

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Suspended" ] ; then
        echo "Suspended VM with id ${vmid} and name: $name"
        echo "Resuming  VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.on ${vmid}
    fi
done

This resumes (as there is no vim-cmd vmsvc/power.resume vmid) all VMs with power state Suspended and waits for each VM to complete resume.

`vim-cmd-power-off-powered-on-VMs.sh`

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on   VM with id ${vmid} and name: $name"
        echo "Powering off VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.off ${vmid}
    fi
done

This powers off all VMs with power state Power on and waits for each VM to start powering off (but does not wait for them to complete powering off, so parts run in parallel).

`vim-cmd-power-on-powered-off-VMs.sh`, `vim-cmd-power-on-shutdown-VMs.sh` and `vim-cmd-startup-shutdown-VMs.sh`

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered off" ] ; then
        echo "Powered off VM with id ${vmid} and name: $name"
        echo "Powering on VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.on ${vmid}
    fi
done

These powers on all VMs with power state Powered off and waits for each VM to complete power on (but does not wait for them to complete boot, so part runs in parallel!).

These are exactly the same, as you cannot distinguish a VM that has been shutdown by vim-cmd vmsvc/power.off vmid from vim-cmd vmsvc/shutdown vmid.

`vim-cmd-reboot-powered-on-VMs.sh`

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on VM with id ${vmid} and name: $name"
        echo "Rebooting  VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.reboot ${vmid}
    fi
done

This reboots all VMs with power state Power on and waits for each VM to start rebooting (but does not wait for them to complete reboot, so parts run in parallel).

`vim-cmd-reset-powered-on-VMs.sh`

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on VM with id ${vmid} and name: $name"
        echo "Resetting  VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.reset ${vmid}
    fi
done

This resets all VMs with power state Power on and waits for each VM to start rebooting (but does not wait for them to complete reboot, so parts run in parallel).

`vim-cmd-suspendResume-powered-on-VMs.sh`

#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    # echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
    if [ "${powerState}" == "Powered on" ] ; then
        echo "Powered on      VM with id ${vmid} and name: $name"
        echo "SuspendResuming VM with id ${vmid} and name: $name"
        vim-cmd vmsvc/power.suspendResume ${vmid}
    fi
done

This suspends and resumes on all VMs with power state Power on and waits for each VM to start this shirt cycle (but does not wait for them to complete the suspend and resume cycle, so parts run in parallel).

Future episodes

For now, this is OK enough for me, but I might write a new installment trying to run more of these in parallel (as parts already are done on the ESXi side).

–jeroen

VMware ESXi console: viewing all VMs, suspending and waking them up: part 4

Posted by jpluimers on 2021/04/29

Yesterday we ended with an overview of available and unavailable vim-cmd vmsvc commands and the promise to try running the various power commands on all relevant VMs.

Let’s start with a summary of the commands, so it will be easier to make a list of scripts to run them on relevant VMs.

Available commands

vim-cmd vmsvc/power.getstate vmid
vim-cmd vmsvc/power.hibernate vmid
vim-cmd vmsvc/power.off vmid
vim-cmd vmsvc/power.on vmid
vim-cmd vmsvc/power.reboot vmid
vim-cmd vmsvc/power.reset vmid
vim-cmd vmsvc/power.shutdown vmid
vim-cmd vmsvc/power.suspend vmid
vim-cmd vmsvc/power.suspendResume vmid

Unavailable commands

vim-cmd vmsvc/power.startup vmid
vim-cmd vmsvc/power.resume vmid
vim-cmd vmsvc/power.wakeup vmid

List the `vmid` values, power status and name of all VMs

Getting the `vmid`

Yesterday I showed a small statement that gives the list of vmid values on an ESXi system:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)s+((S.+S)?)s+([S+])s+(.+.vmx)s+(S+)s+(vmx-[[:digit:]]+)s*?((S.+)?)$/1/p"

What I ideally want is not just the vmid and name for each VM from vim-cmd vmsvc/getallvms, but also get the power state information from vim-cmd vmsvc/power.getstate vmid.

For that, we need to parse the output of vim-cmd vmsvc/power.getstate vmid, which can be three outputs:

Retrieved runtime info Powered off
Retrieved runtime info Powered on
Retrieved runtime info Suspended

So basically it involves deleting the first line which was covered in part 2 of this installment, for example on my system:

# vim-cmd vmsvc/power.getstate 10 | sed '1d'
Powered on

Getting VM `name`

Extracting both vmid and name from vim-cmd vmsvc/getallvms at the same time is not easy, heck even impossible, so I decided to go the vim-cmd vmsvc/get.config vmid way.

Getting multiple values out of some output is already very hard in bash, where usually the less difficult way is to use arrays. Since Busybox has an ash shell (see Busybox sh (actually ash derivative dash): checking exit codes), and ash does not do arrays, that route is gone.

To give you an idea how hard this is in bash and how to sort of workaround the lack of array support in ash:

[Wayback] shell – How to assign values to multiple variables in command line with bars – Unix & Linux Stack Exchange (explaining how to do this with multiple assignments)

[Wayback] bash – Extract two values from the output of a command – Unix & Linux Stack Exchange (explaining how to use tee to pass output to multiple distinct commands)

[Wayback] linux – Extract multiple values using sed – Stack Overflow (how to use sed to split output over multiple lines)

[Wayback] linux – how to get elements from list in bash? – Super User (on the bash array syntax)

[Wayback] linux – reading a tuple from a file with awk – Stack Overflow (on how to use multiple arrays in bash)

[Wayback] shell – How to port to bash-style arrays to ash? – Unix & Linux Stack Exchange (explaining that unlike bash, ash has no arrays)

This partial vim-cmd vmsvc/get.config vmid sample output on one of my VMs that shows how to use head -n 31 to get just the first 31 lines of output:

# vim-cmd vmsvc/get.config 10 | head -n 31
Configuration:

(vim.vm.ConfigInfo) {
   changeVersion = "2021-04-07T22:08:30.548274Z", 
   modified = "1970-01-01T00:00:00Z", 
   name = "X9SRI-3F-W10P-EN-MEDIA", 
   guestFullName = "Microsoft Windows 10 (64-bit)", 
   version = "vmx-14", 
   uuid = "564d51ac-f6cf-e40b-b686-2f53a28a4bea", 
   createDate = "2019-05-17T21:37:11.408173Z", 
   instanceUuid = "52403d0e-7ccd-48da-bb21-7e966defccf7", 
   npivNodeWorldWideName = , 
   npivPortWorldWideName = , 
   npivWorldWideNameType = , 
   npivDesiredNodeWwns = , 
   npivDesiredPortWwns = , 
   npivTemporaryDisabled = true, 
   npivOnNonRdmDisks = , 
   locationId = "564d6b18-ecd1-2261-0127-146b3f3bc636", 
   template = false, 
   guestId = "windows9_64Guest", 
   alternateGuestName = "", 
   annotation = "", 
   files = (vim.vm.FileInfo) {
      vmPathName = "[EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx", 
      snapshotDirectory = "[EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA", 
      suspendDirectory = "[EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA", 
      logDirectory = "[EVO860_500GB] VM/X9SRI-3F-W10P-EN-MEDIA", 
      ftMetadataDirectory = 
   }, 
   tools = (vim.vm.ToolsConfigInfo) {

The reason to go the vim-cmd vmsvc/get.config vmid way is that it contains all the configuration info in a kind of JSON format (except the first two lines) and should be relatively easy to parse. Or so at least I hoped.

Basically I am interested in the value of name = "X9SRI-3F-W10P-EN-MEDIA", however, there are multiple name fields in the total configuration:

# vim-cmd vmsvc/get.config 10 | sed -n -E '/name =/p'
   name = "X9SRI-3F-W10P-EN-MEDIA", 
         name = "EVO860_500GB",

So what I really want is the value of name = "X9SRI-3F-W10P-EN-MEDIA", in between the (vim.vm.ConfigInfo) { and files = (vim.vm.FileInfo) { parts.

This can be done using sed as it allows to specify a range using a start and end value using addresses:

[Wayback] sed: Addresses in sed
An address is either a decimal number that counts input lines cumulatively across files, a '$' character that addresses the last line of input, or a context address (which consists of a BRE, as described in Regular Expressions in sed , preceded and followed by a delimiter, usually a slash).

An editing command with no addresses shall select every pattern space.

An editing command with one address shall select each pattern space that matches the address.

An editing command with two addresses shall select the inclusive range from the first pattern space that matches the first address through the next pattern space that matches the second. (If the second address is a number less than or equal to the line number first selected, only one line shall be selected.) Starting at the first line following the selected range, sed shall look again for the first address. Thereafter, the process shall be repeated. Omitting either or both of the address components in the following form produces undefined results:
```
[address[,address]]
```
Range Addresses (sed, a stream editor)[Wayback] Range Addresses (sed, a stream editor)
An address range can be specified by specifying two addresses separated by a comma (,). An address range matches lines starting from where the first address matches, and continues until the second address matches (inclusively):
```
$ seq 10 | sed -n '4,6p'
4
5
6
```
If the second address is a regexp, then checking for the ending match will start with the line following the line which matched the first address: a range will always span at least two lines (except of course if the input stream ends).

…
[Wayback] Regexp Addresses (sed, a stream editor)

For example (with some characters escaped because of [Wayback] ERE syntax (sed, a stream editor)):

# vim-cmd vmsvc/get.config 10 | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/p'
(vim.vm.ConfigInfo) {
   changeVersion = "2021-04-07T22:08:30.548274Z", 
   modified = "1970-01-01T00:00:00Z", 
   name = "X9SRI-3F-W10P-EN-MEDIA", 
   guestFullName = "Microsoft Windows 10 (64-bit)", 
   version = "vmx-14", 
   uuid = "564d51ac-f6cf-e40b-b686-2f53a28a4bea", 
   createDate = "2019-05-17T21:37:11.408173Z", 
   instanceUuid = "52403d0e-7ccd-48da-bb21-7e966defccf7", 
   npivNodeWorldWideName = , 
   npivPortWorldWideName = , 
   npivWorldWideNameType = , 
   npivDesiredNodeWwns = , 
   npivDesiredPortWwns = , 
   npivTemporaryDisabled = true, 
   npivOnNonRdmDisks = , 
   locationId = "564d6b18-ecd1-2261-0127-146b3f3bc636", 
   template = false, 
   guestId = "windows9_64Guest", 
   alternateGuestName = "", 
   annotation = "", 
   files = (vim.vm.FileInfo) {

With [Wayback] BRE syntax (sed, a stream editor) the filter part would be easier: vim-cmd vmsvc/get.config 10 | sed -n -e '/(vim.vm.ConfigInfo) {/,/files = (vim.vm.FileInfo) {/p', but the print part would be more difficult:

# vim-cmd vmsvc/get.config 10 | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/1/p'
X9SRI-3F-W10P-EN-MEDIA

# vim-cmd vmsvc/get.config 10 | sed -n -e '/(vim.vm.ConfigInfo) {/,/files = (vim.vm.FileInfo) {/ s/^ +name = "(.*)",.*?/1/p'
X9SRI-3F-W10P-EN-MEDIA

Since I am used to extended regular expressions (ERE) over basica regular expressions (BRE), I prefer the first solution.

So getting the name in a variable now becomes this:

# name=`vim-cmd vmsvc/get.config 10 | sed -n -e '/(vim.vm.ConfigInfo) {/,/files = (vim.vm.FileInfo) {/ s/^ +name = "(.*)",.*?/1/p'`
# echo ${name}
X9SRI-3F-W10P-EN-MEDIA

List the `vmid` values, power status and name of all VMs

Back to the listing script vim-cmd-list-all-VMs.sh:

#!/bin/sh
# https://wiert.me/2021/04/29/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-4/
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
    powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
    name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
    vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
    echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
done

As a bonus, next to powerState, the script also figures out vmPathName in a similar way to name.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, head, Power User, Scripting, sed, sed script, Software Development, tee, Virtualization, VMware, VMware ESXi | Leave a Comment »

« Previous Entries

	Lars Fosdal on Security alarm provider Woonve…
	Thomas Mueller on Question got closed in May 202…
	Thaddy de Koning on Formulier voor bewindvoerders…
	Thaddy de Koning on Formulier voor bewindvoerders…
	Thaddy de Koning on Formulier voor bewindvoerders…

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

My Flickr Stream

Pages

All categories

Email Subscription

Archive for the ‘sed’ Category

Rate this:

Share this:

Rate this:

Share this:

The kind of modifications I am after

ESXi 7 and up

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

List the vmid values, power status and name of all VMs

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Available commands

Unavailable commands

vim-cmd-hibernate-running-VMs.sh and vim-cmd-suspend-running-VMs.sh

vim-cmd-resume-suspended-VMs.sh and vim-cmd-wakeup-suspended-VMs.sh

vim-cmd-power-off-powered-on-VMs.sh

vim-cmd-power-on-powered-off-VMs.sh, vim-cmd-power-on-shutdown-VMs.sh and vim-cmd-startup-shutdown-VMs.sh

vim-cmd-reboot-powered-on-VMs.sh

vim-cmd-reset-powered-on-VMs.sh

vim-cmd-suspendResume-powered-on-VMs.sh

Future episodes

Rate this:

Share this:

Available commands

Unavailable commands

List the vmid values, power status and name of all VMs

Getting the vmid

Getting VM name

List the vmid values, power status and name of all VMs

Rate this:

Share this:

List the `vmid` values, power status and name of all VMs

`vim-cmd-hibernate-running-VMs.sh` and `vim-cmd-suspend-running-VMs.sh`

`vim-cmd-resume-suspended-VMs.sh` and `vim-cmd-wakeup-suspended-VMs.sh`

`vim-cmd-power-off-powered-on-VMs.sh`

`vim-cmd-power-on-powered-off-VMs.sh`, `vim-cmd-power-on-shutdown-VMs.sh` and `vim-cmd-startup-shutdown-VMs.sh`

`vim-cmd-reboot-powered-on-VMs.sh`

`vim-cmd-reset-powered-on-VMs.sh`

`vim-cmd-suspendResume-powered-on-VMs.sh`

List the `vmid` values, power status and name of all VMs

Getting the `vmid`

Getting VM `name`

List the `vmid` values, power status and name of all VMs