The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘bash’ Category

« Previous Entries
Next Entries »

Merging multiple commands and piping it to one output.

Posted by jpluimers on 2016/10/20

The unix shell is hard, but boy, sometimes it can work like magic, for instance piping two testssl.sh commands into one gist:

retinambpro1tb:testssl.sh jeroenp$ ( ./testssl.sh --version ; ./testssl.sh --local ) | gist -d "testsll version and local ciphers for Mac OS X Darwin binarries supporting zlib"
https://gist.github.com/701496d7fbf929967aa1

The source of this magic was this AskUbuntu answer: How to merge and pipe results from two different commands to single command? – Ask Ubuntu

–jeroen

via: openssl.Darwin.x86_64 lacks zlib support · Issue #164 · drwetter/testssl.sh

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development, Uncategorized | Leave a Comment »

Multi-line comments in a bash shell script

Posted by jpluimers on 2016/10/18

I’m used to multiline comments in many languages, but since I don’t use bash very often, I only recently learned it can do that too.

So this is how to have comments span multiple lines in bash by using the : command which according to the bash man page does nothing:

: [arguments]
No effect; the command does nothing beyond expanding arguments and performing any specified redirections. A zero exit code is returned.

So the syntax is:

: '
your comments here
'

–jeroen

via: Shell Script Put Multiple Line Comment.

Posted in bash, Development, Scripting, Software Development | Leave a Comment »

Some cURL links with tips I used doing some Fritz!Box scripting research

Posted by jpluimers on 2016/09/29

I needed to script a few things on my Fritz!Box. Here are the cURL links that I used to research some Fritz!Box scripting.

My first try was wget, but that didn’t do everything I need, so cURL came to the rescue.

In the end, I didn’t need cookies (a post request with an MD5 based handshake sufficed to get a session SID which is not stored in a Cookie), but that surely will come in useful another time.

Curl man page entries:

The script is and docs are here: jpluimers/bash-fritzclient.

–jeroen

Posted in *nix, bash, cURL, Development, Fritz!, Fritz!Box, Network-and-equipment, Power User, Scripting, Software Development, wget | Leave a Comment »

Converting sendmail .db files to text

Posted by jpluimers on 2016/09/15

If you want to inverse the work of newaliases, you can use makemap to convert them to text:

makemap -u hash access.db

This is quite handy to see if the newaliases indeed put all information into the db file (for instance if you use a script, you can verify it ran correctly).

It works for any db, but you need to be aware of the database format: hash type or btree type.

To my knowledge only one uses the btree type format:

  • userdb.db

All others use hash type format:

  • aliases.db
  • aliases-maillist.db
  • mailertable.db
  • genericstable.db
  • virtusertable.db
  • access.db
  • auth-info.db
  • domaintable.db

–jeroen

via: Reading Sendmail .db files.

Posted in *nix, *nix-tools, bash, Development, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux | Leave a Comment »

ESXi: when esxtop shows garbage.

Posted by jpluimers on 2016/09/14

The ESXi console top to show processes is not available *, the alternative is esxtop. But that can show garbage because the ESXi console has a very limited support of terminals **.

For instance, when connecting from a Mac OS X terminal through ssh, this is my terminal:

# echo $TERM
xterm-256color

The solution:

TERM=xterm esxtop

–jeroen

via: VMware KB: Output of esxtop defaults to non-interactive CSV with unknown TermInfo.

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, ESXi5, ESXi5.1, ESXi5.5, ESXi6, Power User, Scripting, Virtualization, VMware, VMware ESXi | 2 Comments »

vim-cmd suspending/resuming a bunch of VMs

Posted by jpluimers on 2016/09/13

These two vim-cmd scripts come in very handy:


#!/bin/sh
VMS=`vim-cmd vmsvc/getallvms | grep -v Vmid | awk '{print $1}'`
for VM in $VMS ; do
# echo "Probing VM with id: $VM."
PWR=`vim-cmd vmsvc/power.getstate $VM | grep -v "Retrieved runtime info"`
name=`vim-cmd vmsvc/get.config $VM | grep -i "name =" | awk '{print $3}' | head -1 | awk -F'"' '{print $2}'`
echo "VM with id $VM has power state $PWR (name = $name)."
done

view raw

esxi-show-power-state-of-all-VMs.sh

hosted with ❤ by GitHub


#!/bin/sh
# https://wiert.me/2021/04/30/vmware-esxi-console-viewing-all-vms-suspending-and-waking-them-up-part-5/
RUNNING=0
vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"`
for vmid in ${vmids} ; do
# echo "Probing VM with id: $vmid."
powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
# echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
if [ "${powerState}" == "Powered on" ] ; then
RUNNING=1
echo "Powered on VM with id ${vmid} and name: $name"
echo "Suspending VM with id ${vmid} and name: $name"
vim-cmd vmsvc/power.suspend ${vmid} > /dev/null &
fi
done
while true ; do
if [ $RUNNING -eq 0 ] ; then
echo "Gone…"
break
fi
RUNNING=0
for vmid in ${vmids} ; do
# echo "Probing VM with id: $vmid."
powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'`
name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'`
vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'`
# echo "VM with id ${vmid} has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})."
if [ "${powerState}" == "Powered on" ] ; then
RUNNING=1
echo "Waiting for id ${vmid} and name: $name…"
fi
done
sleep 1
done
exit 0

view raw

esxi-suspend-all-VMs.sh

hosted with ❤ by GitHub


#!/bin/sh
SUSPENDED=0
VMS=`vim-cmd vmsvc/getallvms | grep -v Vmid | awk '{print $1}'`
VMstateToProcess="Suspended"
for VM in $VMS ; do
# echo "Probing VM with id: $VM."
PWR=`vim-cmd vmsvc/power.getstate $VM | grep -v "Retrieved runtime info"`
name=`vim-cmd vmsvc/get.config $VM | grep -i "name =" | awk '{print $3}' | head -1 | awk -F'"' '{print $2}'`
echo "VM with id $VM has power state $PWR (name = $name)."
if [ "$PWR" == "$VMstateToProcess" ] ; then
SUSPENDED=1
echo "Suspended VM with id $VM and name: $name"
echo "Resuming VM with id $VM and name: $name"
# you'd think power.suspendResume is the inverse of power.suspend, but actually power.on is:
vim-cmd vmsvc/power.on $VM > /dev/null &
fi
done
while true ; do
if [ $SUSPENDED -eq 0 ] ; then
echo "Gone…"
break
fi
SUSPENDED=0
for VM in $VMS ; do
PWR=`vim-cmd vmsvc/power.getstate $VM | grep -v "Retrieved runtime info"`
if [ "$PWR" == "$VMstateToProcess" ] ; then
name=`vim-cmd vmsvc/get.config $VM | grep -i "name =" | awk '{print $3}' | head -1 | awk -F'"' '{print $2}'`
echo "Waiting for id $VM and name: $name…"
SUSPENDED=1
fi
done
sleep 1
done

view raw

esxi-resume-all-suspended-VMs.sh

hosted with ❤ by GitHub

–jeroen

Posted in bash, Development, ESXi4, ESXi5, ESXi5.1, ESXi5.5, ESXi6, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Modifying openssl to build on Mac without makedepend: using cc -M/gcc -M/clang -M

Posted by jpluimers on 2016/09/08

Coping with OpenSSL is frustrating on so many levels. It would be hilarious if not so many depended on it.

The main thing is that there is so much cruft in openssl but nobody seemingly to care enough about copying with that cruft. Though the intentions of the LibreSSL are great and it has far left cruft, it didn’t gain enough ground in the 2+ years it exists.

For me, the biggest problem was that OpenSSL would fail to build on “modern” OS X versions. These lack makedepend which was part of OS X, but got removed somewhere between 2006 and 8.0 in 2012. Luckily, the 2006 post also mentions the alternative: gcc -M.

Actually cc -M or clang -M work just as well as cc symlinks to clang and gcc calls clang with some parameters.

OpenSSL references makedepend in a lot of Makefiles (see list below). The good part is that config and Configure modify the Makefiles so most people use a manual workaround after running Configure or revert to a macports built or other binaries of makedepend or a homebrew formula.

That is not really an automated solution suitable for general use, let alone suitable for Continuous Integration, so I dug a bit deeper: config is a shell script and Configure is a sherl script: a cleverly crafted shell script starting with the below line invoking Perl with the rest of the script. The reason is that #! shebangs need an absolute location which for Perl can vary. A nice discussion on this trick is at Running Under Some Shell which contains a more elaborate how this works:

Read the rest of this entry »

Posted in bash, Development, Makefile, Perl, Scripting, Software Development | Leave a Comment »

A bash script to btrfs snapshot details like disk sizes (requires btrfs quota to be enabled)

Posted by jpluimers on 2016/09/06

One thing I’ve not figured out yet is this: How does enabling btrfs quotas impact the system? – Unix & Linux Stack Exchange

My gut feeling says it’s not much. But if someone knows: please chime in.

Anyway: btrfs quota support helps you to see how big snapshots (and other subvolumes) are.

Enabling and disabling is easy (the slash is the path from where to enable quota):

btrfs quota enable /

btrfs quota disable /

The bad thing is that there is no command that gives just the quota status. You have to revert to btrfs qgroup show /

This will show you quota groups when quota are enabled:

qgroupid         rfer         excl 
--------         ----         ---- 
0/5          16.00KiB     16.00KiB

or errors when quota are disabled either of these two errors:

# btrfs qgroup show /
ERROR: can't perform the search - No such file or directory
ERROR: can't list qgroups: No such file or directory

# btrfs qgroup show /
ERROR: can't list qgroups: quotas not enabled

You might also want to check the syslogs as there was a [PATCH 1/2] btrfs: syslog when quota is enabled.

The btrfs qgroup output is the base for agronick/btrfs-size: A script that will print out a list of BTRFS subvolumes along with their size in megabytes and their name.

It actually shows it in human readable form (kilo/mega/giga/… bytes) and is documented at Get the size of your BTRFS Snapshots | PoisonPacket. It gives me a nice overview that corresponds to the snapper list output (see below).

The script has two drawbacks:

./btrfs-size/btrfs-size.sh: line 55: can't + 0: syntax error: invalid arithmetic operator (error token is "'t + 0")

  • it can take a while for btrfs quota enable / to actually deliver the quota for all the qgroupid values snapper uses. Until that time, the columns ID/Total/Exclusive Data are empty.
  • it won’t warn you if the quota data is out of data; to check that, run the below script

# btrfs qgroup show / > /dev/null
WARNING: Qgroup data inconsistent, rescan recommended

In that case, run btrfs quota rescan / and wait for it to complete by running this command until the warning is gone:

# btrfs qgroup show / > /dev/null
WARNING: Rescan is running, qgroup data may be incorrect

If the warning reverts to WARNING: Qgroup data inconsistent, rescan recommended then you might see the effects from this bug and should not worry about it: Bug 933428 – btrfs qgroup – allways shows WARNING: Qgroup data inconsistent.

For more btrfs quota background information, see: UseCases – btrfs Wiki.

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, Development, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux, Tumbleweed | 1 Comment »

List goals/targets in GNU make that contain variables in their definition – Stack Overflow

Posted by jpluimers on 2016/08/24

*nix systems tools often are have write-only maintenance tricks:

alias make-list-target='make -qp | awk -F":" "/^[a-zA-Z0-9][^$#\/\t=]*:([^=]|$)/ {split(\$1,A,/ /);for(i in A)print A[i]}"'

Like usual, I have only a vague idea what the regex expressions do.
–jeroen

via List goals/targets in GNU make that contain variables in their definition – Stack Overflow.

Posted in Awk, bash, Development, Scripting, Software Development | Leave a Comment »

bash – How can I convert tabs to spaces in every file of a directory? – Stack Overflow

Posted by jpluimers on 2016/08/05

Kev:

Try the command line tool expand.

expand -t 4 input >output

Chrono Kitsune:

And for those systems that don’t use the GNU Core Utilities, you have a decent chance of expand being installed since it is standardized by The Open Group’s Single Unix Specification. See Issue 6, which is from 2001, though some updates were applied, hence the year of publication being 2004: expand

–jeroen

via bash – How can I convert tabs to spaces in every file of a directory? – Stack Overflow.

Posted in *nix, *nix-tools, bash, bash, Development, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux | Leave a Comment »

« Previous Entries
Next Entries »