June 2023
M	T	W	T	F	S	S
	1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30

Archive for the ‘ash/dash development’ Category

Don’t fall for the golden hammer: avoid git empty commits, especially for kicking off parts of your CI/CD

Posted by jpluimers on 2022/08/16

A while back Kristian Köhntopp (isotopp) wrote a blog post after quite a Twitter argument where he poses against using git empty commits. I’m with Kris: don’t use them for anything, especially not for kicking off your CI/CD.

Basically his blog post is all about avoiding to think you have a golden hammer, and avoid falling for the Law of the instrument – Wikipedia.

Originally, Abraham Maslow said in 1966:

“I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”

For me this has all to do with preventing technical debt: find the right tool to kick your CI/CD pipeline after part of that chain somehow malfunctioned is way better than polluting the commit history with empty commits.

His blog post: [Wayback/Archive.is] Empty commits and other wrong tools for the job | Die wunderbare Welt von Isotopp

The most important bit in it:

And since we are talking about CI/CD pipelines: Don’t YAML them. Don’t JSON them. Don’t XML them.

…

Programming in any of these three is wrong use of tooling, and you should not do it.

YAML, JSON and XML are for declarative things.

Python, Go and Rust are for procedural things.

Bash is for interactive use only.

Use the proper tooling for the job. Be an engineer.

This very much reminds me of an Entwickler Konferenz keynote a long time ago, where Neal Ford made the point that most software engineers act very much unlike what is expected from traditional engineering way of operating where the engineer is both responsible and liable for his actions.

The start of the Twitter thread: [Archive.is] Kristian Köhntopp on Twitter: “A lot of people right now that git is an API and triggering CI/CD pipelines with empty commits replaces the equivalent of a Kubernetes controller for their fragile pile of bash in git triggers. This is broken and begets more brokenness. Evidence:… “

The tweet that started the subtweet: [Archive.is] Florian Haas on Twitter: “(For anyone wondering, what’s nice about this one is it works in any CI. So you don’t have to remember how to manually kick off a GitLab CI pipeline or GitHub Action or Zuul job, you just push an empty commit and off you go.)”

Other relevant tweets:

Yes, you want to avoid shell too (anything like for instance sh, ash, dash, bash or zsh), but you have to know it (and understand why to avoid it) as often it is the only interactive way to access systems from the console.

And of course Kris also wrote a big document on that too, which is available as full PDF (Wayback), full HTML (Wayback) and chaptered HTML Die UNIX Shell /bin/sh.

But more importantly, Kris wrote [Wayback/Archive.is] Using Python to bash | Die wunderbare Welt von Isotopp which is about using Python to do things you might be tempted to do in the shell. It quotes

Shell is a thing you want to understand and then not use, because you learned to understand it.

which is from the German post in thread [Wayback/Archive.is] Bashprogrammierung, wo gehts am besten los which quotes Kris’ 1998 message:

From kris Tue Sep 1 11:26:12 1998
From: kris
Newsgroups: de.comp.os.unix.misc
Subject: Re: Shell-Frage, find, xargs, kopieren von vielen Dateien
References: <6seh24$q9a$2...@nz12.rz.uni-karlsruhe.de>
From: kr...@koehntopp.de (Kristian Koehntopp)
Alignment: chaotic/neutral
X-Copyright: (C) Copyright 1987-1998 Kristian Koehntopp -- All rights
reserved.
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

Marc.Hab...@gmx.de (Marc Haber) writes:
>mir ist das ganze Zeug mit der Shell, find, xargs und Konsorten noch
>reichlich verschlüsselt.

http://www.koehntopp.de/kris/artikel/unix/shellprogrammierung/

>xargs hin oder sollte ich besser ein Perlskript schreiben?

Verwende Perl. Shell will man koennen, dann aber nicht verwenden.

Kristian

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, Conference Topics, Conferences, Continuous Integration, Development, DVCS - Distributed Version Control, Event, git, Power User, Scripting, sh, Sh Shell, Software Development, Source Code Management, Technical Debt | Leave a Comment »

Kris on Twitter is a bit radical against shell scripts. Learn why.

Posted by jpluimers on 2022/04/13

I say to people: only use shell interactively, don’t write scripts. Never. Not one.
But Kris, they ask, why so radical?
Because of this:

is the literal English Google Translation of the German text

Ich sage den Leuten: benutzt Shell nur interaktiv, schreibt keine Scripte. Nie. Nicht eines.
Aber Kris, fragen sie, wieso so Radikal?
Deswegen:

then links to [Wayback/Archive] Jan Schaumann on Twitter: “TIL zgrep(1) is a shell script. BSD basically does “zcat | grep”, but GNU does “gzip -dc | sed”. How did I learn that? The fun way! CVE-2022-1271, arbitrary-file-write and code execution vulnerability in GNU zgrep / gzip. …”:

Read the rest of this entry »

Posted in *nix, *nix-tools, Apple, ash/dash, ash/dash development, bash, bash, BSD, Development, Mac, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »

When some virtual machines cannot run VMware Tools: Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly)

Posted by jpluimers on 2022/04/12

An interesting set of scripts from [Wayback/Archive.is] No Joke IT: Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly).

If all ESXi virtual machines support running of VMware Tools, then the solution is a plain /sbin/shutdown.sh && /sbin/poweroff (see [Wayback/Archive.is] No Joke IT: Shut down ESXi 5.1 guest VMs and the host (free edition) via SSH – the easy way!).

Code is in the repository at [Wayback/Archive.is] sixdimensionalarray/esxidown: A shell script to shutdown VMware ESXi host servers, with these two main files:

[Wayback/Archive.is] esxidown/async.sh at master · sixdimensionalarray/esxidown

#!/bin/sh
# Runs a shell command asynchronously.

if [ "$1" != "" ]; then
    nohup sh $1 > /dev/null 2>&1 &
fi

[Wayback/Archive.is] esxidown/esxidown.sh at master · sixdimensionalarray/esxidown with this interesting code fragment right before the script finishes:

# exit maintenance mode immediately before server has a chance to shutdown/power off
# NOTE: it is possible for this to fail, leaving the server in maintenance mode on reboot!
echo "Exiting maintenance mode..."
if [ $TEST -eq 0 ]; then
    esxcli system maintenanceMode set -e false -t 0
fi

Note: the No Joke IT web-site has vanished, so only the [Wayback] and [Archive.is] links of it still work. The github code was still there at the time of writing.

Via: [Wayback] Solved: Read only Files – VMware Technology Network VMTN

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Some notes on replacing parts of a text file with template text using sed on a Busybox system

Posted by jpluimers on 2022/03/17

Note before you think about putting stuff in /etc/rc.local.d/local.sh: that script will not be executed when UEFI booting.

In a very lightweight Busybox system, I wanted to modify some configuration files automatically using fragments stored in template files.

The system has diff, but no patch.

The basic idea is to use sed to insert the template files into certain spots of the configuration file when certain marker texts are not present. So I want the opposite of [Wayback] Hey Stephen Wood: Try patch instead of sed in shell scripts.

Basically the idea is a poor-man’s patch, described in Too bad: ESXi busybox has diff, but not patch « The Wiert Corner – irregular stream of stuff.

Some links that might help me with this:

[Wayback] regex – Using sed to delete all lines between two matching patterns – Stack Overflow (thanks users [Wayback] Lri and [Wayback] Akito)
- [Wayback/Archive.is] akito-libbash/setup.bash at dd91364083f13d1132d68489172bbce664b9c9c0 · theAkito/akito-libbash an actual example close to what I am after (it appends, I want to keep the end of the file in tact).
[Wayback] bash – removing lines between two patterns (not inclusive) with sed – Stack Overflow
- [Wayback] sed, a stream editor: sed script overview
- [Wayback] sed, a stream editor: Multiple commands syntax
[Wayback] shell – How to append multiple lines to a file – Unix & Linux Stack Exchange
- [Wayback] Sed Command in Linux – Append and Insert Lines to a File ~ Your Own Linux..!
[Wayback] text processing – How to insert the content of a file into another file before a pattern (marker)? – Unix & Linux Stack Exchange

One alternative would have been to use ed (which is part of the normal Busybox), but ESXi Busybox omits ed like it omits patch.

Too bad that sed commands are too different from ed commands, as I could have used diff -e on another system based on ideas here:

I might give it one more go, as vi is sort of derived from ed via ex (see vi: Creation – Wikipedia), which means that vi “colon mode” (officially command mode: [Wayback] Vim documentation: cmdline) is very similar to ed.

Another alternative would be awk, but I have done so little work with it awk, that I’m hesitating to use a new tool. Some links:

[Wayback] awk – Check for multi-line content in a file – Stack Overflow

And finally, ash could be used:

[Wayback] text processing – Script matching literal pattern over multiple lines? – Unix & Linux Stack Exchange

The kind of modifications I am after

Below are a few links with examples of the kind of modifications I want to make. Most patch just /etc/rc.local.d/local.sh, but some others introduce other changes as well.

Note that especially with networking settings, local.sh commands might not have any effect (for instance when having slow DHCP or other network issues), see for instance [Wayback/Archive.is] I’m running ESXi 5.5 and my persistent route in local.sh is not taking effect after boot. : vmware.

There is a very convoluted way around using local.sh by using the VIB authoring tool as described in [Wayback] How to create persistent firewall rules on ESXi. It requires lowering the software acceptance level to Community Supported (esxcli software acceptance set --level=CommunitySupported), which gives you a hard time installing ESXi updates.

I got that VIB idea from [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN, as:

The local.sh file gets overwritten often with upgrades so it would mean another step during the process.

From the same thread comes [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN

set the sticky bit on your separate xml-file – then it will be backed up and persist through reboot: chmod +t

run backup manually before the first reboot: /sbin/auto-backup.sh because backup runs only once per hour

Within vSphere, one could use [Wayback] Configure ESXi Hosts with Host Profiles, but a standalone ESXi box is not part of vSphere, so that won’t work.

ESXi 7 and up

ESXi 7 makes the above harder as for instance user root cannot change file rights any more, so eventually I might revert to a VM that auto-boots when ESXi comes up, then patches the right files in place over PowerCLI (read-only) or SSH.

Need to give this some thought later:

[Wayback] virtualization – ESXi 7.X file permissions – how to buypass new security measures – Server Fault
[Archive.is] The root account can no longer change permissions or executable files in ESXi 7.0.x (78689)
[Wayback/Archive.is] PowerCLI, ESXi 6.5 free and Start-VM: Current license or ESXi version prohibits execution of the requested operation : homelab

PowerCLI commands on the free version are limited to commands that are “read-only”, so you can only find out information rather than perform actions, if that makes sense.
So you can find out if a VM is turned on, find it’s uptime, but you can not turn it on or reboot it
This presumably is to prevent automation without a license

…

Having a quick look at the VMware KB, it looks like PowerCLI was limited on the free version to “read-only” operations from 5.0 (I.e. when it went from ESX to ESXI) along with vCLI and vSphere-Perl, so for some time from the looks of it

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Awk, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, PowerCLI, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Too bad: ESXi busybox has `diff`, but not `patch`

Posted by jpluimers on 2022/03/02

On my ESXi boxes, I have a directory with local scripts that in part depend on the machine.

So I contemplated patching the dending parts with patch.

Then I found out that the BusyBox that VMware built for ESXi does have diff, but not patch:

# $(readlink -f "`which diff`")
BusyBox v1.29.3 (2021-01-17 01:25:00 PST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list
   or: function [arguments]...

    BusyBox is a multi-call binary that combines many common Unix
    utilities into a single executable.  Most people will create a
    link to busybox for each function they wish to use and BusyBox
    will act like whatever it was invoked as.

Currently defined functions:
    addgroup, adduser, arch, ash, awk, basename, bunzip2, bzcat, bzip2, cat, chgrp, chmod, chown, chvt, cksum, clear, cp, crond,
    cut, date, dd, delgroup, deluser, diff, dirname, dnsdomainname, du, echo, egrep, eject, env, expr, false, fdisk, fgrep, find,
    fstrim, getty, grep, groups, gunzip, gzip, halt, head, hexdump, hostname, inetd, init, kill, ln, logger, login, ls, lzop,
    lzopcat, md5sum, mkdir, mkfifo, mknod, mktemp, more, mv, nohup, nslookup, od, passwd, poweroff, printf, readlink, reboot,
    reset, resize, rm, rmdir, sed, seq, setsid, sh, sha1sum, sha256sum, sha3sum, sha512sum, sleep, sort, ssl_client, stat, stty,
    sum, sync, tail, tar, taskset, tee, test, time, timeout, touch, true, uname, uniq, unlink, unlzop, unzip, usleep, vi, watch,
    wc, wget, which, who, xargs, zcat

This list is much shorter than the applets that are supported in [Wayback] BusyBox – The Swiss Army Knife of Embedded Linux, so VMware did cut out quite a few.

Generating the above output

The command-line trick above first expands diff using the output of which diff, then finds out where it links to through the readlink -f wrapper there the back-quotes “`” get this output:

# readlink -f "`which diff`"
/usr/lib/vmware/busybox/bin/busybox

Finally the $(...) executes the output of readlink.

It is based on [Wayback] bash – How to resolve symbolic links in a shell script – Stack Overflow

readlink -f "$path"

Editor’s note: The above works with GNU readlink and FreeBSD/PC-BSD/OpenBSD readlink, but not on OS X as of 10.11.GNU readlink offers additional, related options…

Need to devise a way to apply patches

Given there is no patch, I need to think about a good way to apply patches, for instance to snip this into /etc/rc.local.d/local.sh in a reliable way:

## BEGIN-PATCH-PATH

# local binaries are in /vmfs/volumes/NVMe980PRO_1TB/local-bin/
# link that directory from /opt/bin
# then add /opt/bin to the PATH in /etc/profile so that on each logon it becomes available
# this means you need to logon twice after reboot:
# - first to patch /etc/profile
# - second to have the correct PATH loaded from /etc/profile
# direcory exist trick from https://stackoverflow.com/questions/59838/how-can-i-check-if-a-directory-exists-in-a-bash-shell-script

patch_etc_profile_PATH() {
    if [ -d "$1" ]; then
      ln -s "$1" "/opt/bin"
      sed -i -e 's!PATH=/bin:/sbin!PATH=/bin:/sbin:/opt/bin/!' /etc/profile
    fi
}

patch_etc_profile_PATH /vmfs/volumes/NVMe980PRO_1TB/local-bin/

## END-PATCH-PATH

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

« Previous Entries

	Mike Verhagen on geerlingguy/my-backup-plan: Ho…
	Attila Kovacs on docs.embarcadero.com unreachab…
	David Blue on MacOS: converting a man page t…
	jpluimers on How do I pretty-print JSON in…
	jpluimers on Delphi 10.2 Tokyo Godzilla ISO…

The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

Twitter Updates

My Flickr Stream

Pages

All categories

Email Subscription

Archive for the ‘ash/dash development’ Category

Don’t fall for the golden hammer: avoid git empty commits, especially for kicking off parts of your CI/CD

Kris on Twitter is a bit radical against shell scripts. Learn why.

When some virtual machines cannot run VMware Tools: Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly)

Some notes on replacing parts of a text file with template text using sed on a Busybox system

The kind of modifications I am after

ESXi 7 and up

Too bad: ESXi busybox has `diff`, but not `patch`

Generating the above output

Need to devise a way to apply patches

The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

Twitter Updates

My Flickr Stream

Pages

All categories

Email Subscription

Archive for the ‘ash/dash development’ Category

Don’t fall for the golden hammer: avoid git empty commits, especially for kicking off parts of your CI/CD

Rate this:

Share this:

Like this:

Kris on Twitter is a bit radical against shell scripts. Learn why.

Rate this:

Share this:

Like this:

When some virtual machines cannot run VMware Tools: Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly)

Rate this:

Share this:

Like this:

Some notes on replacing parts of a text file with template text using sed on a Busybox system

The kind of modifications I am after

ESXi 7 and up

Rate this:

Share this:

Like this:

Too bad: ESXi busybox has `diff`, but not `patch`

Generating the above output

Need to devise a way to apply patches

Rate this:

Share this:

Like this: