The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,908 other followers

Archive for the ‘bash’ Category

« Previous Entries

Don’t fall for the golden hammer: avoid git empty commits, especially for kicking off parts of your CI/CD

Posted by jpluimers on 2022/08/16

A while back Kristian Köhntopp (isotopp) wrote a blog post after quite a Twitter argument where he poses against using git empty commits. I’m with Kris: don’t use them for anything, especially not for kicking off your CI/CD.

Basically his blog post is all about avoiding to think you have a golden hammer, and avoid falling for the Law of the instrument – Wikipedia.

Originally, Abraham Maslow said in 1966:

“I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”

For me this has all to do with preventing technical debt: find the right tool to kick your CI/CD pipeline after part of that chain somehow malfunctioned is way better than polluting the commit history with empty commits.

His blog post: [Wayback/Archive.is] Empty commits and other wrong tools for the job | Die wunderbare Welt von Isotopp

The most important bit in it:

And since we are talking about CI/CD pipelines: Don’t YAML them. Don’t JSON them. Don’t XML them.

…

Programming in any of these three is wrong use of tooling, and you should not do it.

  • YAML, JSON and XML are for declarative things.
  • Python, Go and Rust are for procedural things.
  • Bash is for interactive use only.

Use the proper tooling for the job. Be an engineer.

This very much reminds me of an Entwickler Konferenz keynote a long time ago, where Neal Ford made the point that most software engineers act very much unlike what is expected from traditional engineering way of operating where the engineer is both responsible and liable for his actions.

The start of the Twitter thread: [Archive.is] Kristian Köhntopp on Twitter: “A lot of people right now that git is an API and triggering CI/CD pipelines with empty commits replaces the equivalent of a Kubernetes controller for their fragile pile of bash in git triggers. This is broken and begets more brokenness. Evidence:… “

The tweet that started the subtweet: [Archive.is] Florian Haas on Twitter: “(For anyone wondering, what’s nice about this one is it works in any CI. So you don’t have to remember how to manually kick off a GitLab CI pipeline or GitHub Action or Zuul job, you just push an empty commit and off you go.)”

Other relevant tweets:

Yes, you want to avoid shell too (anything like for instance sh, ash, dash, bash or zsh), but you have to know it (and understand why to avoid it) as often it is the only interactive way to access systems from the console.

And of course Kris also wrote a big document on that too, which is available as full PDF (Wayback), full HTML (Wayback) and chaptered HTML Die UNIX Shell /bin/sh.

But more importantly, Kris wrote [Wayback/Archive.is] Using Python to bash | Die wunderbare Welt von Isotopp which is about using Python to do things you might be tempted to do in the shell. It quotes

Shell is a thing you want to understand and then not use, because you learned to understand it.

which is from the German post in thread [Wayback/Archive.is] Bashprogrammierung, wo gehts am besten los which quotes Kris’ 1998 message:

From kris Tue Sep 1 11:26:12 1998
From: kris
Newsgroups: de.comp.os.unix.misc
Subject: Re: Shell-Frage, find, xargs, kopieren von vielen Dateien
References: <6seh24$q9a$2...@nz12.rz.uni-karlsruhe.de>
From: kr...@koehntopp.de (Kristian Koehntopp)
Alignment: chaotic/neutral
X-Copyright: (C) Copyright 1987-1998 Kristian Koehntopp -- All rights
reserved.
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

Marc.Hab...@gmx.de (Marc Haber) writes:
>mir ist das ganze Zeug mit der Shell, find, xargs und Konsorten noch
>reichlich verschlüsselt.

http://www.koehntopp.de/kris/artikel/unix/shellprogrammierung/

>xargs hin oder sollte ich besser ein Perlskript schreiben?

Verwende Perl. Shell will man koennen, dann aber nicht verwenden.

Kristian

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, Conference Topics, Conferences, Continuous Integration, Development, DVCS - Distributed Version Control, Event, git, Power User, Scripting, sh, Sh Shell, Software Development, Source Code Management, Technical Debt | Leave a Comment »

Filezilla SFTP: figuring out the cause of “Connection timed out after 20 seconds of inactivity” part 2: about ((non-)?(interactive|login) ?){2} bash shells

Posted by jpluimers on 2022/05/12

Last year, I wrote about Filezilla: figuring out the cause of “Connection timed out after 20 seconds of inactivity” about sftp connection problems.

The solution there was to exclude part of bashrc with an if [Wayback] statement so bash would skip it during sftp, but not during ssh login:

[WayBack] linux – Use .bashrc without breaking sftp – Server Fault

  • From answer 1 (thanks [WayBack] Mike):

    Try doing this instead

    if [ "$SSH_TTY" ]
then
   source .bashc_real
fi
  • From Answer 2 (thanks [WayBack] Insyte):

    A good trick for testing the cleanliness of your login environment is to ssh in with a command, which simulates the same way scp/sftp connect. For example: ssh myhost /bin/true will show you exactly what scp/sftp sees when they connect.

That caused some scripts not to be run when switching user, for instance by doing sudo su -.

The reason for that was that I forgot to put enough research in part of Answer 2, so I quote a few bits more of it (highlights and code markup mine):

… it’s worth pointing out that you can accomplish this carefully selecting which startup files to put the verbose stuff in. From the bash man page:

When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. This may be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc.

The sftp/scp tools start an interactive non-login shell, so .bashrc will be sourced.

For further reading, there is the underlying bash manual as a PDF file [Wayback] and html document tree [Wayback]. Note it is large (the PDF is 190 pages).

I find the easiest way to navigate around bash documentation through these links:

Types of shell invocations: to login or to non-login and to interactive or to non-interactive

Basically, from the above answer there are [Archive.is] 4 types of shells (confirmed by these parts of the bash documentation: [Wayback] Section 6.1: Invoking-Bash and [Wayback] Section 6.2: Bash-Startup-Files):

  1. interactive login
  2. interactive non-login
  3. non-interactive login
  4. non-interactive non-login

And there are various means the shells can start (ssh, local console, …). The "$SSH_TTY" trick only checks interactive login via ssh, but fails to detect others.

So I did some digging for the correct information to log, which including the above are:

  • [Wayback] Section 4.3.1: The-Set-Builtin
    • -h Locate and remember (hash) commands as they are looked up for execution. This option is enabled by default.
    • -m Job control is enabled (see Job Control). All processes run in a separate process group. When a background job completes, the shell prints a line containing its exit status.
    • -B The shell will perform brace expansion (see Brace Expansion). This option is on by default.
    • -H Enable ‘!’ style history substitution (see History Interaction). This option is on by default for interactive shells.

    Note that in addition to this, there is the non-settable option i: The current shell is interactive (see the -i in section 6.1 below).

  • [Wayback] Section 4.3.2: The-Shopt-Builtin
    • login_shell The shell sets this option if it is started as a login shell (see Invoking Bash). The value may not be changed.
  • [Wayback] Section 6.1: Invoking-Bash

    There are several single-character options that may be supplied at invocation which are not available with the set builtin.

    • -i Force the shell to run interactively. Interactive shells are described in Interactive Shells.

    …

    A login shell is one whose first character of argument zero is ‘-’, or one invoked with the –login option.

  • [Wayback] Section 6.2: Bash-Startup-Files explains about these shell invocation types:
    • interactive login shell
    • interactive non-login shell
    • non-interactive shell
  • [Wayback] Section 6.3.1: Is-this-Shell-Interactive

    To determine within a startup script whether or not Bash is running interactively, test the value of the ‘-’ special parameter. It contains i when the shell is interactive. For example:

    case "$-" in
*i*)    echo This shell is interactive ;;
*)  echo This shell is not interactive ;;
esac

    Alternatively, startup scripts may examine the variable PS1; it is unset in non-interactive shells, and set in interactive shells. Thus:

    if [ -z "$PS1" ]; then
        echo This shell is not interactive
else
        echo This shell is interactive
fi

From theory to practice

After reading the above documentation links, I put the below code in the global .bashrc (which of course caused trouble with sftp, so I commented it out later):

echo "Option flags: '$-'"
echo "PS1: '$PS1'"
echo "shopt login_shell: '$(shopt login_shell)'"
echo "Parameter zero: '$0'"
[ "$SSH_TTY" ] ; echo "[ \"\$SSH_TTY\" ] outcome: $?"

And the output after these commands:

  1. ssh user@host
    Option flags: 'himBH'
PS1: '\u@\h:\w> '
shopt login_shell: 'login_shell     on'
Parameter zero: '-bash'
[ "$SSH_TTY" ] outcome: 0

    Verdict: interactive, login

  2. ssh user@host

    followed by

    sudo su -
    Option flags: 'himBH'
PS1: '\[\]\h:\w #\[\] '
shopt login_shell: 'login_shell     on'
Parameter zero: '-bash'
[ "$SSH_TTY" ] outcome: 1

    Verdict: interactive, login

  3. ssh user@host

    followed by

    bash
    Option flags: 'himBH'
PS1: '\u@\h:\w> '
shopt login_shell: 'login_shell     off'
Parameter zero: 'bash'
[ "$SSH_TTY" ] outcome: 0

    Verdict: interactive, non-login

  4. ssh user@host

    followed by

    sudo su -

    then by

    bash
    Option flags: 'himBH'
PS1: '\[\]\h:\w #\[\] '
shopt login_shell: 'login_shell     off'
Parameter zero: 'bash'
[ "$SSH_TTY" ] outcome: 1

    Verdict: interactive, non-login

  5. ssh user@host /bin/true
    Option flags: 'hBc'
PS1: ''
shopt login_shell: 'login_shell     off'
Parameter zero: 'bash'
[ "$SSH_TTY" ] outcome: 1

    Verdict: non-interactive, non-login

The final one is what for instance sftp will see. It excludes the non-interactive mark in the shopt option flags.

Modifications to my .bashrc file

Since the [Wayback] test for "$SSH_TTY" is inconsistent with the login being interactive, I modified the .bashrc section

if [ "$SSH_TTY" ]
then
   source .bashc_real
fi

to become

if [[ $- =~ i ]]
then
   # only during interactive login shells
   source .bashc_real
fi

I know the [[...]] over test shorthand [...] is a bashism, see [Wayback] if statement – Is double square brackets [[ ]] preferable over single square brackets [ ] in Bash? – Stack Overflow for why I like it.

More relevant documentation

I based the above changes not only on the mentioned StackOverflow post, but also doing some more Googling revealing these useful documentation and question/answer links:

  • [Wayback] Section 3.2.5.2: Conditional Constructs; [[ expression ]]

    [[…]]

    [[ expression ]]

    Return a status of 0 or 1 depending on the evaluation of the conditional expression expression. Expressions are composed of the primaries described below in Bash Conditional Expressions. Word splitting and filename expansion are not performed on the words between the [[ and ]]; tilde expansion, parameter and variable expansion, arithmetic expansion, command substitution, process substitution, and quote removal are performed. Conditional operators such as ‘-f’ must be unquoted to be recognized as primaries.

    …

    An additional binary operator, ‘=~’, is available, with the same precedence as ‘==’ and ‘!=’. When it is used, the string to the right of the operator is considered a POSIX extended regular expression and matched accordingly (using the POSIX regcomp and regexec interfaces usually described in regex(3)). The return value is 0 if the string matches the pattern, and 1 otherwise. If the regular expression is syntactically incorrect, the conditional expression’s return value is 2.

  • [Wayback] Section 4.1: Bourne Shell Builtins; test or [...] (Bash Reference Manual)
    test expr

    Evaluate a conditional expression expr and return a status of 0 (true) or 1 (false). Each operator and operand must be a separate argument. Expressions are composed of the primaries described below in Bash Conditional Expressions. test does not accept any options, nor does it accept and ignore an argument of -- as signifying the end of options.

    When the [ form is used, the last argument to the command must be a ].

  • [Wayback] bash – What are the special dollar sign shell variables? – Stack Overflow (thanks [Wayback] kojiro!):
    • $1, $2, $3, … are the positional parameters.
    • "$@" is an array-like construct of all positional parameters, {$1, $2, $3 ...}.
    • "$*" is the IFS expansion of all positional parameters, $1 $2 $3 ....
    • $# is the number of positional parameters.
    • $- current options set for the shell.
    • $$ pid of the current shell (not subshell).
    • $_ most recent parameter (or the abs path of the command to start the current shell immediately after startup).
    • $IFS is the (input) field separator.
    • $? is the most recent foreground pipeline exit status.
    • $! is the PID of the most recent background command.
    • $0 is the name of the shell or shell script.

    Most of the above can be found under Special Parameters in the Bash Reference Manual. There are all the environment variables set by the shell.

    For a comprehensive index, please see the Reference Manual Variable Index.

  • [Wayback] bash – Differentiate Interactive login and non-interactive non-login shell – Ask Ubuntu (thanks [Wayback] terdon)

    Briefly (see here for more details), with examples:

    • interactive login shell: You log into a remote computer via, for example ssh. Alternatively, you drop to a tty on your local machine (Ctrl+Alt+F1) and log in there.
    • interactive non-login shell: Open a new terminal.
    • non-interactive non-login shell: Run a script. All scripts run in their own subshell and this shell is not interactive. It only opens to execute the script and closes immediately once the script is finished.
    • non-interactive login shell: This is extremely rare, and you’re unlikey to encounter it. One way of launching one is echo command | ssh server. When ssh is launched without a command (so ssh instead of ssh command which will run command on the remote shell) it starts a login shell. If the stdin of the ssh is not a tty, it starts a non-interactive shell. This is why echo command | ssh server will launch a non-interactive login shell. You can also start one with bash -l -c command.

    If you want to play around with this, you can test for the various types of shell as follows:

    • Is this shell interactive?Check the contents of the $- variable. For interactive shells, it will include i: 
      ## Normal shell, just running a command in a terminal: interacive
$ echo $-
himBHs
## Non interactive shell
$ bash -c 'echo $-'
hBc
    • Is this a login shell?There is no portable way of checking this but, for bash, you can check if the login_shell option is set: 
      ## Normal shell, just running a command in a terminal: interacive
$ shopt login_shell 
login_shell     off
## Login shell; 
$ ssh localhost
$ shopt login_shell 
login_shell     on

    Putting all this together, here’s one of each possible type of shell:

    ## Interactive, non-login shell. Regular terminal
$ echo $-; shopt login_shell
himBHs
login_shell     off

## Interactive login shell
$ bash -l
$ echo $-; shopt login_shell
himBHs
login_shell     on

## Non-interactive, non-login shell
$ bash -c 'echo $-; shopt login_shell'
hBc
login_shell     off

## Non-interactive login shell
$ echo 'echo $-; shopt login_shell' | ssh localhost
Pseudo-terminal will not be allocated because stdin is not a terminal.
hBs
login_shell     on
  • [Wayback] Difference between Login Shell and Non-Login Shell? – Unix & Linux Stack Exchange

    A login shell is the first process that executes under your user ID when you log in for an interactive session. The login process tells the shell to behave as a login shell with a convention: passing argument 0, which is normally the name of the shell executable, with a - character prepended (e.g. -bash whereas it would normally be bash. Login shells typically read a file that does things like setting environment variables: /etc/profile and ~/.profile for the traditional Bourne shell, ~/.bash_profile additionally for bash†, /etc/zprofile and ~/.zprofile for zsh†, /etc/csh.login and ~/.login for csh, etc.

    When you log in on a text console, or through SSH, or with su -, you get an interactive login shell. When you log in in graphical mode (on an X display manager), you don’t get a login shell, instead you get a session manager or a window manager.

    It’s rare to run a non-interactive login shell, but some X settings do that when you log in with a display manager, so as to arrange to read the profile files. Other settings (this depends on the distribution and on the display manager) read /etc/profile and ~/.profile explicitly, or don’t read them. Another way to get a non-interactive login shell is to log in remotely with a command passed through standard input which is not a terminal, e.g. ssh example.com <my-script-which-is-stored-locally (as opposed to ssh example.com my-script-which-is-on-the-remote-machine, which runs a non-interactive, non-login shell).

    When you start a shell in a terminal in an existing session (screen, X terminal, Emacs terminal buffer, a shell inside another, etc.), you get an interactive, non-login shell. That shell might read a shell configuration file (~/.bashrc for bash invoked as bash, /etc/zshrc and ~/.zshrc for zsh, /etc/csh.cshrc and ~/.cshrc for csh, the file indicated by the ENV variable for POSIX/XSI-compliant shells such as dash, ksh, and bash when invoked as sh, $ENV if set and ~/.mkshrc for mksh, etc.).

    When a shell runs a script or a command passed on its command line, it’s a non-interactive, non-login shell. Such shells run all the time: it’s very common that when a program calls another program, it really runs a tiny script in a shell to invoke that other program. Some shells read a startup file in this case (bash runs the file indicated by the BASH_ENV variable, zsh runs /etc/zshenv and ~/.zshenv), but this is risky: the shell can be invoked in all sorts of contexts, and there’s hardly anything you can do that might not break something.

    † I’m simplifying a little, see the manual for the gory details.

If you want to avoid the [[...]] bashishm, then read [Wayback] Bashism: How to make bash scripts work in dash – Greg’s Wiki.

–jeroen

Posted in *nix, *nix-tools, ash/dash, bash, bash, Communications Development, Conference Topics, Conferences, Development, Event, Internet protocol suite, Power User, Scripting, SFTP, Software Development, SSH, TCP | Leave a Comment »

Zypper: list info on all patterns, so you can find out which pattern provides a package

Posted by jpluimers on 2022/05/11

I wanted to know which pattern provides [WayBack] etckeeper which is in the [WayBack] openSUSE Software package etckeeper.

It seems no built-in search query can do that, so I built one my own.

Since the result takes quite a while to produce, the output is a pattern.txt that you can manually search.

This is the command:

zypper search -t pattern | grep "|" | tail -n +2 | perl -pe 's/^.*? \| //' | perl -pe 's/ *\| .*$//' | xargs -I {} sh -c "zypper info -t pattern {}" > patterns.txt

The content is like this (the 2017 date shows I wrote this a long time ago):

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, etckeeper, Linux, Perl, Power User, Scripting, sed, Software Development | Leave a Comment »

Kris on Twitter is a bit radical against shell scripts. Learn why.

Posted by jpluimers on 2022/04/13

I say to people: only use shell interactively, don’t write scripts. Never. Not one.
But Kris, they ask, why so radical?
Because of this:

is the literal English Google Translation of the German text

Ich sage den Leuten: benutzt Shell nur interaktiv, schreibt keine Scripte. Nie. Nicht eines.
Aber Kris, fragen sie, wieso so Radikal?
Deswegen:

then links to [Wayback/Archive] Jan Schaumann on Twitter: “TIL zgrep(1) is a shell script. BSD basically does “zcat | grep”, but GNU does “gzip -dc | sed”. How did I learn that? The fun way! CVE-2022-1271, arbitrary-file-write and code execution vulnerability in GNU zgrep / gzip. …”:

Read the rest of this entry »

Posted in *nix, *nix-tools, Apple, ash/dash, ash/dash development, bash, bash, BSD, Development, Mac, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »

Jilles posted a small script to show offline/online status based on ping

Posted by jpluimers on 2022/03/16

Jilles posted a small script to show offline/online status based on ping in [Archive.is] Jilles on Twitter: “#!/bin/bash HOST={1ST HOP HERE} while true;do p=$(ping -c1 $HOST) if [ $? -ne 0 ];then s=offline else s=online\ fi echo $(date +%F\ %T) $s – $(echo $p | sed -e ‘s/^PING.*— 1/1/g’) sleep 10 done”

#!/bin/bash
HOST={1ST HOP HERE}
while true;do
  p=$(ping -c1 $HOST)
  if [ $? -ne 0 ];then
    s=offline
  else
    s=online\ 
  fi
  echo $(date +%F\ %T) $s - $(echo $p | sed -e 's/^PING.*--- 1/1/g')
  sleep 10
done

The reason was that his ISP had connection problems for the block of homes where Jilles lives.

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, grep, Power User, Scripting, sed, Software Development | Leave a Comment »

« Previous Entries
 
%d bloggers like this: