TL;DR
There is a non-standard -r option to xargs that allows it to skip executing when there are no arguments at all.
On some operating systems, the -r is default.
MacOS has no -r, but does not execute xargs if there are no arguments given.
Archive for the ‘bash’ Category
-r argument to pipe (no argument for MacOS)- If no input is given to xargs, don’t let xargs run the utility – Unix & Linux Stack Exchange
Posted by jpluimers on 2021/07/28
Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development, xargs | Leave a Comment »
Listing information on all active interfaces on MacOS part 2: adding DHCP/BOOTP and routing details
Posted by jpluimers on 2021/07/27
This is a continuation of yesterdays
Listing information on all active interfaces on MacOS part 1: getting the active interface names.
It is based on ideas in these StackExchange posts:
- [WayBack] shell – Extracting ip address and interface name from ifconfig – Stack Overflow
- [WayBack] bash – Show active interfaces in terminal – Stack Overflow
I threw most of the implementation details in the ideas away, as they were way to much based on empirical trial and error, than proper research.
So I tried doing the research and came up with the things below.
Getting the IPv4 address and DHCP/BOOTP information of a NIC
By using the ipconfig command, you can get specific details for a NIC like an IPv4 (with the getifaddr) or DHCP (with the getpacket option to get the latest DHCP packet):
for i in $(ifconfig -l -u); do if ifconfig $i | grep -q "status: active" ; then echo $i; fi; done | xargs -n1 -I_nic_ sh -c 'echo "_nic_: $(ipconfig getifaddr _nic_)"'
or DHCP/BOOTP:
for i in $(ifconfig -l -u); do if ifconfig $i | grep -q "status: active" ; then echo $i; fi; done | xargs -n1 -I_nic_ sh -c 'echo "_nic_: $(ipconfig getpacket _nic_)"'
The latter returns a very long list, which I wanted to shorten into a more readable format.
ipconfig syntax
You can find more information in the [Archive.is] ipconfig(8) [osx man page] / [WayBack] ipconfig Man Page – macOS – SS64.com excerpt:
Share this:
Posted in *nix, *nix-tools, Apple, bash, Development, DNS, ifconfig, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »
Filezilla: figuring out the cause of “Connection timed out after 20 seconds of inactivity”
Posted by jpluimers on 2021/07/21
On one of my Raspberry Pi boxes, somehow I could not access files over SFTP (SSH File Transfer Protocol) via FileZilla.
I would consistently get this error:
"Connection timed out after 20 seconds of inactivity"
Figuring the exact cause took a while.
TL;DR: SFTP uses an interactive non-login shell, then interprets the output from that shell. For that kind of shell, ensure few or none scripts run that output text.
These links finally got me to the cause
Share this:
Posted in *nix, *nix-tools, bash, bash, Communications Development, Conference Topics, Conferences, Development, Event, Internet protocol suite, Power User, Scripting, SFTP, Software Development, SSH, TCP | Leave a Comment »
Busybox sh (actually ash derivative dash): checking exit codes
Posted by jpluimers on 2021/04/20
Even if you include a double quotes "sh" in a Google search to force only sh (in the early days this was the Thompson shell, but nowadays usually a Bourne shell or derivative) results, almost all unix like scripting examples you find are based on bash (the Bourne again shell), so I was glad I dug a bit deeper into what the actual Busybox shell is.
I wanted to know which shell Busybox uses and what capabilities it has, as ESXi ships with this very slimmed down set of tools (called applets in Busybox speak).
It does not even include ssh: that gap is often filled by [Wayback] Dropbear SSH, which was used by ESXi and named dbclient (I think with ESXi 6.0 it was replaced with a more regular ssh implementation): [Wayback] How to compile a statically linked rsync binary for ESXi.
Busybox shell source code is at [Wayback] ash.c\shell – busybox – BusyBox: The Swiss Army Knife of Embedded Linux and indicates the shell is the ash (the Almquist shell) derivative dash (yes, you guessed it right: the Debian Almquist shell), ported from NetBSD and debianized:
* Copyright (c) 1997-2005 Herbert Xu <herbert@gondor.apana.org.au> * was re-ported from NetBSD and debianized. ... //config: The most complete and most pedantically correct shell included with //config: busybox. This shell is actually a derivative of the Debian 'dash' //config: shell (by Herbert Xu), which was created by porting the 'ash' shell //config: (written by Kenneth Almquist) from NetBSD.
nx like systems have a shell hell similar to Windows DLL hell: there are too many, and their differences and be both subtle and frustrating. To get a feel, browse through Source: Comparison of command shells – Wikipedia (yes, some shells from other operating environments like DOS, OS/2, VMS and Windows, but the majority is nx).
Since ash is sufficiently different from bash (for example [Wayback] ash – exit code for a piped process), I always want to know what shell code (which often comes from bash as it is so ubiquitous) will work.
There is hardly any shell documentation at the Busybox site. There is [Wayback] BusyBox – The Swiss Army Knife of Embedded Linux, the source code at [Wayback] ash.c\shell – busybox – BusyBox: The Swiss Army Knife of Embedded Linux does not offer much either,
A manual page of it is at [Archive.is] ash(1) [minix man page]. There you see the age: back then, “exit status” is used where nowadays many people would use “exit code”. It does not explain how to check for specific exit codes.
Because ash is derived from the Bourne shell, this page was of great help for me to grasp exit code handing: [Wayback] Exit Codes – Shell Scripting Tutorial
A Bourne Shell Programming / Scripting Tutorial for learning about using the Unix shell.
Here two examples from that page to get me going:
#!/bin/sh # Second attempt at checking return codes grep "^${1}:" /etc/passwd > /dev/null 2>&1 if [ "$?" -ne "0" ]; then echo "Sorry, cannot find user ${1} in /etc/passwd" exit 1 fi USERNAME=`grep "^${1}:" /etc/passwd|cut -d":" -f1` NAME=`grep "^${1}:" /etc/passwd|cut -d":" -f5` HOMEDIR=`grep "^${1}:" /etc/passwd|cut -d":" -f6` echo "USERNAME: $USERNAME" echo "NAME: $NAME" echo "HOMEDIR: $HOMEDIR"
and
#!/bin/sh # A Tidier approach check_errs() { # Function. Parameter 1 is the return code # Para. 2 is text to display on failure. if [ "${1}" -ne "0" ]; then echo "ERROR # ${1} : ${2}" # as a bonus, make our script exit with the right error code. exit ${1} fi } ### main script starts here ### grep "^${1}:" /etc/passwd > /dev/null 2>&1 check_errs $? "User ${1} not found in /etc/passwd" USERNAME=`grep "^${1}:" /etc/passwd|cut -d":" -f1` check_errs $? "Cut returned an error" echo "USERNAME: $USERNAME" check_errs $? "echo returned an error - very strange!"
This basically means that status code handling is the same as in bash, so constructs can be used like [Wayback] bash – How to check the exit status using an if statement – Stack Overflow:
$?is a parameter like any other. You can save its value to use before ultimately callingexit.exit_status=$? if [ $exit_status -eq 1 ]; then echo "blah blah blah" fi exit $exit_status
Share this:
Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, BusyBox, Development, Power User, Scripting, Software Development, ssh/sshd | 1 Comment »
linux – How can I execute a series of commands in a bash subshell as another user using sudo? – Stack Overflow
Posted by jpluimers on 2021/03/31
Based on [WayBack] linux – How can I execute a series of commands in a bash subshell as another user using sudo? – Stack Overflow:
alias restart-spotlight-service-as-root="sudo bash -c 'echo stop;launchctl stop com.apple.metadata.mds;echo start;launchctl start com.apple.metadata.mds;echo started'"
The bold bits above sudo bash -c 'echo stop;launchctl stop com.apple.metadata.mds;echo start;launchctl start com.apple.metadata.mds;echo started' allow the commands between single quotes to executed in one new bash shell under sudo.
–jeroen
Share this:
Posted in *nix, *nix-tools, Apple, bash, bash, Development, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »
GitHub – pastpages/savepagenow: A simple Python wrapper for archive.org’s “Save Page Now” capturing service
Posted by jpluimers on 2021/03/11
This makes it way easier to save WayBack content:
[WayBack] GitHub – pastpages/savepagenow: A simple Python wrapper for archive.org’s “Save Page Now” capturing service
A poor-mans alternative is the below bash script from [WayBack] Saving of public Google+ content at the Internet Archive’s Wayback Machine by the Archive Team has begun : plexodus:
For Linux, MacOS / OSX, BSD, and other Unix-like operating systems (including Android with Termux, or Windows, with a Unix/Linux environment), the following script (I’ve saved this as
archive-url) will archive the requested URL:#!/bin/bash # archive-url # Archive selected URL at the Internet Archive curl -s -I -H "Accept: application/json" "https://web.archive.org/save/${1}" | grep '^x-cache-key:' | sed "s,https,&://,; s,\(${1}\).*$,\1,"Save that to your execution path (I’ve chosen
~/bin, you might use/usr/local/binor another location on your$PATH, and invoke as, say (again referring to the G+MM homepage):$ archive-url https://plus.google.com/communities/112164273001338979772If you have a list of URLs in a file (or pipelined from command output), you can request all of them to be archived in a simple bash loop. I’m using xargs here to run ten simultaneous requests from the file
gplus-urllist:cat gplus_urllist | while read url do xargs -I{} -P 10 archive-url {}; doneI’ve run this on over 10,000 URLs over a modest residential broadband connection in a hair over two hours.
Note that such requests trigger an archive by the Internet Archive from one of its archiving nodes, you’re not sending the page to the Archive yourself. In particular, archival from regions defaulting to another language may result in the Google+ site content (but not post or comments) being in a different language. I’ve frequently seen my pages turning up in Japanese, for instance.
–jeroen
Share this:
Posted in bash, Development, Python, Scripting, Software Development | Leave a Comment »
explainshell.com: parse and explain just about any shell command
Posted by jpluimers on 2021/02/17
I bumped into the tremendously site [WayBack] explainshell.com – match command-line arguments to their help text only after documenting the relevant cURL options of yesterdays post on checking your CertBot domain expiration dates.
The site allows put in a shell command-line to see the help text that, including matches for each argument.
It works so well because it parses both the shell command-line and the man pages, then constructs a web-page linking the relevant man page content to the shell command-line in the correct shell command-line order.
The
explainshellhas a counterpart showthedocs (both are open source) for explaining other languages (on the one hand more extended as it goes much deeper into parsing for instance SQL, on the other hand more limited as it only supports a few languages). More onshowthedocslater.
The links
- [WayBack] explainshell.com – about.
- [WayBack] GitHub – idank/explainshell: match command-line arguments to their help text
The parsing results
The first bit below is just the text output, and the second bit the screenshot, of a relatively simple command like [WayBack] explainshell.com – curl -fsSL example.org:
-f, --fail (HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal with failed attempts. In normal cases when a HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag will prevent curl from outputting that and return error 22. This method is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).-s, --silent Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute.-S, --show-error When used with -s it makes curl show an error message if it fails.-L, --location (HTTP/HTTPS) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place. If used together with -i, --include or -I, --head, headers from all requested pages will be shown. When authentication is used, curl only sends its credentials to the initial host. If a redirect takes curl to a different host, it won't be able to intercept the user+password. See also --location-trusted on how to change this. You can limit the amount of redirects to follow by using the --max-redirs option. When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will do the following request with a GET if the HTTP response was 301, 302, or 303. If the response code was any other 3xx code, curl will re-send the following request using the same unmodified method.source manpages: curl
The screenshot is even more impressive:
Share this:
Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »
Checking expiration dates for your certbot certificates
Posted by jpluimers on 2021/02/16
I have these two little aliases in my toolbox:
alias "certbot-check-all-by-file=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live" alias "certbot-check-all-by-config=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --server ISPconfig"
First a big fat warning
do not run just any script downloaded through curl. Plenty of reasons why this is dangerous:
- [WayBack] Detecting the use of “curl | bash” server side | Application Security
- [WayBack] One way “curl pipe sh” install scripts can be dangerous [proof of concept] / Jordan Eldredge: script content differs depending on user agent
- [WayBack] sean cassidy : Don’t Pipe to your Shell: scripts having different behaviour when executed partially
- [WayBack] Why using curl | sudo sh is not advised? – Stack Overflow:
You can proof your scripts against partial execution by putting the whole thing into the body of a function, and executing that function on the last line. If you know a script is defined like that, it’s exactly as secure as downloading and then executing some installer.
The first three can mostly prevented by using your own fork of the script repository, then checking each modification of the script, combined with ensuring your fork location does not throw tricks 1 or 2 on you.
That’s why I run the above alias only from a checkssl fork which I can inspect.
Back to the alias
The aliases use quite a few tricks:
- Having curl download a command minimising
curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checksslVia: [WayBack] The missing package manager for macOS (or Linux) — The missing package manager for macOS (or Linux)
- Running that command through bash as if the download were a file by wrapping wrapping it in parenthesis and a less than sign .
bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl)Via: [WayBack] linux – Execute bash script from URL – Stack Overflow
- Passing arguments to the bash command by appending this to the command just like from the regular command-line:
bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live - An
alias[WayBack] with double-quotes around the whole statement:
alias "certbot-check-all-by-file=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live" - Either use the certbot configuration file or apache2 (via
ISPconfigas theapache2parameter value is not yet supported) domain configuration:
alias "certbot-check-all-by-file=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live" alias "certbot-check-all-by-config=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --server ISPconfig"
Using source instead of bash
Note that an alternative alias is this one:
alias "certbot-check-all-by-file=(source <(curl -s https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live)"
However, that needs an extra set of parenthesis, otherwise you will get bumped out of your current shell.
The reason is that bash runs in a [WayBack] subshell, whereas [WayBack] source (and the equivalent [WayBack] “dot” command .) runs in the current shell, but the script performs a gracefull_exit or error_exit which end in an exit [WayBack] terminating the current shell.
The [WayBack] () parenthesis around the source command ensure it runs in a sub-shell.
In this case, you can still pass the --location /etc/letsencrypt/live parameters, as bash is the only shell allowing this: [WayBack] source – Passing variables to a bash script when sourcing it – Unix & Linux Stack Exchange.
References
Related [WayBack] Advanced Bash-Scripting Guide topics:
- [WayBack] Aliases
- [WayBack] Exit and Exit Status
- [WayBack] Special Characters: “dot” command [period]
- [WayBack] Special Characters: ()
- [WayBack] Internal Commands and Builtins: source
- [WayBack] Subshells
Related cURL options from [WayBack] curl – How To Use:
-f: [WayBack]-f,--fail
(HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal with failed attempts. In normal cases when an HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag will prevent curl from outputting that and return error 22.
This method is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).
-s: [WayBack]-s, --silent:
Silent or quiet mode. Don’t show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it.
Use -S, –show-error in addition to this option to disable progress meter but still show error messages.
See also -v, –verbose and –stderr.
-S: [WayBack]-S, --show-errors:
When used with -s, –silent, it makes curl show an error message if it fails.
-L: [WayBack]-L, --location:
(HTTP) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place. If used together with -i, –include or -I, –head, headers from all requested pages will be shown. When authentication is used, curl only sends its credentials to the initial host. If a redirect takes curl to a different host, it won’t be able to intercept the user+password. See also –location-trusted on how to change this. You can limit the amount of redirects to follow by using the –max-redirs option.
When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will do the following request with a GET if the HTTP response was 301, 302, or 303. If the response code was any other 3xx code, curl will re-send the following request using the same unmodified method.
You can tell curl to not change the non-GET request method to GET after a 30x response by using the dedicated options for that: –post301, –post302 and –post303.
Reminders
Reminder to self: see if JSON output is viable. This commit might help.
–jeroen
Share this:
Posted in bash, Conference Topics, Conferences, Development, Event, Scripting, Software Development | Leave a Comment »
bash – convert comma separated values into a list of values using shell script – Stack Overflow
Posted by jpluimers on 2020/09/07
For a simple comma separated list (no quotes), I was expecting a sed script (and indeed it is possible), but tr is more elegant:
Use
trto change , into newlines:
tr , "\n" < list.txt
Source: [WayBack] bash – convert comma separated values into a list of values using shell script – Stack Overflow.
–jeroen
Share this:
Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »
find – display only files starting with . (hidden) – Unix & Linux Stack Exchange
Posted by jpluimers on 2020/08/21
find . -type f -name '\.*' -printMust work if you want list every hidden file down in the directory hierarchy.
This sort of works on Linux, but fails on VMware ESXi (on Linux it only works when applying -maxdepth 1, deeper levels fails because they list all files where the top directory starts with a .):
If you want hidden files and hidden directories, without . and .. :find -regex '\./\..+' -print
This works on both Linux and VMware ESXi:
If you want hidden files and hidden directories, without . and .. :
find . \( -type f -o -type d \) -name '\.*' -print
Based on:
- [WayBack] find – display only files starting with . (hidden) – Unix & Linux Stack Exchange
- [WayBack] shell – `find -name` pattern that matches multiple patterns – Stack Overflow
–jeroen
Share this:
Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »
The Wiert Corner - irregular stream of stuff 