The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘bash’ Category

« Previous Entries
Next Entries »

GitHub – pastpages/savepagenow: A simple Python wrapper for archive.org’s “Save Page Now” capturing service

Posted by jpluimers on 2021/03/11

This makes it way easier to save WayBack content:

[WayBack] GitHub – pastpages/savepagenow: A simple Python wrapper for archive.org’s “Save Page Now” capturing service

A poor-mans alternative is the below bash script from [WayBack] Saving of public Google+ content at the Internet Archive’s Wayback Machine by the Archive Team has begun : plexodus:

For Linux, MacOS / OSX, BSD, and other Unix-like operating systems (including Android with Termux, or Windows, with a Unix/Linux environment), the following script (I’ve saved this as archive-url) will archive the requested URL:

#!/bin/bash
# archive-url
# Archive selected URL at the Internet Archive

curl -s -I -H "Accept: application/json" "https://web.archive.org/save/${1}" |
grep '^x-cache-key:' | sed "s,https,&://,; s,\(${1}\).*$,\1,"

Save that to your execution path (I’ve chosen ~/bin, you might use /usr/local/bin or another location on your $PATH, and invoke as, say (again referring to the G+MM homepage):

$ archive-url https://plus.google.com/communities/112164273001338979772

If you have a list of URLs in a file (or pipelined from command output), you can request all of them to be archived in a simple bash loop. I’m using xargs here to run ten simultaneous requests from the file gplus-urllist:

cat gplus_urllist | while read url do xargs -I{} -P 10 archive-url {}; done

I’ve run this on over 10,000 URLs over a modest residential broadband connection in a hair over two hours.

Note that such requests trigger an archive by the Internet Archive from one of its archiving nodes, you’re not sending the page to the Archive yourself. In particular, archival from regions defaulting to another language may result in the Google+ site content (but not post or comments) being in a different language. I’ve frequently seen my pages turning up in Japanese, for instance.

–jeroen

Posted in bash, Development, Python, Scripting, Software Development | Leave a Comment »

explainshell.com: parse and explain just about any shell command

Posted by jpluimers on 2021/02/17

I bumped into the tremendously site [WayBack] explainshell.com – match command-line arguments to their help text only after documenting the relevant cURL options of yesterdays post on checking your CertBot domain expiration dates.

The site allows put in a shell command-line to see the help text that, including matches for each argument.

It works so well because it parses both the shell command-line and the man pages, then constructs a web-page linking the relevant man page content to the shell command-line in the correct shell command-line order.

The explainshell has a counterpart showthedocs (both are open source) for explaining other languages (on the one hand more extended as it goes much deeper into parsing for instance SQL, on the other hand more limited as it only supports a few languages). More on showthedocs later.

The links

The parsing results

The first bit below is just the text output, and the second bit the screenshot, of a relatively simple command like [WayBack] explainshell.com – curl -fsSL example.org:

curl(1) -fsSL example.org
transfer a URL
 
-f, --fail
       (HTTP)  Fail  silently  (no  output at all) on server errors. This is mostly done to better enable
       scripts etc to better deal with failed attempts. In normal cases  when  a  HTTP  server  fails  to
       deliver  a  document,  it  returns an HTML document stating so (which often also describes why and
       more). This flag will prevent curl from outputting that and return error 22.

       This method is not fail-safe and there are occasions where non-successful response codes will slip
       through, especially when authentication is involved (response codes 401 and 407).
 
-s, --silent
       Silent or quiet mode. Don't show progress meter or error messages.  Makes Curl mute.
 
-S, --show-error
       When used with -s it makes curl show an error message if it fails.
 
-L, --location
       (HTTP/HTTPS) If the server reports that the requested page  has  moved  to  a  different  location
       (indicated  with  a Location: header and a 3XX response code), this option will make curl redo the
       request on the new place. If used together with -i, --include or  -I,  --head,  headers  from  all
       requested pages will be shown. When authentication is used, curl only sends its credentials to the
       initial host. If a redirect takes curl to a different host, it won't  be  able  to  intercept  the
       user+password.  See  also  --location-trusted  on  how to change this. You can limit the amount of
       redirects to follow by using the --max-redirs option.

       When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will
       do  the  following  request  with a GET if the HTTP response was 301, 302, or 303. If the response
       code was any other 3xx code, curl will re-send the following request  using  the  same  unmodified
       method.
source manpages: curl

The screenshot is even more impressive:

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

Checking expiration dates for your certbot certificates

Posted by jpluimers on 2021/02/16

I have these two little aliases in my toolbox:

alias "certbot-check-all-by-file=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live"
alias "certbot-check-all-by-config=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --server ISPconfig"

First a big fat warning

do not run just any script downloaded through curl. Plenty of reasons why this is dangerous:

  1. [WayBack] Detecting the use of “curl | bash” server side | Application Security
  2. [WayBack] One way “curl pipe sh” install scripts can be dangerous [proof of concept] / Jordan Eldredge: script content differs depending on user agent
  3. [WayBack] sean cassidy : Don’t Pipe to your Shell: scripts having different behaviour when executed partially
  4. [WayBack] Why using curl | sudo sh is not advised? – Stack Overflow:

    You can proof your scripts against partial execution by putting the whole thing into the body of a function, and executing that function on the last line. If you know a script is defined like that, it’s exactly as secure as downloading and then executing some installer.

The first three can mostly prevented by using your own fork of the script repository, then checking each modification of the script, combined with ensuring your fork location does not throw tricks 1 or 2 on you.

That’s why I run the above alias only from a checkssl fork which I can inspect.

Back to the alias

The aliases use quite a few tricks:

  1. Having curl download a command minimising
    curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl

    Via: [WayBack] The missing package manager for macOS (or Linux) — The missing package manager for macOS (or Linux)

  2. Running that command through bash as if the download were a file by wrapping wrapping it in parenthesis and a less than sign .
    bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl)

    Via: [WayBack] linux – Execute bash script from URL – Stack Overflow

  3. Passing arguments to the bash command by appending this to the command just like from the regular command-line:
    bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live
  4. An alias [WayBack] with double-quotes around the whole statement:
    alias "certbot-check-all-by-file=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live"
  5. Either use the certbot configuration file or apache2 (via ISPconfig as the apache2 parameter value is not yet supported) domain configuration:
    alias "certbot-check-all-by-file=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live"
alias "certbot-check-all-by-config=bash <(curl -fsSL https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --server ISPconfig"

Using source instead of bash

Note that an alternative alias is this one:

alias "certbot-check-all-by-file=(source <(curl -s https://raw.githubusercontent.com/srvrco/checkssl/master/checkssl) --location /etc/letsencrypt/live)"

However, that needs an extra set of parenthesis, otherwise you will get bumped out of your current shell.

The reason is that bash runs in a [WayBack] subshell, whereas [WayBack] source (and the equivalent [WayBack] “dot” command .) runs in the current shell, but the script performs a gracefull_exit or error_exit which end in an exit [WayBack] terminating the current shell.

The [WayBack] () parenthesis around the source command ensure it runs in a sub-shell.

In this case, you can still pass the --location /etc/letsencrypt/live parameters, as bash is the only shell allowing this: [WayBack] source – Passing variables to a bash script when sourcing it – Unix & Linux Stack Exchange.

References

Related [WayBack] Advanced Bash-Scripting Guide topics:

Related cURL options from [WayBack] curl – How To Use:

  • -f: [WayBack] -f, --fail

    (HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal with failed attempts. In normal cases when an HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag will prevent curl from outputting that and return error 22.

    This method is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).

  • -s: [WayBack] -s, --silent:

    Silent or quiet mode. Don’t show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it.

    Use -S, –show-error in addition to this option to disable progress meter but still show error messages.

    See also -v, –verbose and –stderr.

  • -S: [WayBack] -S, --show-errors:

    When used with -s, –silent, it makes curl show an error message if it fails.

  • -L: [WayBack] -L, --location:

    (HTTP) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place. If used together with -i, –include or -I, –head, headers from all requested pages will be shown. When authentication is used, curl only sends its credentials to the initial host. If a redirect takes curl to a different host, it won’t be able to intercept the user+password. See also –location-trusted on how to change this. You can limit the amount of redirects to follow by using the –max-redirs option.

    When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will do the following request with a GET if the HTTP response was 301, 302, or 303. If the response code was any other 3xx code, curl will re-send the following request using the same unmodified method.

    You can tell curl to not change the non-GET request method to GET after a 30x response by using the dedicated options for that: –post301, –post302 and –post303.

Reminders

Reminder to self: see if JSON output is viable. This commit might help.

–jeroen

Posted in bash, Conference Topics, Conferences, Development, Event, Scripting, Software Development | Leave a Comment »

bash – convert comma separated values into a list of values using shell script – Stack Overflow

Posted by jpluimers on 2020/09/07

For a simple comma separated list (no quotes), I was expecting a sed script (and indeed it is possible), but tr is more elegant:

Use tr to change , into newlines:

tr , "\n" < list.txt

See https://en.wikipedia.org/wiki/Tr_(Unix)

Source: [WayBack] bash – convert comma separated values into a list of values using shell script – Stack Overflow.

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

find – display only files starting with . (hidden) – Unix & Linux Stack Exchange

Posted by jpluimers on 2020/08/21

find . -type f -name '\.*' -print

Must work if you want list every hidden file down in the directory hierarchy.

This sort of works on Linux, but fails on VMware ESXi (on Linux it only works when applying -maxdepth 1, deeper levels fails because they list all files where the top directory starts with a .):

If you want hidden files and hidden directories, without . and .. :

find -regex '\./\..+' -print

This works on both Linux and VMware ESXi:

If you want hidden files and hidden directories, without . and .. :

find . \( -type f -o -type d \) -name '\.*' -print

Based on:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

linux – How do I use sudo to redirect output to a location I don’t have permission to write to? – Stack Overflow

Posted by jpluimers on 2020/07/09

Various ways are explained at [WayBack] linux – How do I use sudo to redirect output to a location I don’t have permission to write to? – Stack Overflow.

Some are for simple commands and can be a one liner (for instance using tee, or executing a secondary shell).

Others are more suited for longer command sequences.

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

How to return a string value from a Bash function – Stack Overflow

Posted by jpluimers on 2020/06/17

Cool: you can return strings both as a function result, and by reference: they are explained in the question, second and fourth answer of [WayBack] How to return a string value from a Bash function – Stack Overflow.

Returning them by reference has two important benefits:

  1. it is much faster (especially useful in tight loop)
  2. you can use echo (normally used to return a result) for debugging purposes

I also needed a bit of switch magic which I found at [WayBack] bash – Switch case with fallthrough? – Stack Overflow and array magic (from [WayBack] Array variables) as arrays are far more readable than indirection (on the why not, see [WayBack] BashFAQ/006 – Greg’s Wiki: How can I use variable variables (indirect variables, pointers, references) or associative arrays?).

So here is a function that returns a specific IPv4 octet.

function getIpv4Octet() {
  IPv4=$1
  octetIndex=$2
  outputVariable=$3

  slice="${IPv4}"
  count=1
  while [ "${count}" -le 4 ]
  do
    octet[${count}]="${slice%%.*}"
    slice="${slice#*.}"
    count=$((count+1))
  done
   
  case "${octetIndex}" in
    "1" | "2" | "3" | "4")
      ;;
    *)
      octetIndex="4"
      ;;
  esac
  eval $outputVariable="${octet[$octetIndex]}"
}

You call it like this:

$ getIpv4Octet "192.168.178.32" 3 result && echo ${result}
178

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

how to count the length of an array defined in bash? – Unix & Linux Stack Exchange

Posted by jpluimers on 2020/06/16

I needed to enumerate all the parameters to a function and access many of them by index in the same function, so thanks to both these:

I got at this:

  declare -a arguments=("$@")
  for index in ${!arguments[@]}; do
    echo $index/${#arguments[@]}:${arguments[$index]}
  done

These are all forms of Array handling or Shell Parameter Expansion with special cases for array variables:

  • ! does indirection, in this case from the array to the index of the array
  • # gets the lengt of the parameter (for arrays: the number of elements)
  • [] acccesse an array variable using an index

Via:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

One day I will dig in the various ways that bash can do evaluation, for now: there is eval, “ and $() and I’m not sure when to choose which.

Posted by jpluimers on 2020/06/10

A while ago, I had to execute a series of aliases of which the names were stored in an array. A simple for loop with en eval call did the job.

Then I found out there are at least two more ways of evaluation in bash, so here are just a few links giving me a head start if I ever dig this up again:

Note that looping over parameters is different than over an array: [WayBack] Loop through an array of strings in Bash? – Stack Overflow

ou can use it like this:

## declare an array variable
declare -a arr=("element1" "element2" "element3")

## now loop through the above array
for i in "${arr[@]}"
do
   echo "$i"
   # or do whatever with individual element of the array
done

# You can access them using echo "${arr[0]}", "${arr[1]}" also

Also works for multi-line array declaration

declare -a arr=("element1" 
                "element2" "element3"
                "element4"
                )

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

bash + sed: quadruple backslash for proper escaping within an alias

Posted by jpluimers on 2020/06/09

This is part of a bash alias where I had to use quadruple backslash in order to escape it for sed:

# The sed with quad //// is to prevent 'unterminated substitute in regular expression':
alias x='... | sed "s/=.*/ \\\\/"'

This is needed because bash will escape \\\\ into \\ which sed then escapes to \.

The easiest way to find this is to replace the sed with echo to see the expansion.

References:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sed, Software Development | 2 Comments »

« Previous Entries
Next Entries »