The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘bash’ Category

Some insights on how readlink approached canonicalisation of a filename having symlinks

Posted by jpluimers on 2022/03/03

Cool, I didn’t realise how readlink operated, but found out a bit more in the answers to [Wayback] symlink – How to get full path of original file of a soft symbolic link? – Unix & Linux Stack Exchange, thanks to [Wayback] daisy, [Wayback] Peter.O and [Wayback] Gilles ‘SO- stop being evil’:

  • Try this line:
    readlink -f `which command`
    

    If command is in your $PATH variable , otherwise you need to specify the path you know.

    -f will return a path to a non-existent final target, so long as the intermediate link targets exist… Use -e to avoid this, ie. -e will return null if the final target does not exist. – Peter.O

  • Under Linux, readlink reads the contents of a symlink, and readlink -f follows symlinks to symlinks to symlinks, etc., until it finds something that isn’t a symlink.

–jeroen

Posted in *nix, *nix-tools, ash/dash, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

Bash functions to encode and decode the ‘Basic’ HTTP Authentication Scheme

Posted by jpluimers on 2022/02/24

IoT devices still often use the ‘Basic’ HTTP Authentication Scheme for authorisation, see [Wayback] RFC7617: The ‘Basic’ HTTP Authentication Scheme (RFC ) and [Wayback] RFC2617: HTTP Authentication: Basic and Digest Access Authentication (RFC ).

Often this authentication is used even over http instead of over https, for instance the Egardia/Woonveilig alarm devices I wrote about yesterday at  Egardia/Woonveilig: some notes about logging on a local gateway to see more detailed information on the security system. This is contrary to guidance in:

  • RFC7617:
       This scheme is not considered to be a secure method of user
       authentication unless used in conjunction with some external secure
       system such as TLS (Transport Layer Security, [RFC5246]), as the
       user-id and password are passed over the network as cleartext.
  • RFC2617:
       "HTTP/1.0", includes the specification for a Basic Access
       Authentication scheme. This scheme is not considered to be a secure
       method of user authentication (unless used in conjunction with some
       external secure system such as SSL [5]), as the user name and
       password are passed over the network as cleartext.

Fiddling with those alarm devices, I wrote these two little bash functions (with a few notes) that work both on MacOS and in Linux:

# `base64 --decode` is platform neutral (as MacOS uses `-D` and Linux uses `-d`)
# `$1` is the encoded username:password
function decode_http_Basic_Authorization(){
  echo $1 | base64 --decode
  echo
}

# `base64` without parameters encodes
# `echo -n` does not output a new-line
# `$1` is the username; `$2` is the password
function encode_http_Basic_Authorization(){
  echo $1:$2 | base64
}

The first decodes the <credentials> from a Authorization: Basic <credentials> header into a username:password clean text followed by a newline.

The second one encodes a pair of username and password parameters into such a <credentials> string.

They are based on these initial posts that were not cross platform or explanatory:

  1. [Wayback] Decode HTTP Basic Access Authentication – Stack Pointer
  2. [Wayback] Create Authorization Basic Header | MJ’s Web Log

–jeroen

Posted in *nix, *nix-tools, Apple, Authentication, bash, bash, Communications Development, Development, HTTP, Internet protocol suite, Linux, Mac OS X / OS X / MacOS, Power User, Scripting, Security, Software Development, TCP, Web Development | Leave a Comment »

Viewing the last lines of the postfix log with journalctl (with help from Unix & Linux Stack Exchange)

Posted by jpluimers on 2022/02/24

Two command-lines I use to view my Postfix logs:

  1. journalctl --unit postfix --since "2 days ago"
  2. journalctl --unit postfix --pager-end

Note that neither of these work well with the --follow (or equivalent -f) option, as this will effectively disable the pager (which by default is less).

The second is via [Wayback] systemd – How to see the latest x lines from systemctl service log – Unix & Linux Stack Exchange (which got the--pagerend bit wrong, as it misses a dash and should be --pager-end, but still thanks [Wayback] Daniel Kmak):

Just:

journalctl -u SERVICE_NAME -e

Parameter -e stands for:

-e –pagerend

That’s the one ! Other answers will go through the whole log to get to its end, which can be veeeeery long for large syslogs.

The last bit (by [Wayback] Léo Germond, thanks!) is why I like it most.

Similarly, specifying --since in the first example will not go through the whole log.

Some background information:

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, Development, journalctl and journald, Linux, postfix, Power User, Scripting, Software Development, systemd | Leave a Comment »

I always forget how simple it is to show the definition of bash function or alias (via: Stack Overflow)

Posted by jpluimers on 2022/02/17

I always facepalm myself after looking up this: [Wayback] Can bash show a function’s definition? – Stack Overflow (thanks [Wayback] Benjamin Bannier!):

Use type. If foobar is e.g. defined in your ~/.profile:

$ type foobar
foobar is a function
foobar {
    echo "I'm foobar"
}

type will also expand aliases, which is a nice bonus :) – [Wayback] Esdras Lopez

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

Getting the primary IP address (plain and CIDR) on Linux and OS X, then nmap scan on the associated subnet

Posted by jpluimers on 2021/12/13

The below answer works on my Linux and OS X systems (each having multiple network adapters configured):

[WayBack] bash – How to get the primary IP address of the local machine on Linux and OS X? – Stack Overflow

ip route get 1 | awk '{print $NF;exit}'

For Linux, I have this bash function:

# note the ";exit" lists the first default route interface, as there can be multiple
function nmap-fingerprint_network_of_default_interface() {
        default_if=$(ip route list | awk '/^default/ {print $5;exit}')
        default_if_cidr=$(ip -o -f inet addr show $default_if | awk '{print $4}')
        nmap -sP $default_if_cidr
}

And for OS X this one:

# requires ipcalc
function nmap-fingerprint_network_of_default_interface() {
        default_if=$(route -q -n get default | awk '/interface:/ {print $2;exit}')
        default_if_address=$(ifconfig $default_if | awk '/inet / {print $2;exit}')
        default_if_netmask_hex=$(ifconfig $default_if | awk '/inet / {print $4;exit}')
        default_if_network_bit_count=$(ipcalc --nocolor --class $default_if_address $default_if_netmask_hex)
        default_if_cidr=$(echo "$default_if_address/$default_if_network_bit_count")
        nmap -sP $default_if_cidr
}

These are the variables used:

  • default_if: network interface of the default route
  • default_if_cidr: IPv4 CIDR of the network interface of the default route (see Classless Inter-Domain Routing: CIDR notation – Wikipedia)
  • default_if_address: IPv4 address of network interface of the default route
  • default_if_netmask_hex: hexadecimal IPv4 network mask of network interface of the default route
  • default_if_network_bit_count: number of set bits in the IPv4 network mask of the network interface of the default route

Links used to get the above functions:

I might have gotten away with a pure bash solution (see [WayBack] Bash script for calculating network and broadcast addresses from ip and netmask or CIDR Notation · GitHub or my post Getting your local IPv4 addresses, netmasks and CIDRs), but the above works and is way shorter, and easier to maintain.

In stead of ipcalc, subnetcalc can do the same calculations and also supports IPv6, so that is something for a future try:

–jeroen

Posted in *nix, *nix-tools, Apple, bash, Color (software development), Development, Mac, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »

Some scripts and tips for easing the maintenance of a postfix based SMTP system

Posted by jpluimers on 2021/11/24

A few scripts and tips I found Googling around.

Deleting queued messages by regular expression pattern

I have seen the below delete-from-mailq.pl script numerous time, usually without any attribution (for instance [Wayback] Postfix Flush the Mail Queue – nixCraft and  [Wayback] postfix-delete.pl – Following script deletes all mail from the mailq which matches the regular expression specified as the first argument · GitHub).

The earliest version I could find was in [Wayback] ‘Re: delete messages from mailq’ – MARC by [Wayback] ‘Ralf Hildebrandt ‘ posts – MARC:

--- snip ---
#!/usr/bin/perl

$REGEXP = shift || die "no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!";

@data = qx</usr/sbin/postqueue -p>;
for (@data) {
  if (/^(\w+)\*?\s/) {
     $queue_id = $1;
  }
  if($queue_id) {
    if (/$REGEXP/i) {
      $Q{$queue_id} = 1;
      $queue_id = "";
    }
  }
}
                                
#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;

foreach (keys %Q) {
  print POSTSUPER "$_\n";
};
close(POSTSUPER);
--- snip ---

And then use:
% delete-from-mailq "^test"

 

Tips

[Wayback] How do I check the postfix queue size? – Server Fault

Lots of great answers and pointers to useful guides/software there.

qstat

[Wayback] Postfix Bottleneck Analysis points to [Wayback] Postfix manual – qshape(1): qshape - Print Postfix queue domain and age distribution, then explains about different scenarion and queues:

postqueue

postqueue -p | tail -n 1

Last line in the postqueue -p shows how many requests and size:

-- 317788 Kbytes in 11860 Requests.

View queues size

I tried finding the original posting of the below script, but could not. If you find it, please let me know.

#!/usr/bin/env perl

# postfix queue/s size
# author: 
# source: http://tech.groups.yahoo.com/group/postfix-users/message/255133

use strict;
use warnings;
use Symbol;
sub count {
        my ($dir) = @_;
        my $dh = gensym();
        my $c = 0;
        opendir($dh, $dir) or die "$0: opendir: $dir: $!\n";
        while (my $f = readdir($dh)) {
                if ($f =~ m{^[A-F0-9]{5,}$}) {
                        ++$c;
                } elsif ($f =~ m{^[A-F0-9]$}) {
                        $c += count("$dir/$f");
                }
        }
        closedir($dh) or die "closedir: $dir: $!\n";
        return $c;
}
my $qdir = `postconf -h queue_directory`;
chomp($qdir);
chdir($qdir) or die "$0: chdir: $qdir: $!\n";
printf "Incoming: %d\n", count("incoming");
printf "Active: %d\n", count("active");
printf "Deferred: %d\n", count("deferred");
printf "Bounced: %d\n", count("bounce");
printf "Hold: %d\n", count("hold");
printf "Corrupt: %d\n", count("corrupt");

Various commands

[Wayback] Inspecting Postfix’s email queue – Tech-G explaining about:

  • mailq
  • postqueue -p
  • postcat -vq XXXXXXXXXX (where XXXXXXXXXX is the message ID)
  • postqueue -f / postfix flush
  • postsuper -d to delete messages

More of these in [Wayback] Postfix Mail Queue Management – Linux Hint and [Wayback] Postfix Bottleneck Analysis: queues.

Makefile

Based on [Wayback] Using “make” for Postfix file maintenance

MAPS = relays.db aliases.db transport.db relocated.db \
        virtual.db sender_checks.db rejected_recips.db \
        helo_access.db

all : $(MAPS)

aliases.db : aliases
        newaliases

%.db : %
        postmap $*

This is my Makefile that runs fine on Tumbleweed (note: all 8-space indents are TAB characters):

MAPS =  /etc/aliases.db \
        transport.db \
        virtual.db \
        helo_access.db \
        canonical.db \
        sasl_passwd.db \
        relocated.db \
        relay.db \
        access.db \
        relay_ccerts.db \
        sender_canonical.db

all : $(MAPS)

aliases.db : aliases
        @echo "Rebuilding $@."
        newaliases

%.db : %
        @echo "Rebuilding $@."
        postmap $*

In the future, I might try [Wayback] Makefile.postfix · GitHub, though I think it is convoluted:


## Postfix: Makefile to update *.db files
POSTCONF= /usr/sbin/postconf
POSTMAP= /usr/sbin/postmap
default: postmap
postmap: Makefile.postmap
@echo 'Updating database files …'
$(MAKE) -f Makefile.postmap
Makefile.postmap: main.cf
@echo 'Updating $@ …'
@set -e; \
rm -f $@.$$$$.tmp; \
echo 'POSTMAP=$(POSTMAP)' >>$@.$$$$.tmp; \
echo 'postmap::' >>$@.$$$$.tmp; \
config_directory="$(PWD)"; \
{ $(POSTCONF) -c $(PWD) || kill $$$$; } \
|tr ' ' '\n' \
|sed -n \
-e 's/,$$//' \
-e 's#^hash:\$$config_directory/##p' \
-e 's#^hash:'"$$config_directory/##p" \
|sort -u \
|while read mapfile; do \
echo "postmap:: $$mapfile.db" >>$@.$$$$.tmp; \
echo "$$mapfile.db: $$mapfile" >>$@.$$$$.tmp; \
echo " \$$(POSTMAP) $$<" >>$@.$$$$.tmp; \
done; \
mv $@.$$$$.tmp $@

 

 

[Wayback] Ralf Hildebrandt

Ralf Hildebrandt is an active and well-known figure in the Postfix community. He’s a systems engineer for T-NetPro, a German telecommunications company and has spoken about Postfix at industry conferences and contributes regularly to a number of open source mailing lists.

Co-author of this book: [Wayback: Book of Postfix State-of-the-Art Message Transport ISBN 9781593270018] (which used to have its own site: [Wayback: The Book of Postfix]

Book of Postfix

State-of-the-Art Message Transport

By Patrick KoetterRalf Hildebrandt

Publisher: No Starch PressRelease Date: March 2005Pages: 496

Best practices for Postfix–the popular alternative to Sendmail. Developed with security and speed in mind, Postfix has become a popular alternative to Sendmail and comes preinstalled in many Linux distributions as the default mailer. The Book of Postfix is a complete guide to Postfix whether used at home, as a mailrelay or virus-scanning gateway, or as a company mailserver. Practical examples show how to deal with daily challenges like protecting mail users from spam and viruses, managing multiple domains, and offering roaming access.

This is a great review of the book: [Wayback] The Book of Postfix (Ralf Hildebrandt, Patrick Koetter)

Related

For my postfix studies… « The Wiert Corner – irregular stream of stuff

–jeroen

 

Posted in *nix, *nix-tools, bash, Communications Development, Development, Internet protocol suite, Makefile, postfix, Power User, Scripting, SMTP, Software Development | Leave a Comment »

checkbashisms(1) – Linux man page

Posted by jpluimers on 2021/11/17

Even with lots of experience, one learns new things every day.

A while ago, I discovered checkbashisms which checks sh shel scripts (usually with extension .sh) scripts to they do not contain code specific to bash.

[Wayback] checkbashisms(1) – Linux man page

checkbashisms, based on one of the checks from the lintian system, performs basic checks on /bin/sh shell scripts for the possible presence of bashisms. It takes the names of the shell scripts on the command line, and outputs warnings if possible bashisms are detected.

Note that the definition of a bashism in this context roughly equates to “a shell feature that is not required to be supported by POSIX”; this means that some issues flagged may be permitted under optional sections of POSIX, such as XSI or User Portability.

In cases where POSIX and Debian Policy disagree, checkbashisms by default allows extensions permitted by Policy but may also provide options for stricter checking.

The source by now is a Perl script (it used to be a bash script) of which you can find the latest version here: [Wayback] scripts/checkbashisms.pl · master · Debian / devscripts · GitLab

Not installed by default

Virtually no distribution has checkbashisms installed by default.

In fact, the package containing checkbashisms heavily varies by distribution.

For OpenSuSE, it is in a package by itself: [Wayback] openSUSE Software: package checkbashisms

checkbashisms

Tool for Checking /bin/sh Scripts for Possible Bashisms

checkbashisms performs basic checks on /bin/sh shell scripts for the possible presence of bashisms. It takes the names of the shell scripts on the command line, and outputs warnings if possible bashisms are detected.

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sh, Sh Shell, Software Development | Leave a Comment »

Some bash parameter propagation links that hopefully will work with ash/dash too

Posted by jpluimers on 2021/10/27

For my link archive; I started with [Wayback] dash get all parameters quoted – Google Search:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Some links on xargs simulation in PowerShell

Posted by jpluimers on 2021/10/13

On nx, I’m used to xargs which allows to convert from a pipe of output into arguments passed to a command. This is useful, as many commands only accept arguments as parameters.

In PowerShell, you can usually avoid an xargs equivalent because commandlet output is a stream of objects that you can post-process using . I for instance used that in PowerShell: recovering from corrupt empty *.nupkg files after a disk was accidentally full during update.

Here are some xargs equivalency examples:

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, CommandLine, Development, Power User, PowerShell, PowerShell, Scripting, Software Development, xargs | Leave a Comment »

Google Cloud Shell: connecting to cloud storage buckets

Posted by jpluimers on 2021/09/28

One of the drawbacks of Google Cloud Shell is that it will delete the home directory after 120 days of not using it: [WayBack] Deletion notice for my Google Cloud Shell home directory – Stack Overflow

This is documented [WayBack] here:

If you do not access Cloud Shell for 120 days, we will delete your home disk. You will receive an email notification before we do so and simply starting a session will prevent its removal.

This only applies to the home directory of your Cloud Shell instance (you may want to store it on Cloud Storage anyway if you want to keep it). Any other Google services you use will be unaffected.

I hardly use the cloud shell, as it is a last resort to shell out from overly protected networks. Fewer and fewer environments restrict so much, so I’ve bumped into the home directory deletion a few times now.

I might use it more in the future, as I recently discovered there is a URL trick so you can start a cloud shell with parameters like an initial git repository: [WayBack] Open in Cloud Shell  |  Google Cloud

The Open in Cloud Shell feature allows you to publish a link that opens the Cloud Console and either automatically clones a Git repository into Cloud Shell or starts Cloud Shell with a custom image. It also allows for instructions to be printed to the terminal to help users interact with the content.

The Open in Cloud Shell feature helps developers experiment with code samples and APIs without having to worry about downloading Cloud SDK, installing required dependencies, or searching for relevant source files. This page explains how to add this feature to your Git repository.

Currently, only GitHub and Bitbucket repositories are whitelisted. If you would like to add a different repository, send feedback with the repository type you’d like to use with Open in Cloud Shell.

Setting up the home directory with my scripts can be a curse, so I have contemplated on these kinds of solutions:

  • store scripts in Google Drive, and mount part of Google Drive into the Cloud Shell
  • store scripts in Google Cloud Storage
  • script the setup of the home directory via a bash script in a gist

Some links that will hopefully help me with that:

  • [WayBack] Use the Google Cloud Shell to Upload files to Google Drive : DataHoarder
  • [WayBack] Open Your Repository In Google Cloud Shell – Daisuke Maki – Medium
  • [WayBack] gsutil – Google cloud storage – Download file from web – Stack Overflow

    mount the bucket in your Cloud Shell using gcsfuse:

    Create a directory in your Cloud Shell user home

     mkdir ~/mybucket
    

    Now mount your bucket in that directory using gcsfuse:

     gcsfuse bucket_name ~/mybucket
    

    Change the current directory to mount point directory:

     cd mybucket
    

    (if you want to have some fun run “df -h .” to see how much space you got in that mount point)

    seems to work for all file sizes:

     curl http://speedtest.tele2.net/10GB.zip | gsutil cp - gs://YOUR_BUCKET_NAME/10GB.zip
    

    Basically curl “streams” the data directly to the bucket.

  • [WayBack] Limitations and restrictions  |  Cloud Shell  |  Google Cloud

    Usage limits

    Cloud Shell inactivity: If you do not access Cloud Shell for 120 days, your home disk will be deleted. You will receive an email notification before its deletion and simply starting a session will prevent its removal. Please consider a different solution on Google Cloud storage for sensitive data you wish to store long term.

    Non-interactive usage: Cloud Shell is intended for interactive use only. Non-interactive sessions will be ended automatically after a warning. Note that Cloud Shell sessions are capped at 12 hours, after which sessions are automatically terminated. You can use a new session immediately after.

    Weekly usage: Cloud Shell also has weekly usage limits. If you reach your usage limit, you’ll need to wait until the specified time (listed under Usage Quota, found under the three dots menu icon) before you can use Cloud Shell again.

    Restoring a session after a service limit violation: If your session is terminated or cannot be established because you exceeded a service limit, Cloud Shell will display an error with a link to a form that allows you to appeal the limit violation. Click the feedback link and submit the form with more information about the tasks you were performing before your session was terminated.

–jeroen

Posted in bash, Development, Google, GoogleCloudShell, GoogleDrive, Internet, Power User, Scripting, Software Development, SpeedTest | Leave a Comment »