The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,572 other followers

Archive for the ‘bash’ Category

Getting the primary IP address (plain and CIDR) on Linux and OS X, then nmap scan on the associated subnet

Posted by jpluimers on 2021/12/13

The below answer works on my Linux and OS X systems (each having multiple network adapters configured):

[WayBack] bash – How to get the primary IP address of the local machine on Linux and OS X? – Stack Overflow

ip route get 1 | awk '{print $NF;exit}'

For Linux, I have this bash function:

# note the ";exit" lists the first default route interface, as there can be multiple
function nmap-fingerprint_network_of_default_interface() {
        default_if=$(ip route list | awk '/^default/ {print $5;exit}')
        default_if_cidr=$(ip -o -f inet addr show $default_if | awk '{print $4}')
        nmap -sP $default_if_cidr
}

And for OS X this one:

# requires ipcalc
function nmap-fingerprint_network_of_default_interface() {
        default_if=$(route -q -n get default | awk '/interface:/ {print $2;exit}')
        default_if_address=$(ifconfig $default_if | awk '/inet / {print $2;exit}')
        default_if_netmask_hex=$(ifconfig $default_if | awk '/inet / {print $4;exit}')
        default_if_network_bit_count=$(ipcalc --nocolor --class $default_if_address $default_if_netmask_hex)
        default_if_cidr=$(echo "$default_if_address/$default_if_network_bit_count")
        nmap -sP $default_if_cidr
}

These are the variables used:

  • default_if: network interface of the default route
  • default_if_cidr: IPv4 CIDR of the network interface of the default route (see Classless Inter-Domain Routing: CIDR notation – Wikipedia)
  • default_if_address: IPv4 address of network interface of the default route
  • default_if_netmask_hex: hexadecimal IPv4 network mask of network interface of the default route
  • default_if_network_bit_count: number of set bits in the IPv4 network mask of the network interface of the default route

Links used to get the above functions:

I might have gotten away with a pure bash solution (see [WayBack] Bash script for calculating network and broadcast addresses from ip and netmask or CIDR Notation · GitHub or my post Getting your local IPv4 addresses, netmasks and CIDRs), but the above works and is way shorter, and easier to maintain.

In stead of ipcalc, subnetcalc can do the same calculations and also supports IPv6, so that is something for a future try:

–jeroen

Posted in *nix, *nix-tools, Apple, bash, Color (software development), Development, Mac, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »

Some scripts and tips for easing the maintenance of a postfix based SMTP system

Posted by jpluimers on 2021/11/24

A few scripts and tips I found Googling around.

Deleting queued messages by regular expression pattern

I have seen the below delete-from-mailq.pl script numerous time, usually without any attribution (for instance [Wayback] Postfix Flush the Mail Queue – nixCraft and  [Wayback] postfix-delete.pl – Following script deletes all mail from the mailq which matches the regular expression specified as the first argument · GitHub).

The earliest version I could find was in [Wayback] ‘Re: delete messages from mailq’ – MARC by [Wayback] ‘Ralf Hildebrandt ‘ posts – MARC:

--- snip ---
#!/usr/bin/perl

$REGEXP = shift || die "no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!";

@data = qx</usr/sbin/postqueue -p>;
for (@data) {
  if (/^(\w+)\*?\s/) {
     $queue_id = $1;
  }
  if($queue_id) {
    if (/$REGEXP/i) {
      $Q{$queue_id} = 1;
      $queue_id = "";
    }
  }
}
                                
#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;

foreach (keys %Q) {
  print POSTSUPER "$_\n";
};
close(POSTSUPER);
--- snip ---

And then use:
% delete-from-mailq "^test"

 

Tips

[Wayback] How do I check the postfix queue size? – Server Fault

Lots of great answers and pointers to useful guides/software there.

qstat

[Wayback] Postfix Bottleneck Analysis points to [Wayback] Postfix manual – qshape(1): qshape - Print Postfix queue domain and age distribution, then explains about different scenarion and queues:

postqueue

postqueue -p | tail -n 1

Last line in the postqueue -p shows how many requests and size:

-- 317788 Kbytes in 11860 Requests.

View queues size

I tried finding the original posting of the below script, but could not. If you find it, please let me know.

#!/usr/bin/env perl

# postfix queue/s size
# author: 
# source: http://tech.groups.yahoo.com/group/postfix-users/message/255133

use strict;
use warnings;
use Symbol;
sub count {
        my ($dir) = @_;
        my $dh = gensym();
        my $c = 0;
        opendir($dh, $dir) or die "$0: opendir: $dir: $!\n";
        while (my $f = readdir($dh)) {
                if ($f =~ m{^[A-F0-9]{5,}$}) {
                        ++$c;
                } elsif ($f =~ m{^[A-F0-9]$}) {
                        $c += count("$dir/$f");
                }
        }
        closedir($dh) or die "closedir: $dir: $!\n";
        return $c;
}
my $qdir = `postconf -h queue_directory`;
chomp($qdir);
chdir($qdir) or die "$0: chdir: $qdir: $!\n";
printf "Incoming: %d\n", count("incoming");
printf "Active: %d\n", count("active");
printf "Deferred: %d\n", count("deferred");
printf "Bounced: %d\n", count("bounce");
printf "Hold: %d\n", count("hold");
printf "Corrupt: %d\n", count("corrupt");

Various commands

[Wayback] Inspecting Postfix’s email queue – Tech-G explaining about:

  • mailq
  • postqueue -p
  • postcat -vq XXXXXXXXXX (where XXXXXXXXXX is the message ID)
  • postqueue -f / postfix flush
  • postsuper -d to delete messages

More of these in [Wayback] Postfix Mail Queue Management – Linux Hint and [Wayback] Postfix Bottleneck Analysis: queues.

Makefile

Based on [Wayback] Using “make” for Postfix file maintenance

MAPS = relays.db aliases.db transport.db relocated.db \
        virtual.db sender_checks.db rejected_recips.db \
        helo_access.db

all : $(MAPS)

aliases.db : aliases
        newaliases

%.db : %
        postmap $*

This is my Makefile that runs fine on Tumbleweed (note: all 8-space indents are TAB characters):

MAPS =  /etc/aliases.db \
        transport.db \
        virtual.db \
        helo_access.db \
        canonical.db \
        sasl_passwd.db \
        relocated.db \
        relay.db \
        access.db \
        relay_ccerts.db \
        sender_canonical.db

all : $(MAPS)

aliases.db : aliases
        @echo "Rebuilding $@."
        newaliases

%.db : %
        @echo "Rebuilding $@."
        postmap $*

In the future, I might try [Wayback] Makefile.postfix · GitHub, though I think it is convoluted:


## Postfix: Makefile to update *.db files
POSTCONF= /usr/sbin/postconf
POSTMAP= /usr/sbin/postmap
default: postmap
postmap: Makefile.postmap
@echo 'Updating database files …'
$(MAKE) -f Makefile.postmap
Makefile.postmap: main.cf
@echo 'Updating $@ …'
@set -e; \
rm -f $@.$$$$.tmp; \
echo 'POSTMAP=$(POSTMAP)' >>$@.$$$$.tmp; \
echo 'postmap::' >>$@.$$$$.tmp; \
config_directory="$(PWD)"; \
{ $(POSTCONF) -c $(PWD) || kill $$$$; } \
|tr ' ' '\n' \
|sed -n \
-e 's/,$$//' \
-e 's#^hash:\$$config_directory/##p' \
-e 's#^hash:'"$$config_directory/##p" \
|sort -u \
|while read mapfile; do \
echo "postmap:: $$mapfile.db" >>$@.$$$$.tmp; \
echo "$$mapfile.db: $$mapfile" >>$@.$$$$.tmp; \
echo " \$$(POSTMAP) $$<" >>$@.$$$$.tmp; \
done; \
mv $@.$$$$.tmp $@

 

 

[Wayback] Ralf Hildebrandt

Ralf Hildebrandt is an active and well-known figure in the Postfix community. He’s a systems engineer for T-NetPro, a German telecommunications company and has spoken about Postfix at industry conferences and contributes regularly to a number of open source mailing lists.

Co-author of this book: [Wayback: Book of Postfix State-of-the-Art Message Transport ISBN 9781593270018] (which used to have its own site: [Wayback: The Book of Postfix]

Book of Postfix

State-of-the-Art Message Transport

By Patrick KoetterRalf Hildebrandt

Publisher: No Starch PressRelease Date: March 2005Pages: 496

Best practices for Postfix–the popular alternative to Sendmail. Developed with security and speed in mind, Postfix has become a popular alternative to Sendmail and comes preinstalled in many Linux distributions as the default mailer. The Book of Postfix is a complete guide to Postfix whether used at home, as a mailrelay or virus-scanning gateway, or as a company mailserver. Practical examples show how to deal with daily challenges like protecting mail users from spam and viruses, managing multiple domains, and offering roaming access.

This is a great review of the book: [Wayback] The Book of Postfix (Ralf Hildebrandt, Patrick Koetter)

Related

For my postfix studies… « The Wiert Corner – irregular stream of stuff

–jeroen

 

Posted in *nix, *nix-tools, bash, Communications Development, Development, Internet protocol suite, Makefile, postfix, Power User, Scripting, SMTP, Software Development | Leave a Comment »

checkbashisms(1) – Linux man page

Posted by jpluimers on 2021/11/17

Even with lots of experience, one learns new things every day.

A while ago, I discovered checkbashisms which checks sh shel scripts (usually with extension .sh) scripts to they do not contain code specific to bash.

[Wayback] checkbashisms(1) – Linux man page

checkbashisms, based on one of the checks from the lintian system, performs basic checks on /bin/sh shell scripts for the possible presence of bashisms. It takes the names of the shell scripts on the command line, and outputs warnings if possible bashisms are detected.

Note that the definition of a bashism in this context roughly equates to “a shell feature that is not required to be supported by POSIX”; this means that some issues flagged may be permitted under optional sections of POSIX, such as XSI or User Portability.

In cases where POSIX and Debian Policy disagree, checkbashisms by default allows extensions permitted by Policy but may also provide options for stricter checking.

The source by now is a Perl script (it used to be a bash script) of which you can find the latest version here: [Wayback] scripts/checkbashisms.pl · master · Debian / devscripts · GitLab

Not installed by default

Virtually no distribution has checkbashisms installed by default.

In fact, the package containing checkbashisms heavily varies by distribution.

For OpenSuSE, it is in a package by itself: [Wayback] openSUSE Software: package checkbashisms

checkbashisms

Tool for Checking /bin/sh Scripts for Possible Bashisms

checkbashisms performs basic checks on /bin/sh shell scripts for the possible presence of bashisms. It takes the names of the shell scripts on the command line, and outputs warnings if possible bashisms are detected.

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sh, Sh Shell, Software Development | Leave a Comment »

Some bash parameter propagation links that hopefully will work with ash/dash too

Posted by jpluimers on 2021/10/27

For my link archive; I started with [Wayback] dash get all parameters quoted – Google Search:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Some links on xargs simulation in PowerShell

Posted by jpluimers on 2021/10/13

On nx, I’m used to xargs which allows to convert from a pipe of output into arguments passed to a command. This is useful, as many commands only accept arguments as parameters.

In PowerShell, you can usually avoid an xargs equivalent because commandlet output is a stream of objects that you can post-process using . I for instance used that in PowerShell: recovering from corrupt empty *.nupkg files after a disk was accidentally full during update.

Here are some xargs equivalency examples:

  • [WayBack] PowerShell tips for bash users, part 1 • Five

    xargs

    Xargs is one of the most powerfull UNIX commands. It is used to build and execute command lines from standard input. For example:
    $ cat dirs | xargs mkdir
    will use cat to take the strings (be it newline or blank character separated) from file ‘dirs’ and pass them through pipe to xargs which will then send one by one line as argument to mkdir which will then create those dirs or complain if those are existent.

    PowerShell equivalent:

    PS> cat dirs | %{mkdir $_}

    There is no ‘xargs’ command in PS, but you can use ‘foreach ‘ loop and pass the piped variable ‘$_’ to the mkdir. Shorthand for ‘foreach’ is ‘%’. This time also only newlines will separate the strings apart. If multiple strings separated by blanks are found in same line, mkdir will create a directory with blanks in the name, while we must quote to have the same in bash:

    $ cat dirs | sed 's|^|"|g' | sed 's|$|"|g' |xargs mkdir

  • [WayBack] What’s the equivalent of xargs in PowerShell? – Stack Overflow

    Q

    The POSIX-defined xargs command takes all of the items it receives from standard input and passes them as command-line arguments to the command it receives on it’s own command line. E.g: grep -rn "String" | xargs rm.

    What’s the equivalent in PowerShell?

    The following questions all ask this:

    but there is no correct answer because all the answers either use ForEach-Object, which will process items one-at-a-time (like xargs -n1) which gets the desired result for the examples given, or store the intermediate result in a variable, which offends my functional commandline-fu.

    A

    There are two ways that I’ve found. The first is probably more idiomatic PowerShell, and the second is more true to the pipe-based spirit of xargs.

    As an example, let’s say we want to pass all our cat pics to myapp.exe.

    Method #1: Command substitution

    You can do something similar to using $(command substitution) in sh by embedding your pipeline in the command string:

    &"myapp.exe" @(Get-ChildItem -Recurse -Filter *.jpg | Another-Step)

    The @(...) creates an array from the command inside it, and PowerShell automatically expands arrays passed to & into seperate command-line parameters.

    However, this does not really answer the question, because it will only work if you have control over the command you want to pass to, which may not be the case.

    Method #2: True piping

    You can also construct a “double pipeline” by having a sub-expression to pipe your objects, collecting them to an array, and then piping the array to your final command.

    ,@(Get-ChildItem -Recurse -Filter *.jpg | Another-Step) | %{&"myapp.exe" $_}

    The @(...) as before collects the items into an array, and the array is then piped to the final command which is invoked using % (ForEach-Object). Ordinarily, this would then loop over each item individually, because PowerShell will automatically flatten the array when it’s piped, but this can be avoided by prepending the , operator. The $_ special variable is then used as normal to embed the passed array.

    So the key is to wrap the pipeline you want to collect in ,@(...), and then pipe that to something in %{...}.

References

–jeroen

Posted in *nix, *nix-tools, bash, CommandLine, Development, Power User, PowerShell, PowerShell, Scripting, Software Development, xargs | Leave a Comment »

 
%d bloggers like this: