The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,861 other subscribers

Archive for the ‘Scripting’ Category

Some scripts and tips for easing the maintenance of a postfix based SMTP system

Posted by jpluimers on 2021/11/24

A few scripts and tips I found Googling around.

Deleting queued messages by regular expression pattern

I have seen the below delete-from-mailq.pl script numerous time, usually without any attribution (for instance [Wayback] Postfix Flush the Mail Queue – nixCraft and  [Wayback] postfix-delete.pl – Following script deletes all mail from the mailq which matches the regular expression specified as the first argument · GitHub).

The earliest version I could find was in [Wayback] ‘Re: delete messages from mailq’ – MARC by [Wayback] ‘Ralf Hildebrandt ‘ posts – MARC:

--- snip ---
#!/usr/bin/perl

$REGEXP = shift || die "no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!";

@data = qx</usr/sbin/postqueue -p>;
for (@data) {
  if (/^(\w+)\*?\s/) {
     $queue_id = $1;
  }
  if($queue_id) {
    if (/$REGEXP/i) {
      $Q{$queue_id} = 1;
      $queue_id = "";
    }
  }
}
                                
#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;

foreach (keys %Q) {
  print POSTSUPER "$_\n";
};
close(POSTSUPER);
--- snip ---

And then use:
% delete-from-mailq "^test"

 

Tips

[Wayback] How do I check the postfix queue size? – Server Fault

Lots of great answers and pointers to useful guides/software there.

qstat

[Wayback] Postfix Bottleneck Analysis points to [Wayback] Postfix manual – qshape(1): qshape - Print Postfix queue domain and age distribution, then explains about different scenarion and queues:

postqueue

postqueue -p | tail -n 1

Last line in the postqueue -p shows how many requests and size:

-- 317788 Kbytes in 11860 Requests.

View queues size

I tried finding the original posting of the below script, but could not. If you find it, please let me know.

#!/usr/bin/env perl

# postfix queue/s size
# author: 
# source: http://tech.groups.yahoo.com/group/postfix-users/message/255133

use strict;
use warnings;
use Symbol;
sub count {
        my ($dir) = @_;
        my $dh = gensym();
        my $c = 0;
        opendir($dh, $dir) or die "$0: opendir: $dir: $!\n";
        while (my $f = readdir($dh)) {
                if ($f =~ m{^[A-F0-9]{5,}$}) {
                        ++$c;
                } elsif ($f =~ m{^[A-F0-9]$}) {
                        $c += count("$dir/$f");
                }
        }
        closedir($dh) or die "closedir: $dir: $!\n";
        return $c;
}
my $qdir = `postconf -h queue_directory`;
chomp($qdir);
chdir($qdir) or die "$0: chdir: $qdir: $!\n";
printf "Incoming: %d\n", count("incoming");
printf "Active: %d\n", count("active");
printf "Deferred: %d\n", count("deferred");
printf "Bounced: %d\n", count("bounce");
printf "Hold: %d\n", count("hold");
printf "Corrupt: %d\n", count("corrupt");

Various commands

[Wayback] Inspecting Postfix’s email queue – Tech-G explaining about:

  • mailq
  • postqueue -p
  • postcat -vq XXXXXXXXXX (where XXXXXXXXXX is the message ID)
  • postqueue -f / postfix flush
  • postsuper -d to delete messages

More of these in [Wayback] Postfix Mail Queue Management – Linux Hint and [Wayback] Postfix Bottleneck Analysis: queues.

Makefile

Based on [Wayback] Using “make” for Postfix file maintenance

MAPS = relays.db aliases.db transport.db relocated.db \
        virtual.db sender_checks.db rejected_recips.db \
        helo_access.db

all : $(MAPS)

aliases.db : aliases
        newaliases

%.db : %
        postmap $*

This is my Makefile that runs fine on Tumbleweed (note: all 8-space indents are TAB characters):

MAPS =  /etc/aliases.db \
        transport.db \
        virtual.db \
        helo_access.db \
        canonical.db \
        sasl_passwd.db \
        relocated.db \
        relay.db \
        access.db \
        relay_ccerts.db \
        sender_canonical.db

all : $(MAPS)

aliases.db : aliases
        @echo "Rebuilding $@."
        newaliases

%.db : %
        @echo "Rebuilding $@."
        postmap $*

In the future, I might try [Wayback] Makefile.postfix · GitHub, though I think it is convoluted:


## Postfix: Makefile to update *.db files
POSTCONF= /usr/sbin/postconf
POSTMAP= /usr/sbin/postmap
default: postmap
postmap: Makefile.postmap
@echo 'Updating database files …'
$(MAKE) -f Makefile.postmap
Makefile.postmap: main.cf
@echo 'Updating $@ …'
@set -e; \
rm -f $@.$$$$.tmp; \
echo 'POSTMAP=$(POSTMAP)' >>$@.$$$$.tmp; \
echo 'postmap::' >>$@.$$$$.tmp; \
config_directory="$(PWD)"; \
{ $(POSTCONF) -c $(PWD) || kill $$$$; } \
|tr ' ' '\n' \
|sed -n \
-e 's/,$$//' \
-e 's#^hash:\$$config_directory/##p' \
-e 's#^hash:'"$$config_directory/##p" \
|sort -u \
|while read mapfile; do \
echo "postmap:: $$mapfile.db" >>$@.$$$$.tmp; \
echo "$$mapfile.db: $$mapfile" >>$@.$$$$.tmp; \
echo " \$$(POSTMAP) $$<" >>$@.$$$$.tmp; \
done; \
mv $@.$$$$.tmp $@

 

 

[Wayback] Ralf Hildebrandt

Ralf Hildebrandt is an active and well-known figure in the Postfix community. He’s a systems engineer for T-NetPro, a German telecommunications company and has spoken about Postfix at industry conferences and contributes regularly to a number of open source mailing lists.

Co-author of this book: [Wayback: Book of Postfix State-of-the-Art Message Transport ISBN 9781593270018] (which used to have its own site: [Wayback: The Book of Postfix]

Book of Postfix

State-of-the-Art Message Transport

By Patrick KoetterRalf Hildebrandt

Publisher: No Starch PressRelease Date: March 2005Pages: 496

Best practices for Postfix–the popular alternative to Sendmail. Developed with security and speed in mind, Postfix has become a popular alternative to Sendmail and comes preinstalled in many Linux distributions as the default mailer. The Book of Postfix is a complete guide to Postfix whether used at home, as a mailrelay or virus-scanning gateway, or as a company mailserver. Practical examples show how to deal with daily challenges like protecting mail users from spam and viruses, managing multiple domains, and offering roaming access.

This is a great review of the book: [Wayback] The Book of Postfix (Ralf Hildebrandt, Patrick Koetter)

Related

For my postfix studies… « The Wiert Corner – irregular stream of stuff

–jeroen

 

Posted in *nix, *nix-tools, bash, Communications Development, Development, Internet protocol suite, Makefile, postfix, Power User, Scripting, SMTP, Software Development | Leave a Comment »

Random User Generator | Home

Posted by jpluimers on 2021/11/23

Cool tool for when you ever need random users to test a system [Wayback] Random User Generator | Home:

Random user generator is a FREE API for generating placeholder user information. Get profile photos, names, and more. It’s like Lorem Ipsum, for people.

This was used when extracting Parler data to substantiate evidence around the 20210106 USA Capitol riots.

You can even use a simple HTTP GET like [Wayback] randomuser.me/api and get a JSON result like this.

{"results":[{"gender":"female","name":{"title":"Miss","first":"Malou","last":"Mortensen"},"location":{"street":{"number":2669,"name":"Lyngbyvej"},"city":"Sundby","state":"Syddanmark","country":"Denmark","postcode":48047,"coordinates":{"latitude":"-35.1307","longitude":"113.7480"},"timezone":{"offset":"+1:00","description":"Brussels, Copenhagen, Madrid, Paris"}},"email":"malou.mortensen@example.com","login":{"uuid":"981747de-66fe-40b0-87ea-adfe403fe1be","username":"purpleostrich871","password":"sweets","salt":"x86aQbIB","md5":"55497ac53530b428f98b9d36267ceeef","sha1":"358b94ffabe7d827c34da15791e5d6717c594428","sha256":"6e357e887877e29b7e6d53073f648174382c53c24f83479e25fed9c82075ed32"},"dob":{"date":"1995-06-05T04:50:35.145Z","age":26},"registered":{"date":"2018-07-21T00:59:50.523Z","age":3},"phone":"02990797","cell":"94800012","id":{"name":"CPR","value":"050695-9954"},"picture":{"large":"https://randomuser.me/api/portraits/women/27.jpg","medium":"https://randomuser.me/api/portraits/med/women/27.jpg","thumbnail":"https://randomuser.me/api/portraits/thumb/women/27.jpg"},"nat":"DK"}],"info":{"seed":"8971869bb62b73d7","results":1,"page":1,"version":"1.3"}}

Via:

–jeroen

Read the rest of this entry »

Posted in Communications Development, Development, HTTP, Internet protocol suite, JavaScript/ECMAScript, JSON, Python, REST, Scripting, Software Development, TCP | Leave a Comment »

Tool for debugging makefiles – Stack Overflow

Posted by jpluimers on 2021/11/23

I’m not good at makefiles, so I needed a tool to help me debug some relatively simple ones.

[Wayback] Tool for debugging makefiles – Stack Overflow came to the rescue.

The suggestion in the first answer was enough: run make -n.

But the other suggestions are great too, so here I quote them (thanks [Wayback] Rob Wells and [Wayback] User Rajish):

Have you been looking at the output from running make -n and make -np, and the biggie make -nd?

Are you using a fairly recent version of gmake?

Have you looked at the free chapter on [Wayback] Debugging Makefiles available on O’Reilly’s site for their excellent book “Managing Projects with GNU Make” ([Wayback] Amazon Link).


I’m sure that [Waybackremake is what you are looking for.

From the homepage:

remake is a patched and modernized version of GNU make utility that adds improved error reporting, the ability to trace execution in a comprehensible way, and a debugger.

It has gdb-like interface and is supported by mdb-mode in (x)emacs which means breakponts, watches etc. And there’s [Wayback] DDD if you don’t like (x)emacs

The free chapter: [Wayback] https://www.oreilly.com/openbook/make3/book/ch12.pdf

Remake documentation: [Wayback] remake – GNU Make with comprehensible tracing, profiling, extended error messages, and a debugger — remake 4.3+dbg-1.4 documentation

Tutorials

[Wayback] Makefile Tutorial by Example

–jeroen

Posted in Development, Makefile, Scripting, Software Development | Leave a Comment »

GitHub’s commitment to npm ecosystem security | The GitHub Blog – no npm package can historically ben tracked to be authentic

Posted by jpluimers on 2021/11/18

What started as [Wayback/Archive] GitHub’s commitment to npm ecosystem security | The GitHub Blog with this qoute:

on November 2 we received a report to our [Wayback/Archive] security bug bounty program of a vulnerability that would allow an attacker to publish new versions of any npm package using an account without proper authorization. …

This vulnerability existed in the npm registry beyond the timeframe for which we have telemetry to determine whether it has ever been exploited maliciously. … the timeframe for which we have available telemetry, which goes back to September 2020 …

This basically means you cannot trust the authenticity of any npm package published through the registry.

It sparked reactions like these:

–jeroen

Posted in Development, JavaScript/ECMAScript, Node.js, Scripting, Software Development | Leave a Comment »

checkbashisms(1) – Linux man page

Posted by jpluimers on 2021/11/17

Even with lots of experience, one learns new things every day.

A while ago, I discovered checkbashisms which checks sh shel scripts (usually with extension .sh) scripts to they do not contain code specific to bash.

[Wayback] checkbashisms(1) – Linux man page

checkbashisms, based on one of the checks from the lintian system, performs basic checks on /bin/sh shell scripts for the possible presence of bashisms. It takes the names of the shell scripts on the command line, and outputs warnings if possible bashisms are detected.

Note that the definition of a bashism in this context roughly equates to “a shell feature that is not required to be supported by POSIX”; this means that some issues flagged may be permitted under optional sections of POSIX, such as XSI or User Portability.

In cases where POSIX and Debian Policy disagree, checkbashisms by default allows extensions permitted by Policy but may also provide options for stricter checking.

The source by now is a Perl script (it used to be a bash script) of which you can find the latest version here: [Wayback] scripts/checkbashisms.pl · master · Debian / devscripts · GitLab

Not installed by default

Virtually no distribution has checkbashisms installed by default.

In fact, the package containing checkbashisms heavily varies by distribution.

For OpenSuSE, it is in a package by itself: [Wayback] openSUSE Software: package checkbashisms

checkbashisms

Tool for Checking /bin/sh Scripts for Possible Bashisms

checkbashisms performs basic checks on /bin/sh shell scripts for the possible presence of bashisms. It takes the names of the shell scripts on the command line, and outputs warnings if possible bashisms are detected.

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sh, Sh Shell, Software Development | Leave a Comment »

How do I restart sshd on my Unix system | StarNet Knowledge Database – PC X, X Windows, X 11 & More – StarNet

Posted by jpluimers on 2021/11/17

[Wayback] How do I restart sshd on my Unix system | StarNet Knowledge Database – PC X, X Windows, X 11 & More – StarNet

RedHat and Fedora Core Linux

/sbin/service sshd restart

Suse linux

/etc/rc.d/sshd restart

Debian/Ubuntu

/etc/init.d/sshd restart

Solaris 9 and below

/etc/init.d/sshd stop
/etc/init.d/sshd start

Solaris 10

svcadm disable ssh
svcadm enable ssh

AIX

stopsrc -s sshd
startsrc -s sshd

HP-UX

/sbin/init.d/secsh stop
/sbin/init.d/secsh start

Note that for opensuse, by now you need this to restart sshd:

/usr/sbin/rcsshd restart

Edit 20211118: some tweets in reaction to this post

–jeroen

Posted in *nix, *nix-tools, Debian, Development, Linux, openSuSE, Power User, RedHat, Scripting, Software Development, SuSE Linux, systemd, Tumbleweed | Leave a Comment »

unix/linux: using paste to turn separate lines into a comma separated list

Posted by jpluimers on 2021/11/16

Never to old to learn new things: I was totally unaware of the GNU paste tool that is available on virtually all unix/Linux/BSD core installs.

Thanks [WayBack] zeppelin for answering this question at [WayBack] linux – Turning separate lines into a comma separated list with quoted entries – Unix & Linux Stack Exchange:

You can add quotes with sed and then merge lines with paste, like that:

sed 's/^\|$/"/g'|paste -sd, -

If you are running a GNU coreutils based system (i.e. Linux), you can omit the trailing '-'.

If you input data has DOS-style line endings (as @phk suggested), you can modify the command as follows:

sed 's/\r//;s/^\|$/"/g'|paste -sd, -

Now I can get a comma separated list of for instance ssh available mac algorithms:

# ssh -Q mac | paste -sd, -
hmac-sha1,hmac-sha1-96,hmac-sha2-256,hmac-sha2-512,hmac-md5,hmac-md5-96,umac-64@openssh.com,umac-128@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-md5-etm@openssh.com,hmac-md5-96-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com

Documentation:

–jeroen

Posted in *nix, *nix-tools, Development, Power User, Scripting, Software Development | Leave a Comment »

Infusion pump and PCA (patient-controlled analgesia) calculation

Posted by jpluimers on 2021/11/09

This is a great PCA calculator: [Archive.is] CADD calculator 0.5BETA END USER Google Docs – Google Sheets

Via:

Related:

–jeroen

Read the rest of this entry »

Posted in Development, Excel, Google, GoogleDocs, GoogleSheets, Office, Office Automation, Office VBA, Power User, Scripting, Software Development | Leave a Comment »

Windows 10: remove applications from the uninstall list

Posted by jpluimers on 2021/11/04

After doing Windows upgrades to Windows 10, every now and then I bump into applications that do not fully uninstall themselves and get stuck on the uninstall list (that you get when running appwiz.cpl or browse to the Control Pannel installed programs list).

[WayBack] How to Manually Remove Programs from the Add/Remove Programs List mentions to inspect registry key HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall, but that didn’t include some of the applications.

Then I found [WayBack] Remove entry from Windows 10 Apps & Features – Super User, where the answers mentions two other keys (thanks users [WayBack] Kreiggott and [WayBack] NutCracker):

  • HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall
  • HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Uninstall

Neat!

So I made the below PowerShell script to dump installed programs.

It grabs the list of registry keys containing installed software and their registry values, then empirically filters out most values that are also now shown in AppWiz.cpl.

Like database work, the values can have properties having a value or being null. So it’s SQL like expression galore to do the filtering.

This post is slightly related to Still unsolved since 2015 NetBeans: Bug 251538 – Your Installer is Creating Invalid Data for the NoModify DWORD Key which crashes enumeration of the Uninstall Key in at least PowerShell, where I already did (without documenting) some Uninstall spelunking.

## The collection of registry keys gives Name and Property of each registry key; where Property is compound containing all registry values of that key.
## Get-ItemProperty will get you all the values on which you can filter, including a few special PS* values that allow you to browse back to the registry key.

# x86 installs on x64 hardware: http://stackoverflow.com/questions/12199372/get-itemproperty-not-returning-all-properties/12200100#12200100
$nonUninstallableSoftwareRegistryKeys = (@
(Get-Item HKCU:\Software\Microsoft\Windows\CurrentVersion\Uninstall\*)) + 
(Get-Item HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\*) + 
(Get-Item HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\*)
    
#$nonUninstallableSoftwareRegistryKeys.GetType().FullName
#$nonUninstallableSoftwareRegistryKeys | Get-Member
#$nonUninstallableSoftwareRegistryKeys | Out-GridView
#$nonUninstallableSoftwareRegistryKeys | Get-ItemProperty | Get-Member
#$nonUninstallableSoftwareRegistryKeys | Get-ItemProperty | Out-GridView
#Return
    
$nonUninstallableSoftwareRegistryNameValues = $nonUninstallableSoftwareRegistryKeys | 
    Get-ItemProperty |
    Where-Object {
        $_.SystemComponent -ne 1 -and $_.NoRemove -ne 1 -and
        $_.UninstallString -ne "" -and $_.UninstallString -ne $null
    }
# Filters out most things that AppWiz.cpl will leave out as well.
# Might need more fine tuning, but is good enough for now.

# PSPath shows the path to the underlying registry key of each value
$nonUninstallableSoftwareRegistryNameValues |
    Select-Object SystemComponent, NoRemove, DisplayName, DisplayVersion, UninstallString, PSChildName <#, PSPath #> |
    Sort-Object DisplayName |
    Out-GridView
# Need to find a good way to output this in a really wide Format-Table text format.

–jeroen

Posted in CommandLine, Development, Power User, PowerShell, PowerShell, Scripting, Software Development, Windows, Windows 10 | Leave a Comment »

Terminating a script in PowerShell – Stack Overflow

Posted by jpluimers on 2021/11/03

I have the same problem mentioned in the answer to [WayBack] Terminating a script in PowerShell – Stack Overflow: confused by most answers, and keeping to forget what each method means (there is Exit, Return, Break and (if you love exception handling to do simple flow control), Throw.

So here is the full quote of what [WayBack] User New Guy answered:

Read the rest of this entry »

Posted in *nix, CommandLine, Development, Power User, PowerShell, PowerShell, Scripting, Software Development, Windows | Leave a Comment »