The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,861 other subscribers

Archive for the ‘*nix-tools’ Category

« Previous Entries
Next Entries »

ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes

Posted by jpluimers on 2022/01/17

On VMware ESXi, with the  Busybox ash/dash shell, I wanted to get the current UTC timestamp in ISO 8601 format without dashes (-) or especially colons (:) and plus-signs (+) you have to back-slash escape colons or double quote parameters, which is often can be a pain).

This is why we can’t have good things: Getting the UTC 8610 timestamp was far less easy than I hoped for.

First of all, Busybox only allows for a precision of seconds, not milliseconds, and the specification format needs better documentation as per [Wayback] embedded linux – How to get ISO8601 seconds format from “date” in busybox? – Stack Overflow:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

On learning vi/vim

Posted by jpluimers on 2022/01/13

I saved below thread (if you cannot read German: try Google Translate on it [Archive.is]).

The part that – translated – strikes me most is this:

A mini-challenge where I want to spent 15 minutes per day on: when I need some functionality, instead of hacking around, research how it really is supposed to be done and internalise that way.

This totally aligns with me “learn new things every day” mantra.

Also some interesting vi/vim resources and approaches are mentioned:

If you are better at reading tables or lists for commands, I can recommend [Wayback] Basic vi Commands (Computer Science Department, Colorado State University).

One of the hardest things for me to remember somehow is the search/replace syntax, though it is very similar to that of sed. Useful examples of most permutations are at [Wayback] Vi For Smarties – Lesson Eight under “Advanced Search and Replace”:

  1. Use :s/foo/bar/ to replace the first occurrence of the word foo on the current line with the word bar.
  2. Use :s/foo/bar/g to replace all occurrences of the word foo on the current line with the word bar.
  3. Use :%s/foo/bar/g to replace all occurrences of the word foo in the current file with the word bar. Leaving off the g at the end only replaces the first occurrence of foo on each line of the current file.
  4. Use :%s/foo//g to delete all occurrences of the word foo in the current file. Leaving off the percent sign (%), of course, only does this for the current line.
  5. Use :%s/foo/bar/gc to have Vi query you before each attempt to replace the word foo with the word bar.

It is part of a series of tutorials starting at [Wayback] Vi For Smarties – Lesson One, and after the last lesson directs to the in-depth – as prose written – tutorial series [Archive.is] Wayback: The Vi/Ex Editor (which, despite marked by “[ Editor’s Note: . This tutorial series is a “work in progress” and will evolve over time. We’ll announce each new addition and expand this table of contents.]” got ditched shortly after January 2006)

https://web.archive.org/web/20160108053134/http://www.networkcomputin…

.

Note that like sed, you can use a different search/replace character than /: [Wayback] vim – / in vi Search and replace? – Stack Overflow

  • Escape the / with \ like this: \/.
  • Use different search character than /, for instance ., or #.

Back to the twitter thread

[Archive.is] Das königliche Wir on Twitter: “Ich lerne jetzt vi(m). Klingt beknackt, oder? Aber ich hab überlegt: ich möchte effizienter werden, meinen Kram stressfreier bewältigen. Und ich finde, es bietet sich an da an Baustellen anzusetzen, die täglich relevant sind. Und vi(m) nutze ich in der Tat täglich. 1/”

Ich lerne jetzt vi(m).

Klingt beknackt, oder? Aber ich hab überlegt: ich möchte effizienter werden, meinen Kram stressfreier bewältigen. Und ich finde, es bietet sich an da an Baustellen anzusetzen, die täglich relevant sind. Und vi(m) nutze ich in der Tat täglich.

Aber auch wenn ich ihn nutze und über den »Hilfe, wie komm ich aus dem Editor wieder raus?!«-Witz nur sehr müde die Augen rolle gehe ich davon aus, dass ich nicht mal einen Bruchteil der Möglichkeiten ausschöpfe, die er bietet (1. Release 1976, älter als ich!). Das ist spannend.

Das ist jetzt meine Mini-Challenge, auf die ich jeden Tag eine Viertelstunde verwenden will: wenn ich eine Funktion brauche mich nicht mehr drum herum zu hacken, sondern recherchieren wie es richtig geht und das dann gefälligst auch verinnerlichen. Mal sehen, ob das so klappt

In dem Zuge will ich auch wieder verstärkt (neo)mutt in Verbindung mit vim nutzen – das wäre ein wunderbares tägliches Training 😎 Mails schreiben muss man irgendwie immer.

neomutt bietet leider auch einen Eimer voll Funktionen, die ich noch nicht ordentlich nutze. Gnah.

An euren Replies erkenne ich, dass das mit dem »sich die Kürzel merken« echt heikel zu sein scheint 🤔 Wie handhabt ihr das im Alltag? Einfach ein paar Basics wie :u und CTRL+r und gut ist? Ich bin neugierig. Oder nutzt ihr alle nano? (Ich glaub, dann muss ich entfolgen) 😂

Nur als kleinen Zwischenstand: nach dem Lesen eurer Antworten schließe ich, dass ich mich mit meinem Kenntnisstand nicht verstecken muss 😂 Da hat mir der virtuelle Schulterblick schon weiter geholfen.

vi(m), weil ich Admin bin und dieses Tool auf jedem System und ohne X-Geraffels üblicherweise vorfinde (wenn auch bei neueren Installationen dieser absurde „visual mode“ der Default ist 😳). Auch unter (Open)Solaris, IRIX, was weiß ich.

Antworten:

Patrick Ben Koetter (@patrickbkoetter)

Ich habe diverse Plugins ausprobiert und lande immer wieder bei “normal“, weil ich einfach nicht die Disziplin aufbringe das Zeug richtig zu üben.

Marcel Weinberg (@winem_)

Denke so geht es vielen, die eben teilweise auch schon viele Jahre in der IT sind. Ist dir vimtutor bekannt? Den regelmäßig durchzutanzen hilft auch etwas, auch wenn man einiges immer mal vergisst oder einfach nicht nutzt.

Kristian Köhntopp (@isotopp)

vi movement Kommandos haben System. Erkenne und lerne das System. hjkl + prefix+hjkl, Marken, prefix+jump to mark und so weiter.

Und bleibe von den verblödeten Plugins weg. Die braucht kein Mensch und machen vim nur langsam im Start und kompliziert.

Boris (@k3rvyn)

Was mich so richtig weiter gebracht hat, war dieser Talk:
https://www.youtube.com/watch?v=5r6yzFEXajQ

winfreak (@winfreak)

Ich bilde mir kleine Merkphrasen für movements…
Aus „ciw“ wird „change inner word“, z.B.
Das kann ich mir deutlich besser merken.

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, Development, Power User, Scripting, Software Development, vi/vim | Leave a Comment »

A few tools on my list for research: fswatch, entr, ack and ag

Posted by jpluimers on 2022/01/12

Based on the below tweet and replies, these tools are not on my research list:

[Archive.is] Aaron Patterson on Twitter: “Here is my “autotest for lazy people”. It watches for changes in the lib or test directory, and runs rake test on change. There are definitely better, purpose built tools for this. But I am lazy…”

–jeroen

Posted in *nix, *nix-tools, Development, fgrep, find, grep, Power User, Scripting, Software Development | Leave a Comment »

Busybox ash/dash – Hexadecimal To Decimal in Shell Script (via Stack Overflow)

Posted by jpluimers on 2021/12/21

This works fine on “BusyBox v1.29.3 (2019-05-21 15:22:06 PDT) multi-call binary.” that is included with VMware ESXi 6.5 update 3:

[Wayback] bash – Hexadecimal To Decimal in Shell Script – Stack Overflow

Dealing with a very lightweight embedded version of busybox on Linux means many of the traditional commands are not available (bc, printf, dc, perl, python)

echo $((0x2f))
47

hexNum=2f
echo $((0x${hexNum}))
47

Credit to [Wayback] Peter Leung for this solution.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, Power User, Scripting, Software Development | Leave a Comment »

Getting the primary IP address (plain and CIDR) on Linux and OS X, then nmap scan on the associated subnet

Posted by jpluimers on 2021/12/13

The below answer works on my Linux and OS X systems (each having multiple network adapters configured):

[WayBack] bash – How to get the primary IP address of the local machine on Linux and OS X? – Stack Overflow

ip route get 1 | awk '{print $NF;exit}'

For Linux, I have this bash function:

# note the ";exit" lists the first default route interface, as there can be multiple
function nmap-fingerprint_network_of_default_interface() {
        default_if=$(ip route list | awk '/^default/ {print $5;exit}')
        default_if_cidr=$(ip -o -f inet addr show $default_if | awk '{print $4}')
        nmap -sP $default_if_cidr
}

And for OS X this one:

# requires ipcalc
function nmap-fingerprint_network_of_default_interface() {
        default_if=$(route -q -n get default | awk '/interface:/ {print $2;exit}')
        default_if_address=$(ifconfig $default_if | awk '/inet / {print $2;exit}')
        default_if_netmask_hex=$(ifconfig $default_if | awk '/inet / {print $4;exit}')
        default_if_network_bit_count=$(ipcalc --nocolor --class $default_if_address $default_if_netmask_hex)
        default_if_cidr=$(echo "$default_if_address/$default_if_network_bit_count")
        nmap -sP $default_if_cidr
}

These are the variables used:

  • default_if: network interface of the default route
  • default_if_cidr: IPv4 CIDR of the network interface of the default route (see Classless Inter-Domain Routing: CIDR notation – Wikipedia)
  • default_if_address: IPv4 address of network interface of the default route
  • default_if_netmask_hex: hexadecimal IPv4 network mask of network interface of the default route
  • default_if_network_bit_count: number of set bits in the IPv4 network mask of the network interface of the default route

Links used to get the above functions:

I might have gotten away with a pure bash solution (see [WayBack] Bash script for calculating network and broadcast addresses from ip and netmask or CIDR Notation · GitHub or my post Getting your local IPv4 addresses, netmasks and CIDRs), but the above works and is way shorter, and easier to maintain.

In stead of ipcalc, subnetcalc can do the same calculations and also supports IPv6, so that is something for a future try:

–jeroen

Posted in *nix, *nix-tools, Apple, bash, Color (software development), Development, Mac, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »

Kristian Köhntopp on Twitter: “Playing with a Ceph Storage Volume in a VM: The new openstack backend provisions an all-flash Ceph volume, which after tuning delivers a commit latency of around 1.1ms or so”

Posted by jpluimers on 2021/12/07

Not on an average VM (8 threads and 16gb memory) or network (100gbit), but ceph isn’t average solution.

For my link archive this long Twitter thread:

Archive.is Kristian Köhntopp on Twitter: “Playing with a Ceph Storage Volume in a VM: The new openstack backend provisions an all-flash Ceph volume, which after tuning delivers a commit latency of around 1.1ms or so. My dev VM has 8 Threads and 16 GB of memory, and mounts the Ceph Volume with XFS in /a.”

Hopefully this one day makes into a blog post at [Wayback] Die wunderbare Welt von Isotopp | Something old, something new, something borrowed, something blue.

–jeroen

Posted in *nix, *nix-tools, Linux, Power User, Storage | Leave a Comment »

console convert pcap to wav: not easily possible; use the WireShark GUI to do

Posted by jpluimers on 2021/12/01

Wanting a simple way on the console to convert a .pcap file to a .wav file, I searched for [Wayback] console convert pcap to wav – Google Search.

The reason is that [Wayback] fritzcap (written in Python) sometimes crashes while doing the conversion of a phone recording, so then only the .pcap file is available. I still want to figure this out, but given my health situation, I might not be able to in time.

Read the rest of this entry »

Posted in *nix, *nix-tools, Audio, Development, ffmpeg, Fritz!, Fritz!Box, fritzcap, Hardware, Media, Network-and-equipment, Power User, Python, Scripting, Software Development, Wireshark | Leave a Comment »

Some scripts and tips for easing the maintenance of a postfix based SMTP system

Posted by jpluimers on 2021/11/24

A few scripts and tips I found Googling around.

Deleting queued messages by regular expression pattern

I have seen the below delete-from-mailq.pl script numerous time, usually without any attribution (for instance [Wayback] Postfix Flush the Mail Queue – nixCraft and  [Wayback] postfix-delete.pl – Following script deletes all mail from the mailq which matches the regular expression specified as the first argument · GitHub).

The earliest version I could find was in [Wayback] ‘Re: delete messages from mailq’ – MARC by [Wayback] ‘Ralf Hildebrandt ‘ posts – MARC:

--- snip ---
#!/usr/bin/perl

$REGEXP = shift || die "no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!";

@data = qx</usr/sbin/postqueue -p>;
for (@data) {
  if (/^(\w+)\*?\s/) {
     $queue_id = $1;
  }
  if($queue_id) {
    if (/$REGEXP/i) {
      $Q{$queue_id} = 1;
      $queue_id = "";
    }
  }
}
                                
#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;

foreach (keys %Q) {
  print POSTSUPER "$_\n";
};
close(POSTSUPER);
--- snip ---

And then use:
% delete-from-mailq "^test"

 

Tips

[Wayback] How do I check the postfix queue size? – Server Fault

Lots of great answers and pointers to useful guides/software there.

qstat

[Wayback] Postfix Bottleneck Analysis points to [Wayback] Postfix manual – qshape(1): qshape - Print Postfix queue domain and age distribution, then explains about different scenarion and queues:

postqueue

postqueue -p | tail -n 1

Last line in the postqueue -p shows how many requests and size:

-- 317788 Kbytes in 11860 Requests.

View queues size

I tried finding the original posting of the below script, but could not. If you find it, please let me know.

#!/usr/bin/env perl

# postfix queue/s size
# author: 
# source: http://tech.groups.yahoo.com/group/postfix-users/message/255133

use strict;
use warnings;
use Symbol;
sub count {
        my ($dir) = @_;
        my $dh = gensym();
        my $c = 0;
        opendir($dh, $dir) or die "$0: opendir: $dir: $!\n";
        while (my $f = readdir($dh)) {
                if ($f =~ m{^[A-F0-9]{5,}$}) {
                        ++$c;
                } elsif ($f =~ m{^[A-F0-9]$}) {
                        $c += count("$dir/$f");
                }
        }
        closedir($dh) or die "closedir: $dir: $!\n";
        return $c;
}
my $qdir = `postconf -h queue_directory`;
chomp($qdir);
chdir($qdir) or die "$0: chdir: $qdir: $!\n";
printf "Incoming: %d\n", count("incoming");
printf "Active: %d\n", count("active");
printf "Deferred: %d\n", count("deferred");
printf "Bounced: %d\n", count("bounce");
printf "Hold: %d\n", count("hold");
printf "Corrupt: %d\n", count("corrupt");

Various commands

[Wayback] Inspecting Postfix’s email queue – Tech-G explaining about:

  • mailq
  • postqueue -p
  • postcat -vq XXXXXXXXXX (where XXXXXXXXXX is the message ID)
  • postqueue -f / postfix flush
  • postsuper -d to delete messages

More of these in [Wayback] Postfix Mail Queue Management – Linux Hint and [Wayback] Postfix Bottleneck Analysis: queues.

Makefile

Based on [Wayback] Using “make” for Postfix file maintenance

MAPS = relays.db aliases.db transport.db relocated.db \
        virtual.db sender_checks.db rejected_recips.db \
        helo_access.db

all : $(MAPS)

aliases.db : aliases
        newaliases

%.db : %
        postmap $*

This is my Makefile that runs fine on Tumbleweed (note: all 8-space indents are TAB characters):

MAPS =  /etc/aliases.db \
        transport.db \
        virtual.db \
        helo_access.db \
        canonical.db \
        sasl_passwd.db \
        relocated.db \
        relay.db \
        access.db \
        relay_ccerts.db \
        sender_canonical.db

all : $(MAPS)

aliases.db : aliases
        @echo "Rebuilding $@."
        newaliases

%.db : %
        @echo "Rebuilding $@."
        postmap $*

In the future, I might try [Wayback] Makefile.postfix · GitHub, though I think it is convoluted:


## Postfix: Makefile to update *.db files
POSTCONF= /usr/sbin/postconf
POSTMAP= /usr/sbin/postmap
default: postmap
postmap: Makefile.postmap
@echo 'Updating database files …'
$(MAKE) -f Makefile.postmap
Makefile.postmap: main.cf
@echo 'Updating $@ …'
@set -e; \
rm -f $@.$$$$.tmp; \
echo 'POSTMAP=$(POSTMAP)' >>$@.$$$$.tmp; \
echo 'postmap::' >>$@.$$$$.tmp; \
config_directory="$(PWD)"; \
{ $(POSTCONF) -c $(PWD) || kill $$$$; } \
|tr ' ' '\n' \
|sed -n \
-e 's/,$$//' \
-e 's#^hash:\$$config_directory/##p' \
-e 's#^hash:'"$$config_directory/##p" \
|sort -u \
|while read mapfile; do \
echo "postmap:: $$mapfile.db" >>$@.$$$$.tmp; \
echo "$$mapfile.db: $$mapfile" >>$@.$$$$.tmp; \
echo " \$$(POSTMAP) $$<" >>$@.$$$$.tmp; \
done; \
mv $@.$$$$.tmp $@

view raw

Makefile.postfix

hosted with ❤ by GitHub

 

 

[Wayback] Ralf Hildebrandt

Ralf Hildebrandt is an active and well-known figure in the Postfix community. He’s a systems engineer for T-NetPro, a German telecommunications company and has spoken about Postfix at industry conferences and contributes regularly to a number of open source mailing lists.

Co-author of this book: [Wayback: Book of Postfix State-of-the-Art Message Transport ISBN 9781593270018] (which used to have its own site: [Wayback: The Book of Postfix]

Book of Postfix

State-of-the-Art Message Transport

By Patrick Koetter, Ralf Hildebrandt

Publisher: No Starch PressRelease Date: March 2005Pages: 496

Best practices for Postfix–the popular alternative to Sendmail. Developed with security and speed in mind, Postfix has become a popular alternative to Sendmail and comes preinstalled in many Linux distributions as the default mailer. The Book of Postfix is a complete guide to Postfix whether used at home, as a mailrelay or virus-scanning gateway, or as a company mailserver. Practical examples show how to deal with daily challenges like protecting mail users from spam and viruses, managing multiple domains, and offering roaming access.

This is a great review of the book: [Wayback] The Book of Postfix (Ralf Hildebrandt, Patrick Koetter)

Related

For my postfix studies… « The Wiert Corner – irregular stream of stuff

–jeroen

 

Posted in *nix, *nix-tools, bash, Communications Development, Development, Internet protocol suite, Makefile, postfix, Power User, Scripting, SMTP, Software Development | Leave a Comment »

checkbashisms(1) – Linux man page

Posted by jpluimers on 2021/11/17

Even with lots of experience, one learns new things every day.

A while ago, I discovered checkbashisms which checks sh shel scripts (usually with extension .sh) scripts to they do not contain code specific to bash.

[Wayback] checkbashisms(1) – Linux man page

checkbashisms, based on one of the checks from the lintian system, performs basic checks on /bin/sh shell scripts for the possible presence of bashisms. It takes the names of the shell scripts on the command line, and outputs warnings if possible bashisms are detected.

Note that the definition of a bashism in this context roughly equates to “a shell feature that is not required to be supported by POSIX”; this means that some issues flagged may be permitted under optional sections of POSIX, such as XSI or User Portability.

In cases where POSIX and Debian Policy disagree, checkbashisms by default allows extensions permitted by Policy but may also provide options for stricter checking.

The source by now is a Perl script (it used to be a bash script) of which you can find the latest version here: [Wayback] scripts/checkbashisms.pl · master · Debian / devscripts · GitLab

Not installed by default

Virtually no distribution has checkbashisms installed by default.

In fact, the package containing checkbashisms heavily varies by distribution.

For OpenSuSE, it is in a package by itself: [Wayback] openSUSE Software: package checkbashisms

checkbashisms

Tool for Checking /bin/sh Scripts for Possible Bashisms

checkbashisms performs basic checks on /bin/sh shell scripts for the possible presence of bashisms. It takes the names of the shell scripts on the command line, and outputs warnings if possible bashisms are detected.

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sh, Sh Shell, Software Development | Leave a Comment »

How do I restart sshd on my Unix system | StarNet Knowledge Database – PC X, X Windows, X 11 & More – StarNet

Posted by jpluimers on 2021/11/17

[Wayback] How do I restart sshd on my Unix system | StarNet Knowledge Database – PC X, X Windows, X 11 & More – StarNet

RedHat and Fedora Core Linux

/sbin/service sshd restart

Suse linux

/etc/rc.d/sshd restart

Debian/Ubuntu

/etc/init.d/sshd restart

Solaris 9 and below

/etc/init.d/sshd stop
/etc/init.d/sshd start

Solaris 10

svcadm disable ssh
svcadm enable ssh

AIX

stopsrc -s sshd
startsrc -s sshd

HP-UX

/sbin/init.d/secsh stop
/sbin/init.d/secsh start

Note that for opensuse, by now you need this to restart sshd:

/usr/sbin/rcsshd restart

Edit 20211118: some tweets in reaction to this post

–jeroen

Posted in *nix, *nix-tools, Debian, Development, Linux, openSuSE, Power User, RedHat, Scripting, Software Development, SuSE Linux, systemd, Tumbleweed | Leave a Comment »

« Previous Entries
Next Entries »