The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,862 other subscribers

Archive for the ‘*nix’ Category

NVMe and SATA health data on ESXi: some links to investigate

Posted by jpluimers on 2021/08/25

(Edit 20221202: added one more link on “REALLOCATED SECTOR CT below threshold”)

Somehow, health data of my NVMe and SATA drives do not show up as health information on the web-ui of my ESXi playground rig.

So far, I noticed that ESXi runs a smartd, but does not ship with a smartctl, nor health data ends up in the web user interface. So you cannot see the state of NVMe and SATA devices easily.

Still these devices deteriorate over time and afterwards die, so below are some links to investigate later.

Goal is to use my own thresholds to set warning and error levels.

Some log entries:

Read the rest of this entry »

Posted in *nix, *nix-tools, ESXi6, ESXi6.5, Power User, PowerCLI, smartmontools/smartctl/smartd, Virtualization, VMware, VMware ESXi | Leave a Comment »

linux – How can I find all hardlinked files on a filesystem? – Super User

Posted by jpluimers on 2021/08/25

[WayBack] linux – How can I find all hardlinked files on a filesystem? – Super User

use the following line (for sure you have to replace /PATH/FOR/SEARCH/ with whatever you want to search):

find /PATH/FOR/SEARCH/ -xdev -printf '%i\t%n\t%p\n' | fgrep -f <(find . -xdev -printf '%i\n' | sort -n | uniq -d) | sort -n

this scans the filesystem only once, shows inode, number of hardlinks and path of files with more than one hardlink and sorts them according to the inode.

if you are annoyed by error messages for folders you aren’t allowed to read, you can expand the line to this:

find /PATH/FOR/SEARCH/ -xdev -printf '%i\t%n\t%p\n' 2> /dev/null | fgrep -f <(find . -xdev -printf '%i\n' 2> /dev/null | sort -n | uniq -d) | sort -n

It uses these commands:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, fgrep, find, Power User, Scripting, Software Development | 1 Comment »

Favourite Shortcut Key? (Soundcheck Question) – Computerphile – YouTube

Posted by jpluimers on 2021/08/20

Still a cool video. Many shortcuts for various operating systems and machines, including BBC B, Linux, Windows, and MacOS.

–jeroen

Read the rest of this entry »

Posted in *nix, 6502, Apple, BBC Micro B, History, Linux, Mac OS X / OS X / MacOS, Power User, Windows | Leave a Comment »

Windows chocolatey Wireshark install: ensure you install nmap too, so you have a pcap interface for capturing!

Posted by jpluimers on 2021/08/19

Wireshark is indispensable when doing network communications development or DevOps.

This is my choco-install-network-tools.bat batch file to install Wireshark and the pcap dependency which nmap provides:

choco install --yes nmap
:: wireshark requires a pcap for capturing; nmap comes with npcap which fulfills this dependency
:: see:
:: - https://chocolatey.org/packages/wireshark
:: - https://chocolatey.org/packages/win10pcap
:: - https://chocolatey.org/packages/WinPcap
:: - https://chocolatey.org/packages/nmap
choco install --yes wireshark

Yes, I know: Windows Subsystem for Linux could have an easier installation, but the above:

See:

–jeroen

Posted in *nix, *nix-tools, Communications Development, Development, Internet protocol suite, nmap, Power User, Software Development, Windows, Windows 10, Windows 8.1, WSL Windows Subsystem for Linux | Leave a Comment »

bash – Search for a previous command with the prefix I just typed – Unix & Linux Stack Exchange

Posted by jpluimers on 2021/08/18

[WayBack] bash – Search for a previous command with the prefix I just typed – Unix & Linux Stack Exchange answered by [WayBack] John1024:

What you are looking for is Ctrl-R.

Type Ctrl-R and then type part of the command you want. Bash will display the first matching command. Keep typing CtrlR and bash will cycle through previous matching commands.

To search backwards in the history, type Ctrl-S instead. (If Ctrl-S doesn’t work that way for you, that likely means that you need to disable XON/XOFF flow control: to do that, run stty -ixon.)

This is documented under “Searching” in man bash.

Comment by [WayBack] HongboZhu:

Ctrl-Q to quit the frozen state, if you already hit Ctrl-S without turning off flow control first and got your terminal frozen.

A far more elaborate answer with many other tips is from [WayBack] Peter Cordes:

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

Auto connect SSH without autossh?

Posted by jpluimers on 2021/08/10

Hopefully an example ssh config will follow.

[WayBack] Jeroen Pluimers on Twitter: “Would you mind sharing a trimmed down version of your ~/.ssh/config file? The bits from your posts are a bit fragmented now, so I’ve lost the overview (:”

–jeroen

Read the rest of this entry »

Posted in *nix, Communications Development, Development, Internet protocol suite, Power User, SSH, ssh/sshd, TCP | Leave a Comment »

On my list of things to try: Amazon SES for outbound/inbound email handling

Posted by jpluimers on 2021/08/10

SES mail servers at the time of writing

*n*x:

# nslookup -type=TXT amazonses.com | grep "v=spf1"
amazonses.com   text = "v=spf1 ip4:199.255.192.0/22 ip4:199.127.232.0/22 ip4:54.240.0.0/18 ip4:69.169.224.0/20 ip4:76.223.180.0/23 ip4:76.223.188.0/24 ip4:76.223.189.0/24 ip4:76.223.190.0/24 -all"I

Windows

C:\>nslookup -type=TXT amazonses.com | find "v=spf1"
Non-authoritative answer:
        "v=spf1 ip4:199.255.192.0/22 ip4:199.127.232.0/22 ip4:54.240.0.0/18 ip4:69.169.224.0/20 ip4:76.223.180.0/23 ip4:76.223.188.0/24 ip4:76.223.189.0/24 ip4:76.223.190.0/24 -all"

These addresses use a compact CIDR notation to denote ranges of networks containing ranges of network IPv4 addresses.

CIRD processing to sendmail access file

(this is linux sendmail only)

Converting the nslookup outout to a CIDR based sendmail /etc/mail/access excerpt goes via a pipe sequence of multiple sed commands:

# nslookup -type=TXT amazonses.com | grep "v=spf1" | sed 's/\(^.*"v=spf1 ip4:\| -all"$\)//g' | sed 's/\ ip4:/\n/g' | xargs -I {} sh -c "prips {} | sed 's/$/\tRELAY/g'"
199.255.192.0   RELAY
199.255.192.1   RELAY
...
76.223.190.254  RELAY
76.223.190.255  RELAY

What happens here is this:

  1. Filter out only spf1 records using grep.
  2. Remove the head (.*v=spf1 ip4:) and tail ( -all") of the output, see [WayBack] use of alternation “|” in sed’s regex – Super User.
  3. Replaces all ip4: with newlines (so the output get split over multiple lines), see [WayBack] linux – splitting single line into multiple line in numbering format using awk – Stack Overflow.
  4. Convert the CIDR notation to individual IP addresses (as sendmail cannot handle CIDR),
    1. This uses a combination of xargs with the  sh trick to split the CIDR list into separate arguments, and prips (which prints the IP addresses for a CIDR); see:
    2. Alternatively, use
  5. Replaces all end-of-line anchor ($) with a tab followed by RELAY, see

You can append the output of this command to /etc/mail/access, then re-generate /etc/mail/access.db and restart sendmail; see for instance [WayBack] sendmail access.db by example | LinuxWebLog.com.

Without the xargs, the output would look like this:

# nslookup -type=TXT amazonses.com | grep "v=spf1" | sed 's/\(^.*"v=spf1 ip4:\| -all"$\)//g' | sed 's/\ ip4:/\n/g'
199.255.192.0/22
199.127.232.0/22
54.240.0.0/18
69.169.224.0/20
76.223.180.0/23
76.223.188.0/24
76.223.189.0/24
76.223.190.0/24

Via

–jeroen

Posted in *nix, *nix-tools, Amazon SES, Amazon.com/.de/.fr/.uk/..., Cloud, Communications Development, Development, Infrastructure, Internet protocol suite, Power User, sendmail, SMTP, Software Development | Leave a Comment »

autossh on Windows from a service: automatically starting a tunnel no matter anyone being logged on

Posted by jpluimers on 2021/08/09

There is an autossh binary for Windows available on GitHub: [WayBack] GitHub – jazzl0ver/autossh: Windows binary for autossh v1.4c.

Combined with NSSM (which for instance you can install through [WayBack] Chocolatey Software | NSSM – the Non-Sucking Service Manager) you can not only automatically build and maintain an SSH connection, but also ensure the autossh process is up and running as a service without the need for an active logon.

This allows for SSH based tunnels from and to your Windows system.

For this usage scenario, there is no need for these tools any more:

Future research:

One time steps

These are in part based on:

1. Download autoSSH

Download the most recent [WayBack] Releases · jazzl0ver/autossh · GitHub  (see below for updates).

I used the 1.4g version: [WayBack] autossh.exe, then put on my Windows PATH.

2. Install NSSM

Since it is on chocolatey ([WayBack] Chocolatey Software | NSSM – the Non-Sucking Service Manager 2.24.101.20180116), this will suffice:

choco install --yes nssm

3 .Prepare remote computer so it allows enough SSH retries

Check the value of MaxAuthTries in /etc/ssh/sshd_config.

# grep MaxAuthTries /etc/ssh/sshd_config MaxAuthTries 1

The value needs to be at least 3 or higher for ssh-copy-id to work properly.

When changing the value, be sure to restart the sshd daemon.

Without a low value of MaxAuthTries in /etc/ssh/sshd_config, ssh-copy-id will give an error ERROR: Received disconnect from myRemoteComputer port 2222:2: Too many authentication failures.

See also these link via [WayBack ]“INFO: attempting to log in with the new key(s), to filter out any that are already installed” “Too many authentication failures” – Google Search:

4. Temporarily allow the remote account to perform interctive logon

Temporarily change the user shell to /bin/bash to allow [WayBack] ssh-copy-id to work at all.

This is explained in more detail by [WayBack] shell – ssh dissable login, but allow copy-id – Server Fault.

5. Generate public and private key pairs

You need an ssh public and private key, then transfer this to your Windows client. You can for instance use these as a base:

For instance (where myLocalUser is the local user generate the key-pair for for, and myRemoteUser plus myRemoteComputer is the remote user and computer you want to autossh to):

  • ssh-keygen -t rsa -b 4096 -f %UserProfile%\.ssh\id_rsa_myLocalUser@%ComputerName%_autossh_myRemoteUser@myRemoteComputer
  • ssh-keygen -t ed25519 -f %UserProfile%\.ssh\id_ed25519_myLocalUser@%ComputerName%_autossh_myRemoteUser@myRemoteComputer

6. install git (for ssh-copy-id and bash)

Since git includes ssh-copy-id (which you need in the next step, it is at %Program Files%\Git\usr\bin\ssh-copy-id) and git is on chocolatey ([WayBack] Chocolatey Software | Git (Install) 2.23.0):

choco install --yes git.install --params "/GitAndUnixToolsOnPath /NoGitLfs /SChannel /NoAutoCrlf /WindowsTerminal"

7. Copy the public parts of the generated key pairs to the remote account on the remote machine

Use bash with ssh-copy-id to transfer the generated public keys to a remote system (replace 2222 with the SSH port number on the remote computer; often it is just 22):

pushd %UserProfile%\.ssh
bash -c "ssh-copy-id -i %UserProfile%\.ssh\id_rsa_myLocalUser@%ComputerName%_autossh_myRemoteUser@myRemoteComputer -p 2222 myRemoteUser@myRemoteComputer"
bash -c "ssh-copy-id -i %UserProfile%\.ssh\id_ed25519_myLocalUser_%ComputerName%_autossh_myRemoteUser@myRemoteComputer -p 2222 myRemoteUser@myRemoteComputer"
popd

This sounds overly complicated, but is the only way to incorporate the environment variables.

8. Test with ssh, then with autossh

These two ssh commands should succeed; choose the one for which you prefer the rsa or ed25519 algorithm.

  • ssh -i %UserProfile%\.ssh\id_rsa_myLocalUser@%ComputerName%_autossh_myRemoteUser@myRemoteComputer -p 2222 myRemoteUser@myRemoteComputer
  • ssh -i %UserProfile%\.ssh\id_ed25519_myLocalUser_%ComputerName%_autossh_myRemoteUser@myRemoteComputer -p 2222 myRemoteUser@myRemoteComputer

After this, try with autossh:

  • autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -i %UserProfile%\.ssh\id_rsa_myLocalUser@%ComputerName%_autossh_myRemoteUser@myRemoteComputer -p 2222 myRemoteUser@myRemoteComputer
  • autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -i %UserProfile%\.ssh\id_ed25519_myLocalUser_%ComputerName%_autossh_myRemoteUser@myRemoteComputer -p 2222 myRemoteUser@myRemoteComputer

This disables the autossh port monitoring (the -M 0 option, but uses a combination of interval/count-max from ssh itself to monitor the connection (the -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" options).

Note that there is no default monitoring port, as it can be any one: [WayBack] linux – What is the default monitoring port for autossh? – Super User

9. Install autossh as a service

a

Steps

  1. a
  2. b
  3. c
  4. d
  5. e

SSH logon

Depending on which algorithm you like most, use either of the below 2 (replace 2222 with the SSH port number on the remote computer; often it is just 22):

  • ssh -i %UserProfile%\.ssh\id_rsa_myLocalUser@%ComputerName%_autossh_myRemoteUser@myRemoteComputer -p 2222 myRemoteUser@myRemoteComputer
  • ssh -i %UserProfile%\.ssh\id_ed25519_myLocalUser_%ComputerName%_autossh_myRemoteUser@myRemoteComputer -p 2222 myRemoteUser@myRemoteComputer

 

C:\Users\jeroenp>ssh-keygen -t ed25519 -f %UserProfile%\.ssh\id_ed25519_myUser_%ComputerName%_autossh_revue
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\jeroenp\.ssh\id_ed25519_myUser_D10U003_autossh_revue.
Your public key has been saved in C:\Users\jeroenp\.ssh\id_ed25519_myUser_D10U003_autossh_revue.pub.
The key fingerprint is:
SHA256:6qjzXhQtZpTzU6aryHMYuwVs5b4a/2COKxFGFQj0Eg4 jeroenp@D10U003
The key's randomart image is:
+--[ED25519 256]--+
|E+ oo...         |
|o =  .o.  o      |
| + .  *o.+       |
|  +. = o+        |
| . .+ o So       |
|  ...+ ..        |
|   o.=B.         |
|  o *@oo         |
|  .*O*=..        |
+----[SHA256]-----+

C:\Users\jeroenp>ssh-keygen -t rsa -b 4096 -f %UserProfile%\.ssh\id_rsa_myUser_%ComputerName%_autossh_revue
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\jeroenp\.ssh\id_rsa_myUser_D10U003_autossh_revue.
Your public key has been saved in C:\Users\jeroenp\.ssh\id_rsa_myUser_D10U003_autossh_revue.pub.
The key fingerprint is:
SHA256:WaWRoAnr4OuXAnc+MekpbdnNto71SgdMykp7XqylQr8 jeroenp@D10U003
The key's randomart image is:
+---[RSA 4096]----+
|    .   .....    |
|     o o  .+     |
|  . . o . o      |
| . o . + o       |
|  . o.o S        |
|. .o*o . .       |
| o.*oO.o* .      |
| .o %ooO+o       |
|  .= oE++o.      |
+----[SHA256]-----+

NSSM

NSSM is really cool to run any application as a service: [WayBack] NSSM – the Non-Sucking Service Manager

nssm is a service helper which doesn’t suck. srvany and other service helper programs suck because they don’t handle failure of the application running as a service. If you use such a program you may see a service listed as started when in fact the application has died. nssm monitors the running service and will restart it if it dies. With nssm you know that if a service says it’s running, it really is. Alternatively, if your application is well-behaved you can configure nssm to absolve all responsibility for restarting it and let Windows take care of recovery actions.

nssm logs its progress to the system Event Log so you can get some idea of why an application isn’t behaving as it should.

nssm also features a graphical service installation and removal facility. Prior to version 2.19 it did suck. Now it’s quite a bit better.

After installing, everything is command-line based (I cut away some blank lines for readability):

C:\bin\bin>nssm --help
NSSM: The non-sucking service manager
Version 2.24-101-g897c7ad 64-bit, 2017-04-26
Usage: nssm [ ...]

To show service installation GUI:

        nssm install []

To install a service without confirmation:

        nssm install   [ ...]

To show service editing GUI:

        nssm edit 

To retrieve or edit service parameters directly:

        nssm dump 
        nssm get   []
        nssm set   [] 
        nssm reset   []

To show service removal GUI:

        nssm remove []

To remove a service without confirmation:

        nssm remove  confirm

To manage a service:

        nssm start 
        nssm stop 
        nssm restart 
        nssm status 
        nssm statuscode 
        nssm rotate 
        nssm processes 

Windows binary autossh version

If it is behind on [WayBack] autossh (see version history at [WayBack] autossh/CHANGES.txt), then just ask for a new version; usually it gets built and released quickly: [WayBack] Any plans for 1.4g? · Issue #3 · jazzl0ver/autossh · GitHub

[WayBack] Releases · jazzl0ver/autossh · GitHub  at the time of writing:

–jeroen

Posted in *nix, *nix-tools, Communications Development, Development, Internet protocol suite, Power User, SSH, TCP | Leave a Comment »

Listing information on all active interfaces on MacOS part 1: getting the active interface names

Posted by jpluimers on 2021/07/29

Listing Listing information on all active interfaces on MacOS is a process involving multiple pieces, which then can be combined together.

Listing all active interfaces try 1

This involves both the -l (list with optional criteria) and -u parameter (the up criterion) as per excerpts from the [Archive.is] ifconfig(8) [osx man page] / [WayBack] ifconfig Man Page – macOS – SS64.com:

NAME
     ifconfig -- configure network interface parameters

SYNOPSIS
     ...
     ifconfig -l [-d] [-u] [address_family]
     ...

DESCRIPTION
     The ifconfig utility is used to assign an address to a network interface and/or configure network interface parameters.

     The following options are available:

     ...

     address_family
             Specify the address family which affects interpretation of the remaining parameters.  Since an interface can receive transmissions
             in differing protocols with different naming schemes, specifying the address family is recommended.  The address or protocol fami-
             lies currently supported are ``inet'', ``inet6'', and ``link''.  The default is ``inet''.  ``ether'' and ``lladdr'' are synonyms
             for ``link''.

     ...

     The -l flag may be used to list all available interfaces on the system, with no other additional information.  Use of this flag is mutually
     exclusive with all other flags and commands, except for -d (only list interfaces that are down) and -u (only list interfaces that are up).

Example:

ifconfig -l -u

Each interface on one line:

ifconfig -l -u | xargs -n1 echo

The problem is that on my system, it also lists bridges as active, whereas they are not:

# ifconfig -l -u | xargs -n1 echo
lo0
en1
en2
en0
p2p0
awdl0
bridge0
utun0
en10

# ifconfig bridge0
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    options=63<RXCSUM,TXCSUM,TSO4,TSO6>
    ether 6a:00:02:9a:23:f0 
    Configuration:
        id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
        maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
        root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
        ipfilter disabled flags 0x2
    member: en1 flags=3<LEARNING,DISCOVER>
            ifmaxaddr 0 port 5 priority 0 path cost 0
    member: en2 flags=3<LEARNING,DISCOVER>
            ifmaxaddr 0 port 6 priority 0 path cost 0
    Address cache:
    nd6 options=201<PERFORMNUD,DAD>
    media: 
    status: inactive

So this is where the MacOS and BSD documentation is inaccurate.

Interface types

The above interfaces are many more than just ethernet or WiFi interfaces; there is a list at [WayBack] macos – What are en0, en1, p2p, and so on, that are displayed after executing ifconfig? – Stack Overflow by [WayBackmcint:

In arbitrary order of my familarity / widespread relevance:

lo0 is loopback.

en0 at one point “ethernet”, now is WiFi (and I have no idea what extra en1 or en2 are used for).

fw0 is the FireWire network interface.

stf0 is an IPv6 to IPv4 tunnel interface to support the transition from IPv4 to the IPv6 standard.

gif0 is a more generic tunneling interface [46]-to-[46].

awdl0 is Apple Wireless Direct Link

p2p0 is related to AWDL features. Either as an old version, or virtual interface with different semantics than awdl.

many VPNs will add additional devices, often “utun#” or “utap#” following TUN/TAP (L3/L2)virtual networking devices.

More on AWDL at [WayBack] ios – What is AWDL (Apple Wireless Direct Link) and how does it work? – Stack Overflow.

Listing all active interfaces try 2

Read the rest of this entry »

Posted in *nix, *nix-tools, Apple, bash, Development, ifconfig, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »

-r argument to pipe (no argument for MacOS)- If no input is given to xargs, don’t let xargs run the utility – Unix & Linux Stack Exchange

Posted by jpluimers on 2021/07/28

TL;DR

There is a non-standard -r option to xargs that allows it to skip executing when there are no arguments at all.

On some operating systems, the -r is default.

MacOS has no -r, but does not execute xargs if there are no arguments given.

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development, xargs | Leave a Comment »