The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,854 other subscribers

Archive for the ‘*nix-tools’ Category

« Previous Entries
Next Entries »

On my list of *n*x things to play with: script and ttyrec

Posted by jpluimers on 2023/01/26

Because of [Archive] PragmaticProgrammers on Twitter: “Helpful Unix trick: use script to log your session. …” / Twitter:

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, bash, bash, Batch-Files, Development, Power User, Scripting, Software Development | Leave a Comment »

veltman/clmystery: A command-line murder mystery; an interactive tutorial for learning the *n*x command-line

Posted by jpluimers on 2023/01/18

Since I’m on a series of interactive tutorial sites, yesterday’s The SQL Murder Mystery made me bump into a project by [Archive] Noah Veltman (@veltman) / Twitter: [Wayback/Archive] veltman/clmystery: A command-line murder mystery

There’s been a murder in Terminal City, and TCPD needs your help.

To figure out whodunit, you need access to a command line.

Once you’re ready, clone this repo, or download it as a zip file.

Open a Terminal, go to the location of the files, and start by reading the file ‘instructions’.

I did a quick [Archive] clmystery – Twitter Search / Twitter and found the first ever Twitter mention to be this one from 2013 (boy, have I been living under a stone <g>): [Archive] RoR Group on Twitter: “A command-line murder mystery (clmystery) …” / Twitter.

Cool things:

–jeroen

Posted in *nix, *nix-tools, Apple, Development, Interactive Tutorials, Learning/Teaching, LifeHacker, Linux, Mac OS X / OS X / MacOS, Power User, Software Development, Terminal | 2 Comments »

Getting your public IP address from the command-line when http and https are blocked: use DNS

Posted by jpluimers on 2022/12/28

Years ago, I wrote Getting your public IP address from the command-line. All methods were http based, so were very easy to execute using cURL.

But then in autumn 2021, Chris Bensen wrote this cool little blog-post [Wayback/Archive] Chris Bensen: How do I find my router’s public IP Address from the command line?:

dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com

At first sight, I thought it was uncool, as the command was quite long and there was no explanation of the dig command trick.

But then, knowing that dig is a DNS client, it occurred to me: this perfectly works when http and https are disabled by your firewall, but the DNS protocol works and gives the correct result:

# dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com
"80.100.143.119"

This added the below commands and aliases to my tool chest for *nix based environments like Linux and MacOS (not sure yet about Windows yet :), but that still doesn’t explain why it worked. So I did some digging…

IPv4

  • command:
    dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com
  • command removing outer double quotes:
    dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs
  • alias:
    alias "whatismyipv4_dns=dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs"

IPv6

  • command:
    dig -6 TXT +short o-o.myaddr.l.google.com @ns1.google.com
  • command removing outer double quotes:
    dig -6 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs
  • alias:
    alias "whatismyipv6_dns=dig -6 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs"

How it works

Let’s stick to dig and IPv4 as that not having IPv6 (regrettably still) is the most common situation today:

# dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com
"80.100.143.119"

What it does is request the DNS TXT record of o-o.myaddr.l.google.com from the Google DNS server ns1.google.com and returns the WAN IPv4 address used in the DNS request, which is for instance explained in [Wayback/Archive] What is the mechanics behind “dig TXT o-o.myaddr.l.google.com @ns1.google.com” : linuxadmin.

Since these are TXT records, dig will automatically double quote them, which xargs can remove (see below how and why):

# dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs
80.100.143.119

The DNS query will fail when requesting the Google Public DNS servers 8.8.8.8 or 8.8.4.4:

# dig -4 TXT +short o-o.myaddr.l.google.com @8.8.8.8
"2a00:1450:4013:c1a::103"
"edns0-client-subnet 80.101.239.0/24"

Or, with quotes removed (the -L 1 ensures that xargs performs the quote-pair removal action on each line):

# dig -4 TXT +short o-o.myaddr.l.google.com @8.8.8.8 | xargs -L 1
2a00:1450:4013:c1a::103
edns0-client-subnet 80.101.239.0/24

This request is both slower than requesting the ns1.google.com server and wrong.

The reason is that only ns1.google.com understands the special o-o.myaddr.l.google.com hostname which instructs it to return the IP address of the requesting dig DNS client.

That 8.8.8.8 returns a different IP address and an additional edns0-client-subnet with less accurate information is explained in an answer to [Wayback/Archive] linux – Getting the WAN IP: difference between HTTP and DNS – Stack Overflow by [Wayback/Archive] argaz referring to this cool post: [Wayback/Archive] Which CDNs support edns-client-subnet? – CDN Planet.

Not just ns1.google.com: any DNS server serving the google.com domain

Since o-o.myaddr.l.google.com is part of the google.com domain, the above works for any DNS server serving the google.com domain (more on that domain: [Wayback/Archive] General DNS overview  |  Google Cloud).

Getting the list of DNS servers is similar to getting the list of MX servers which I explained in Getting the IP addresses of gmail MX servers, replacing MX record type (main exchange) with the NS record type (name server) and the gmail.com domain with the google.com domain:

# dig @8.8.8.8 +short NS google.com
ns3.google.com.
ns1.google.com.
ns2.google.com.
ns4.google.com.

The ns1.google.com DNS server is a special one of the NS servers: it is the start of authority server, which you can query using the SOA record type that also gives slightly more details for this server:

# dig @8.8.8.8 +short SOA google.com
ns1.google.com. dns-admin.google.com. 410477869 900 900 1800 60

The difference between using NS and SOA records with dig are explained in the [Wayback] dns – How do I find the authoritative name-server for a domain name? – Stack Overflow answer by [Wayback/Archive] bortzmeyer who also explains how to help figuring out SOA and NS discrepancies (note to self: check out the check_soa tool originally by Michael Fuhr (I could not find recent content of him, so he might have passed away) of which source code is now at [Wayback/Archive] Net-DNS/check_soa at master · NLnetLabs/Net-DNS).

So this works splendid as well using ns4.google.com on my test system:

# dig -4 TXT +short o-o.myaddr.l.google.com @ns4.google.com | xargs
80.100.143.119

The xargs removes outer quotes removal trick

[Wayback/Archive] string – Shell script – remove first and last quote (“) from a variable – Stack Overflow (thanks quite anonymous [Wayback/Archive] user1587520):

> echo '"quoted"' | xargs
quoted

xargs uses echo as the default command if no command is provided and strips quotes from the input.

More on https versus DNS requests

Some notes are in [Wayback/Archive] How to get public IP address from Linux shell, but note the telnet trick now fails as myip.gelma.net is gone (latest live version was archived in the Wayback Machine in august 2019).

Via

–jeroen

Posted in *nix, *nix-tools, Apple, bash, bash, Batch-Files, Communications Development, Development, DNS, Internet protocol suite, Linux, Mac, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development, TCP | Leave a Comment »

Exporting your Twitter content, converting to Markdown and getting the image alt-texts (thanks @isotopp/@HBeckPDX/@weiglemc for the info and @kcgreenn/@dreamjar for the comic!)

Posted by jpluimers on 2022/11/12

This is fine #Twitter

This is fine #Twitter (illustration inspired by KC Green; creation video below)

(Edit 20221114: script for high-res images; more tweets from Jan) (Edit 20221116: hat-tip to Sam) (Edit 20221120: archiving t.co links by Michele Weigle) (Edit 20221122: added article by Johan van der Knijff) (20221128 Tapue export tool by Mike Hucka)

Time to be prepared:

The below will help you exporting your Twitter content (Tweets, DMs, media), perform some conversions on them and optionally delete (parts of) your content.

Important: keep your Twitter account afterwards (to prevent someone from creating a new account with the same handle).

Read the rest of this entry »

Posted in *nix, *nix-tools, Awk, Development, Lightweight markup language, MarkDown, Mastodon, Power User, SocialMedia, Software Development, Twitter | Leave a Comment »

Stop using tail -f (mostly)

Posted by jpluimers on 2022/11/07

Some interesting bits from [WayBack] Stop using tail -f (mostly):

  • say you want to watch the file production.log: 
    $ less +F production.log

Important
log
information
here

Waiting for data... (interrupt to abort)

    Here you have pretty much the same behaviour you’d get with tail.

    Now let’s say something interesting appears, and you want to search all the occurrences of “foo”. You can just hit Ctrl-c to go to “normal” less mode (as if you had opened the file without the +F flag), and then you have all the normal less features you’d expect, including the search with /foo. You can go to the next or previous occurrence with n or N, up and down with j and k, create marks with m and do all sort of things that less(1) says you can do.

    Once you are done, just hit F to go back to watching mode again. It’s that easy.

  • When you need to watch multiple files at the same time, tail -f can actually give you a better output.

Related: [WayBack] shell – The ‘less’ command-line equivalent of ‘tail -f’ – Unix & Linux Stack Exchange

I prefer tail -F

-F – The -F option implies the -f option, but tail will also check to see if the file being followed has been renamed or rotated.

The less equivalent: less +F --follow-name

–jeroen

Posted in *nix, *nix-tools, less, Power User | Leave a Comment »

On my reading list: Windows Console and PTY

Posted by jpluimers on 2022/10/25

With the rise of *nix tools and infrastructure on Windows (including, but certainly not limited to Visual Studio Code and Windows Subsystem for Linux), I need to get acquainted to the new ways these interface to the Windows Console.

Since Windows Console is from the (now obsolete) UCS-2 days, so it is not even fully Unicode aware, and has trouble with UTF-8, UTF-16.

So here are some links for my reading list:

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, CommandLine, ConPTY, Console (command prompt window), Development, Linux, Power User, Software Development, Windows, Windows 10, Windows 11, Windows Development, Windows Terminal, WSL Windows Subsystem for Linux | Leave a Comment »

Python Breaking Distance calculator (includes the German DIN 1451 font for traffic signage)

Posted by jpluimers on 2022/09/20

If you ever want a good visual representation to compare the breaking distance for a car at two different speeds, and see at what speed you will hit the human “obstacle”, then use the Python script mkbremsweg.py.

Git repository: [Wayback/Archive.is] joschtl / bremsweg · GitLab.

The image is generated in the current directory

It probably won’t work on Windows as it dynamically builds a very long command-line calling ImageMagick tool [Wayback/Archive.is] convert once to do all the drawing.

The text in the picture for now is hardcoded in German, but would be easy to adopt.

The fonts used are and FreeSans and [Wayback/Archive.is] Alte DIN 1451 Mittelschrift Font Family · 1001 Fonts (which the Germans use for Traffic Signage and is very similar to fonts used in other countries).

Calculations are based on [Wayback/Archive.is] Bremsweg-Rechner für Anhalteweg & Bremsweg – Johannes Strommer.

Via:

–jeroen

Posted in *nix, *nix-tools, cars, Development, ImageMagick, LifeHacker, Power User, Python, Scripting, Software Development, Traffic, Windows | Leave a Comment »

Since about 5 months now, there has been a new Chocolatey package maintainer for OpenVPN

Posted by jpluimers on 2022/08/26

Last winter, I discovered that the OpenVPN version on Chocolatey was really old: it had not been updated since 2019.

Most  Chocolatey maintainers are volunteers and sometimes the burden can become too large. Back then the maintainer was [Wayback/Archive] Chocolatey Software | wget, but luckily [Wayback/Archive] Chocolatey Software | dgalbraith has stepped in and in March 2022 bumped the version from [Wayback/Archive] Chocolatey Software | OpenVPN 2.4.7 to [Wayback/Archive] Chocolatey Software | OpenVPN – Open Source SSL VPN Solution 2.5.4 and kept maintaining (currently there is [Wayback/Archive] Chocolatey Software | OpenVPN – Open Source SSL VPN Solution 2.5.7).

Read the rest of this entry »

Posted in *nix, *nix-tools, Chocolatey, Hardware, Network-and-equipment, OpenVPN, Power User, ssh/sshd, VPN, Windows | Leave a Comment »

Don’t fall for the golden hammer: avoid git empty commits, especially for kicking off parts of your CI/CD

Posted by jpluimers on 2022/08/16

A while back Kristian Köhntopp (isotopp) wrote a blog post after quite a Twitter argument where he poses against using git empty commits. I’m with Kris: don’t use them for anything, especially not for kicking off your CI/CD.

Basically his blog post is all about avoiding to think you have a golden hammer, and avoid falling for the Law of the instrument – Wikipedia.

Originally, Abraham Maslow said in 1966:

“I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”

For me this has all to do with preventing technical debt: find the right tool to kick your CI/CD pipeline after part of that chain somehow malfunctioned is way better than polluting the commit history with empty commits.

His blog post: [Wayback/Archive.is] Empty commits and other wrong tools for the job | Die wunderbare Welt von Isotopp

The most important bit in it:

And since we are talking about CI/CD pipelines: Don’t YAML them. Don’t JSON them. Don’t XML them.

…

Programming in any of these three is wrong use of tooling, and you should not do it.

  • YAML, JSON and XML are for declarative things.
  • Python, Go and Rust are for procedural things.
  • Bash is for interactive use only.

Use the proper tooling for the job. Be an engineer.

This very much reminds me of an Entwickler Konferenz keynote a long time ago, where Neal Ford made the point that most software engineers act very much unlike what is expected from traditional engineering way of operating where the engineer is both responsible and liable for his actions.

The start of the Twitter thread: [Archive.is] Kristian Köhntopp on Twitter: “A lot of people right now that git is an API and triggering CI/CD pipelines with empty commits replaces the equivalent of a Kubernetes controller for their fragile pile of bash in git triggers. This is broken and begets more brokenness. Evidence:… “

The tweet that started the subtweet: [Archive.is] Florian Haas on Twitter: “(For anyone wondering, what’s nice about this one is it works in any CI. So you don’t have to remember how to manually kick off a GitLab CI pipeline or GitHub Action or Zuul job, you just push an empty commit and off you go.)”

Other relevant tweets:

Yes, you want to avoid shell too (anything like for instance sh, ash, dash, bash or zsh), but you have to know it (and understand why to avoid it) as often it is the only interactive way to access systems from the console.

And of course Kris also wrote a big document on that too, which is available as full PDF (Wayback), full HTML (Wayback) and chaptered HTML Die UNIX Shell /bin/sh.

But more importantly, Kris wrote [Wayback/Archive.is] Using Python to bash | Die wunderbare Welt von Isotopp which is about using Python to do things you might be tempted to do in the shell. It quotes

Shell is a thing you want to understand and then not use, because you learned to understand it.

which is from the German post in thread [Wayback/Archive.is] Bashprogrammierung, wo gehts am besten los which quotes Kris’ 1998 message:

From kris Tue Sep 1 11:26:12 1998
From: kris
Newsgroups: de.comp.os.unix.misc
Subject: Re: Shell-Frage, find, xargs, kopieren von vielen Dateien
References: <6seh24$q9a$2...@nz12.rz.uni-karlsruhe.de>
From: kr...@koehntopp.de (Kristian Koehntopp)
Alignment: chaotic/neutral
X-Copyright: (C) Copyright 1987-1998 Kristian Koehntopp -- All rights
reserved.
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

Marc.Hab...@gmx.de (Marc Haber) writes:
>mir ist das ganze Zeug mit der Shell, find, xargs und Konsorten noch
>reichlich verschlüsselt.

http://www.koehntopp.de/kris/artikel/unix/shellprogrammierung/

>xargs hin oder sollte ich besser ein Perlskript schreiben?

Verwende Perl. Shell will man koennen, dann aber nicht verwenden.

Kristian

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, Conference Topics, Conferences, Continuous Integration, Development, DVCS - Distributed Version Control, Event, git, Power User, Scripting, sh, Sh Shell, Software Development, Source Code Management, Technical Debt | Leave a Comment »

How can you export the Visual Studio Code extension list? (via: Stack Overflow)

Posted by jpluimers on 2022/06/16

Adapted from [Archive.is] How can you export the Visual Studio Code extension list? – Stack Overflow, presuming that code is on the PATH:

  1. From the command-line interface on MacOS, Linux, BSD or on Windows with git installed:
    code --list-extensions | xargs -L 1 echo code --install-extension
  2. From the command-line interface on MacOS, Linux, BSD or on Windows without git installed:
    code --list-extensions | % { "code --install-extension $_" }

    or, as I think, more clearly (see also [WayBack] syntax – What does “%” (percent) do in PowerShell? – Stack Overflow):

    code --list-extensions | foreach { "code --install-extension $_" }

    or even more explanatory:

    code --list-extensions | ForEach-Object { "code --install-extension $_" }
  3. From the command-line interface on Windows as a plain cmd.exe command:
    @for /f %l in ('code --list-extensions') do @echo code --install-extension %l
  4. On Windows as a plain cmd.exe batch file (in a .bat/.cmd script):
    @for /f %%l in ('code --list-extensions') do @echo code --install-extension %%l
  5. The above two on Windows can also be done using PowerShell:
    PowerShell -Command "code --list-extensions | % { """""code --install-extension $_""""" }"

    Note that here too, the % can be expanded into foreach or ForEach-Object for clarity.

All of the above prepend “code --install-extension ” (note the trailing space) before each installed Visual Studio Code extension.

They all give you a list like this which you can execute on any machine having Visual Studio Code installed and its code on the PATH, and a working internet connection:

code --install-extension DavidAnson.vscode-markdownlint
code --install-extension ms-vscode.powershell
code --install-extension yzhang.markdown-all-in-onex

(This is about the minimum install for me to edit markdown documents and do useful things with PowerShell).

Of course you can pipe these to a text-file script to execute them later on.

The double-quote escaping is based on [Wayback/Archive.is] How to escape PowerShell double quotes from a .bat file – Stack Overflow:

you need to escape the " on the command line, inside a double quoted string. From my testing, the only thing that seems to work is quadruple double quotes """" inside the quoted parameter:

powershell.exe -command "echo '""""X""""'"

Via: [Archive.is] how to save your visual studio code extension list – Google Search

--jeroen

Posted in *nix, *nix-tools, .NET, bash, Batch-Files, CommandLine, Console (command prompt window), Development, Mac OS X / OS X / MacOS, Power User, PowerShell, PowerShell, Software Development, Visual Studio and tools, vscode Visual Studio Code, Windows, Windows 10, Windows 7, Windows 8, Windows 8.1, Windows Development, Windows Server 2008, Windows Server 2008 R2, Windows Server 2012, Windows Server 2012 R2, Windows Server 2016, WSL Windows Subsystem for Linux, xargs | Leave a Comment »

« Previous Entries
Next Entries »