The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,862 other subscribers

Archive for the ‘*nix’ Category

xxd examples of big/little/middle endianness (thanks @jilles_com!)

Posted by jpluimers on 2023/04/18

Cool one-liner program via [Archive] Jilles🏳️‍🌈 (@jilles_com) / Twitter:

for s in 0123456789ABCDEF 172.16.0.254 Passwd:admin;do echo -en "Big    Endian: $s\nMiddle Endian: ";echo -n $s|xxd -e -g 4 | xxd -r;echo -en "\nLittle Endian: ";echo -n $s|xxd -e -g 2 | xxd -r;echo -en "\nReversed     : ";echo -n $s|xxd -p -c1 | tac | xxd -p -r;echo -e "\n";done

Note that the hex are bytes, not nibbles, so the endianness is OK:

Image

Big Endian: 0123456789ABCDEF
Middle Endian: 32107654BA98FEDC
Little Endian: 1032547698BADCFE
Reversed : FEDCBA9876543210

Big Endian: 172.16.0.254
Middle Endian: .2710.61452.
Little Endian: 71.2610.2.45
Reversed : 452.0.61.271

Big Endian: Passwd:admin
Middle Endian: ssaPa:dwnimd
Little Endian: aPssdwa:mdni
Reversed : nimda:dwssaP

That nibble/byte thing confused me at first (as I associate hexadecimal output with hex dumps, where each hexadecimal character represents a nibble)) so here are some interesting messages from the thread that Jilles_com started:

Some related man pages:

–jeroen

Posted in *nix, *nix-tools, bash, Development, Power User, Scripting, Software Development, xxd | Leave a Comment »

Installing Windows OpenSSH from the command-line on Windows 10 and 11

Posted by jpluimers on 2023/03/28

While writing On my reading list: Windows Console and PTY, I found out that OpenSSH had become available as an optional Windows feature.

It was in [Wayback/Archive.is] Windows Command-Line: Introducing the Windows Pseudo Console (ConPTY) | Windows Command Line:

Thankfully, OpenSSH was recently ported to Windows and added as a Windows 10 optional feature. PowerShell Core has also adopted ssh as one of its supported PowerShell Core Remoting protocols.

Here are a few links:

Read the rest of this entry »

Posted in *nix, *nix-tools, Communications Development, ConPTY, Console (command prompt window), Development, Internet protocol suite, OpenSSH, Power User, SSH, ssh/sshd, TCP, Windows, Windows 10, Windows 11 | Leave a Comment »

llamasoft/polyshell: A Bash/Batch/PowerShell polyglot!

Posted by jpluimers on 2023/03/16

PolyShell is a script that’s simultaneously valid in Bash, Windows Batch, and PowerShell (i.e. a polyglot).

[Wayback/Archive] llamasoft/polyshell: A Bash/Batch/PowerShell polyglot!

Need to check this out, as often I have scripts that have to go from one language to the other or vice versa.

Maybe it enables one language to bootstrap functionality in the other?

The quest

The above polyglot started with a quest to see if I can could include some PowerShell statements in a batch file with two goals:

  1. if the batch file started from the PowerShell command prompt, then execute the PowerShell code
  2. if the batch file started from the cmd.exe command prompt, then have it start PowerShell with the same command-line arguments

The reasoning is simple:

  1. PowerShell scripts will start from the PATH only when PowerShell is already running
  2. Batch files start from the path when either cmd.exe or PowerShell are running

Lots of users still live in the cmd.exe world, but PowerShell scripts are way more powerful, and since PowerShell is integrated in Windows since version 7, so having a batch file bootstrap PowerShell still makes sense.

Since my guess was about quoting parameters the right way, my initial search for the link below was [Wayback/Archive] powershell execute statement from batch file quoting – Google Search.

I have dug not yet into this, so there are still…

Many links to read

These should give me a good idea how to implement a polyglot batch file/PowerShell script.

–jeroen

Posted in *nix, *nix-tools, bash, bash, Batch-Files, Development, JavaScript/ECMAScript, Perl, Polyglot, Power User, PowerShell, Scripting, Software Development | Leave a Comment »

linux – Newline-separated xargs – Server Fault

Posted by jpluimers on 2023/03/07

A long time ago, on just one system, I forgot which one, I needed explicit [Wayback/Archive] linux – Newline-separated xargs – Server Fault.

The simple solution was to replace the newline with null before running xargs:

tr '\n' '\0'

The clean solution was to install [Wayback/Archive] gnu xargs:

GNU xargs (default on Linux; install findutils from MacPorts on OS X to get it) supports [Wayback/Archive] -d which lets you specify a custom delimiter for input, so you can do

ls *foo | xargs -d '\n' -P4 foo 

–jeroen

Posted in *nix, *nix-tools, bash, Development, Power User, Scripting, Software Development, xargs | Leave a Comment »

Wireshark Cheat Sheet – Commands, Captures, Filters, Shortcuts

Posted by jpluimers on 2023/02/28

[Wayback/Archive] Wireshark Cheat Sheet – Commands, Captures, Filters, Shortcuts

It is available both a huge [Wayback/Archive] jpg (2500×2096 pixels), so it already prints well on A5 or A4 sized paper for reference and as a [Wayback/Archive] PDF (so you can print it on even larger paper sizes).

Via: [Archive] Murdock (@Generic42) / Twitter in a DM.

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, Communications Development, Development, Hardware, Network-and-equipment, Power User, Software Development, Wireshark | Leave a Comment »

On my list of *n*x things to play with: script and ttyrec

Posted by jpluimers on 2023/01/26

Because of [Archive] PragmaticProgrammers on Twitter: “Helpful Unix trick: use script to log your session. …” / Twitter:

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, bash, bash, Batch-Files, Development, Power User, Scripting, Software Development | Leave a Comment »

veltman/clmystery: A command-line murder mystery; an interactive tutorial for learning the *n*x command-line

Posted by jpluimers on 2023/01/18

Since I’m on a series of interactive tutorial sites, yesterday’s The SQL Murder Mystery made me bump into a project by [Archive] Noah Veltman (@veltman) / Twitter: [Wayback/Archive] veltman/clmystery: A command-line murder mystery

There’s been a murder in Terminal City, and TCPD needs your help.

To figure out whodunit, you need access to a command line.

Once you’re ready, clone this repo, or download it as a zip file.

Open a Terminal, go to the location of the files, and start by reading the file ‘instructions’.

I did a quick [Archive] clmystery – Twitter Search / Twitter and found the first ever Twitter mention to be this one from 2013 (boy, have I been living under a stone <g>): [Archive] RoR Group on Twitter: “A command-line murder mystery (clmystery) …” / Twitter.

Cool things:

–jeroen

Posted in *nix, *nix-tools, Apple, Development, Interactive Tutorials, Learning/Teaching, LifeHacker, Linux, Mac OS X / OS X / MacOS, Power User, Software Development, Terminal | 2 Comments »

Chris Bensen on Raspberry Pi and clusters

Posted by jpluimers on 2022/12/29

Oracle’s Pi Supercomputer

Oracle’s Pi Supercomputer ; click on the image for a larger version.

In 2019 ans 2020, [Archive] Chris Bensen and his [Archive] Oracle Groundbreakers team built a really large Raspberry Pi cluster of more than 1k pies, all network booting to become a cluster. It was for instance covered in the [Wayback/Archive] Building the World’s Largest Raspberry Pi Cluster – DZone IoT.

On his [Wayback/Archive] personal blog, he wrote a few posts like [Wayback/Archive] Chris Bensen: Raspberry Pi Overlay Root Filesystem and [Wayback/Archive] Chris Bensen: Get MAC Address for a Pi Cluster.

It made me also bump into [Wayback/Archive] Building the world’s largest Raspberry Pi cluster early 2020,

Since that wast right after the start of my rectum cancer treatment which lasted longer and, because of I got metastases a few months after radiation treatment, required more treatments than anticipated, I put a note in my bog drafts and kind of lost track.

So I was glad that in fall 2021, I bumped into the draft and found an almost year old post [Wayback/Archive] Chris Bensen: All Raspberry Pi Super Computer Posts in One Spot which is an index in all the blog posts and videos that Chris and his team produced on this project.

I then also learned the cluster had been shown on [Wayback/Archive] Oracle OpenWorld 2019, Breakthrough Starts Here and was covered in the [Archive] Top 10 Raspberry Pi Projects of 2019 | Tom’s Hardware (where I got the [Wayback/Archive] Oracle World 2019 having the 1k+ node Raspberry Pi cluster on display picture shown on the right from).

Read the rest of this entry »

Posted in *nix, Development, Hardware Development, Linux, OracleLinux, Power User, Raspberry Pi | Leave a Comment »

Getting your public IP address from the command-line when http and https are blocked: use DNS

Posted by jpluimers on 2022/12/28

Years ago, I wrote Getting your public IP address from the command-line. All methods were http based, so were very easy to execute using cURL.

But then in autumn 2021, Chris Bensen wrote this cool little blog-post [Wayback/Archive] Chris Bensen: How do I find my router’s public IP Address from the command line?:

dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com

At first sight, I thought it was uncool, as the command was quite long and there was no explanation of the dig command trick.

But then, knowing that dig is a DNS client, it occurred to me: this perfectly works when http and https are disabled by your firewall, but the DNS protocol works and gives the correct result:

# dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com
"80.100.143.119"

This added the below commands and aliases to my tool chest for *nix based environments like Linux and MacOS (not sure yet about Windows yet :), but that still doesn’t explain why it worked. So I did some digging…

IPv4

  • command:
    dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com
  • command removing outer double quotes:
    dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs
  • alias:
    alias "whatismyipv4_dns=dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs"

IPv6

  • command:
    dig -6 TXT +short o-o.myaddr.l.google.com @ns1.google.com
  • command removing outer double quotes:
    dig -6 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs
  • alias:
    alias "whatismyipv6_dns=dig -6 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs"

How it works

Let’s stick to dig and IPv4 as that not having IPv6 (regrettably still) is the most common situation today:

# dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com
"80.100.143.119"

What it does is request the DNS TXT record of o-o.myaddr.l.google.com from the Google DNS server ns1.google.com and returns the WAN IPv4 address used in the DNS request, which is for instance explained in [Wayback/Archive] What is the mechanics behind “dig TXT o-o.myaddr.l.google.com @ns1.google.com” : linuxadmin.

Since these are TXT records, dig will automatically double quote them, which xargs can remove (see below how and why):

# dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs
80.100.143.119

The DNS query will fail when requesting the Google Public DNS servers 8.8.8.8 or 8.8.4.4:

# dig -4 TXT +short o-o.myaddr.l.google.com @8.8.8.8
"2a00:1450:4013:c1a::103"
"edns0-client-subnet 80.101.239.0/24"

Or, with quotes removed (the -L 1 ensures that xargs performs the quote-pair removal action on each line):

# dig -4 TXT +short o-o.myaddr.l.google.com @8.8.8.8 | xargs -L 1
2a00:1450:4013:c1a::103
edns0-client-subnet 80.101.239.0/24

This request is both slower than requesting the ns1.google.com server and wrong.

The reason is that only ns1.google.com understands the special o-o.myaddr.l.google.com hostname which instructs it to return the IP address of the requesting dig DNS client.

That 8.8.8.8 returns a different IP address and an additional edns0-client-subnet with less accurate information is explained in an answer to [Wayback/Archive] linux – Getting the WAN IP: difference between HTTP and DNS – Stack Overflow by [Wayback/Archive] argaz referring to this cool post: [Wayback/Archive] Which CDNs support edns-client-subnet? – CDN Planet.

Not just ns1.google.com: any DNS server serving the google.com domain

Since o-o.myaddr.l.google.com is part of the google.com domain, the above works for any DNS server serving the google.com domain (more on that domain: [Wayback/Archive] General DNS overview  |  Google Cloud).

Getting the list of DNS servers is similar to getting the list of MX servers which I explained in Getting the IP addresses of gmail MX servers, replacing MX record type (main exchange) with the NS record type (name server) and the gmail.com domain with the google.com domain:

# dig @8.8.8.8 +short NS google.com
ns3.google.com.
ns1.google.com.
ns2.google.com.
ns4.google.com.

The ns1.google.com DNS server is a special one of the NS servers: it is the start of authority server, which you can query using the SOA record type that also gives slightly more details for this server:

# dig @8.8.8.8 +short SOA google.com
ns1.google.com. dns-admin.google.com. 410477869 900 900 1800 60

The difference between using NS and SOA records with dig are explained in the [Wayback] dns – How do I find the authoritative name-server for a domain name? – Stack Overflow answer by [Wayback/Archive] bortzmeyer who also explains how to help figuring out SOA and NS discrepancies (note to self: check out the check_soa tool originally by Michael Fuhr (I could not find recent content of him, so he might have passed away) of which source code is now at [Wayback/Archive] Net-DNS/check_soa at master · NLnetLabs/Net-DNS).

So this works splendid as well using ns4.google.com on my test system:

# dig -4 TXT +short o-o.myaddr.l.google.com @ns4.google.com | xargs
80.100.143.119

The xargs removes outer quotes removal trick

[Wayback/Archive] string – Shell script – remove first and last quote (“) from a variable – Stack Overflow (thanks quite anonymous [Wayback/Archive] user1587520):

> echo '"quoted"' | xargs
quoted

xargs uses echo as the default command if no command is provided and strips quotes from the input.

More on https versus DNS requests

Some notes are in [Wayback/Archive] How to get public IP address from Linux shell, but note the telnet trick now fails as myip.gelma.net is gone (latest live version was archived in the Wayback Machine in august 2019).

Via

–jeroen

Posted in *nix, *nix-tools, Apple, bash, bash, Batch-Files, Communications Development, Development, DNS, Internet protocol suite, Linux, Mac, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development, TCP | Leave a Comment »

Exporting your Twitter content, converting to Markdown and getting the image alt-texts (thanks @isotopp/@HBeckPDX/@weiglemc for the info and @kcgreenn/@dreamjar for the comic!)

Posted by jpluimers on 2022/11/12

This is fine #Twitter

This is fine (illustration inspired by KC Green; creation video below)

(Edit 20221114: script for high-res images; more tweets from Jan) (Edit 20221116: hat-tip to Sam) (Edit 20221120: archiving t.co links by Michele Weigle) (Edit 20221122: added article by Johan van der Knijff) (20221128 Tapue export tool by Mike Hucka)

Time to be prepared:

The below will help you exporting your Twitter content (Tweets, DMs, media), perform some conversions on them and optionally delete (parts of) your content.

Important: keep your Twitter account afterwards (to prevent someone from creating a new account with the same handle).

Read the rest of this entry »

Posted in *nix, *nix-tools, Awk, Development, Lightweight markup language, MarkDown, Mastodon, Power User, SocialMedia, Software Development, Twitter | Leave a Comment »