The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,854 other subscribers

Archive for the ‘Scripting’ Category

Getting your public IP address from the command-line when http and https are blocked: use DNS

Posted by jpluimers on 2022/12/28

Years ago, I wrote Getting your public IP address from the command-line. All methods were http based, so were very easy to execute using cURL.

But then in autumn 2021, Chris Bensen wrote this cool little blog-post [Wayback/Archive] Chris Bensen: How do I find my router’s public IP Address from the command line?:

dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com

At first sight, I thought it was uncool, as the command was quite long and there was no explanation of the dig command trick.

But then, knowing that dig is a DNS client, it occurred to me: this perfectly works when http and https are disabled by your firewall, but the DNS protocol works and gives the correct result:

# dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com
"80.100.143.119"

This added the below commands and aliases to my tool chest for *nix based environments like Linux and MacOS (not sure yet about Windows yet :), but that still doesn’t explain why it worked. So I did some digging…

IPv4

  • command:
    dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com
  • command removing outer double quotes:
    dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs
  • alias:
    alias "whatismyipv4_dns=dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs"

IPv6

  • command:
    dig -6 TXT +short o-o.myaddr.l.google.com @ns1.google.com
  • command removing outer double quotes:
    dig -6 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs
  • alias:
    alias "whatismyipv6_dns=dig -6 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs"

How it works

Let’s stick to dig and IPv4 as that not having IPv6 (regrettably still) is the most common situation today:

# dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com
"80.100.143.119"

What it does is request the DNS TXT record of o-o.myaddr.l.google.com from the Google DNS server ns1.google.com and returns the WAN IPv4 address used in the DNS request, which is for instance explained in [Wayback/Archive] What is the mechanics behind “dig TXT o-o.myaddr.l.google.com @ns1.google.com” : linuxadmin.

Since these are TXT records, dig will automatically double quote them, which xargs can remove (see below how and why):

# dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com | xargs
80.100.143.119

The DNS query will fail when requesting the Google Public DNS servers 8.8.8.8 or 8.8.4.4:

# dig -4 TXT +short o-o.myaddr.l.google.com @8.8.8.8
"2a00:1450:4013:c1a::103"
"edns0-client-subnet 80.101.239.0/24"

Or, with quotes removed (the -L 1 ensures that xargs performs the quote-pair removal action on each line):

# dig -4 TXT +short o-o.myaddr.l.google.com @8.8.8.8 | xargs -L 1
2a00:1450:4013:c1a::103
edns0-client-subnet 80.101.239.0/24

This request is both slower than requesting the ns1.google.com server and wrong.

The reason is that only ns1.google.com understands the special o-o.myaddr.l.google.com hostname which instructs it to return the IP address of the requesting dig DNS client.

That 8.8.8.8 returns a different IP address and an additional edns0-client-subnet with less accurate information is explained in an answer to [Wayback/Archive] linux – Getting the WAN IP: difference between HTTP and DNS – Stack Overflow by [Wayback/Archive] argaz referring to this cool post: [Wayback/Archive] Which CDNs support edns-client-subnet? – CDN Planet.

Not just ns1.google.com: any DNS server serving the google.com domain

Since o-o.myaddr.l.google.com is part of the google.com domain, the above works for any DNS server serving the google.com domain (more on that domain: [Wayback/Archive] General DNS overview  |  Google Cloud).

Getting the list of DNS servers is similar to getting the list of MX servers which I explained in Getting the IP addresses of gmail MX servers, replacing MX record type (main exchange) with the NS record type (name server) and the gmail.com domain with the google.com domain:

# dig @8.8.8.8 +short NS google.com
ns3.google.com.
ns1.google.com.
ns2.google.com.
ns4.google.com.

The ns1.google.com DNS server is a special one of the NS servers: it is the start of authority server, which you can query using the SOA record type that also gives slightly more details for this server:

# dig @8.8.8.8 +short SOA google.com
ns1.google.com. dns-admin.google.com. 410477869 900 900 1800 60

The difference between using NS and SOA records with dig are explained in the [Wayback] dns – How do I find the authoritative name-server for a domain name? – Stack Overflow answer by [Wayback/Archive] bortzmeyer who also explains how to help figuring out SOA and NS discrepancies (note to self: check out the check_soa tool originally by Michael Fuhr (I could not find recent content of him, so he might have passed away) of which source code is now at [Wayback/Archive] Net-DNS/check_soa at master · NLnetLabs/Net-DNS).

So this works splendid as well using ns4.google.com on my test system:

# dig -4 TXT +short o-o.myaddr.l.google.com @ns4.google.com | xargs
80.100.143.119

The xargs removes outer quotes removal trick

[Wayback/Archive] string – Shell script – remove first and last quote (“) from a variable – Stack Overflow (thanks quite anonymous [Wayback/Archive] user1587520):

> echo '"quoted"' | xargs
quoted

xargs uses echo as the default command if no command is provided and strips quotes from the input.

More on https versus DNS requests

Some notes are in [Wayback/Archive] How to get public IP address from Linux shell, but note the telnet trick now fails as myip.gelma.net is gone (latest live version was archived in the Wayback Machine in august 2019).

Via

–jeroen

Posted in *nix, *nix-tools, Apple, bash, bash, Batch-Files, Communications Development, Development, DNS, Internet protocol suite, Linux, Mac, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development, TCP | Leave a Comment »

Kevlin Henney on Twitter: “#FizzBuzzFriday… ” Python table lookup versus C# functional programming

Posted by jpluimers on 2022/12/27

[Archive.is] Kevlin Henney on Twitter: “#FizzBuzzFriday… “.

Related: [Wayback] Your C# is already functional, but only if you let it | In Absentia:

A few days ago I tweeted a C# code snippet, showing a FizzBuzz implementation using some of the new features in C# 8.0. The tweet “went viral”, as the kids say, with several people admiring the terse

–jeroen

Read the rest of this entry »

Posted in .NET, C#, Development, Functional Programming, Python, Software Development | Leave a Comment »

SimenHolmestad/Privacynator: A utility for removing Personally Identifiable Information (PII) from videos.

Posted by jpluimers on 2022/12/21

[Archive] SimenHolmestad/Privacynator: A utility for removing Personally Identifiable Information (PII) from videos.

Privacynator is an utility for removing Personally Identifiable Information (PII) from videos. It is primarily meant for videos captured from cars.
Privacynator uses the Detectron2 model from facebook research: https://github.com/facebookresearch/detectron2.

It is written in Python 3.

It is unlike this: Need to check if “PrivacyNator” is already out: a local TensorFlowJS app that blurs your screen when you are not behind it

–jeroen

Posted in Development, Python, Scripting, Software Development | Leave a Comment »

Kristian Köhntopp on Twitter: “Basically, show me a Python regex with \d and without ASCII flag, and I can show you a bug, often exploitable.… “

Posted by jpluimers on 2022/12/14

An interesting thought: [Archive] Kristian Köhntopp on Twitter: “Basically, show me a Python regex with \d and without ASCII flag, and I can show you a bug, often exploitable.… “

Basically, input parsing is still very much underrated by most systems and a constant source of peculiarities and therefore bugs, or phrased differently: [Archive] Kristian Köhntopp on Twitter: “In many cases an uncaught exception, and hence a component crash.… “

Kris also states [Archive] Kristian Köhntopp on Twitter: “Again, Python is not alone in this. Perl, when “use utf8;” is active (which it should) also does this, so every single fucking Regex needs a ‘/a‘ at the end. Nobody ever asked \d to match tengwar or klingon numeric symbols.… “.

The point is in the last few words as Arabic numerals are so white spread over the world that the ten digits 0, 1, 2, 3, 4, 5, 6, 7, 8 , 9 they represent should be the de facto \d pattern, but aren’t in Python as per [Wayback/Archive] re — Regular expression operations — Python 3.10.0 documentation: /d (emphasis mine):

Read the rest of this entry »

Posted in Development, Perl, Python, RegEx, Scripting, Software Development | Leave a Comment »

google chrome devtools – Use JavaScript to set value of textbox when .value and events don’t seem to work – Stack Overflow

Posted by jpluimers on 2022/11/29

For my link archive: [Wayback/Archive] google chrome devtools – Use JavaScript to set value of textbox when .value and events don’t seem to work – Stack Overflow

TL;DR

Sometimes fields are blocked from pasting values.

Normally a trick like this works in the chrome development panel console:

document.getElementById('nonPasteElementID').value = 'myValueFromTheClipboard'

With some web development environments this does not work.

For react, after finding the react render name for the input (in the case of the answer, it was “reactNameForInputElement“) this is a solution:

To make it work you will need this:

const input = document.getElementById('nonPasteElementID');
const event = new Event('reactNameForInputElement', { bubbles: true });
input.value = '1';
input.dispatchEvent(event);

–jeroen

Posted in Chrome, Chrome, Development, Google, JavaScript/ECMAScript, Power User, Scripting, Software Development, Web Browsers, Web Development | Leave a Comment »

A while ago I bumped into some GPI Mojibake examples, but soon found out I should use the ftfy test cases

Posted by jpluimers on 2022/11/22

I have been into more and more Mojibake example pages like [Wayback] Mojibake: Question Marks, Strange Characters and Other Issues | GPI

Have you ever found strange characters like these ���  when viewing content in applications or websites in other languages?

They made me realise that all these (including the Mojibake examples on my blog) are just artifacts, but the real list of examples is the set of ftfy test cases at [Wayback/Archive.is] python-ftfy/test_cases.json at master · LuminosoInsight/python-ftfy

I got reminded when Waternet moved from paper mail using “Pyreneeën” to email using “Pyreneeën“. Not as bad as Waterschap AGV did earlier: they took it one level further and made “Pyreneeën” out of it, see Last year, a classic Mojibake was introduced when Waterschap Amstel, Gooi en Vecht redesigned their IT systems.

This seems like a trend where newer systems perform worse than older systems. I wonder why that is.

BTW: the trick on the [Wayback/Archive] Python.org shell to run ftfy (which is not installed by default) is first dropping to the shell (see my post How do I drop a bash shell from within Python? – Stack Overflow), then starting python again:

Read the rest of this entry »

Posted in CP850, Development, Encoding, ftfy, ISO-8859, Mojibake, Python, Scripting, Software Development, Unicode, UTF-8, UTF8 | Leave a Comment »

Exporting your Twitter content, converting to Markdown and getting the image alt-texts (thanks @isotopp/@HBeckPDX/@weiglemc for the info and @kcgreenn/@dreamjar for the comic!)

Posted by jpluimers on 2022/11/12

This is fine #Twitter

This is fine #Twitter (illustration inspired by KC Green; creation video below)

(Edit 20221114: script for high-res images; more tweets from Jan) (Edit 20221116: hat-tip to Sam) (Edit 20221120: archiving t.co links by Michele Weigle) (Edit 20221122: added article by Johan van der Knijff) (20221128 Tapue export tool by Mike Hucka)

Time to be prepared:

The below will help you exporting your Twitter content (Tweets, DMs, media), perform some conversions on them and optionally delete (parts of) your content.

Important: keep your Twitter account afterwards (to prevent someone from creating a new account with the same handle).

Read the rest of this entry »

Posted in *nix, *nix-tools, Awk, Development, Lightweight markup language, MarkDown, Mastodon, Power User, SocialMedia, Software Development, Twitter | Leave a Comment »

Database fiddle sites

Posted by jpluimers on 2022/10/27

I knew there was JSFiddle for live playing around with JavaScript and more in your browser, so I wondered if there was a similar site for databases and SQL queries.

There are, so here are a few database fiddle sites: SQL playgrounds where you can live play with SQL queries (sometimes even without an underlying example database).

All via [Wayback/Archive.is] database fiddle – Google Search:

Read the rest of this entry »

Posted in Conference Topics, Conferences, Database Development, DB2, Development, Event, Firebird, JavaScript/ECMAScript, JSFiddle, MariaDB, MySQL, OracleDB, PL/SQL, PostgreSQL, Scripting, Software Development, SQL, SQL Server, SQLite, T-SQL | Leave a Comment »

PowerShell: working around Get-NetFirewallRule not showing all the fields that Set-NetFirewallRule allows you to set

Posted by jpluimers on 2022/10/26

With APIs, you always hope that Get and Set methods mirror each other. More often than not, they don’t.

Take for instance these two:

They are far from symmetric: [Wayback/Archive] Get-NetFirewallRule shows far less than [Wayback/Archive] Set-NetFirewallRule allows you to set (first and foremost the various port related properties). It can be worked around though.

There are a few posts discussing this, of which I think these two are the most important:

Both above posts via [Wayback/Archive] “Get-NetFirewallRule” “LocalPort” – Google Search.

This is what I was after:

PowerShell "Get-NetFirewallRule -Name 'RemoteDesktop-UserMode-In-TCP' | Select-Object Name,DisplayName,Enabled,Direction,@{Name='Protocol';Expression={($PSItem | Get-NetFirewallPortFilter).Protocol}},Action,@{Name='LocalPort';Expression={($PSItem | Get-NetFirewallPortFilter).LocalPort}}"

Or actually:

Read the rest of this entry »

Posted in CommandLine, Development, Power User, PowerShell, PowerShell, Remote Desktop Protocol/MSTSC/Terminal Services, Scripting, Software Development, Windows | Leave a Comment »

On the research list: ahochsteger/gmail2gdrive: Gmail2GDrive is a Google Apps Script which automatically stores and sorts Gmail attachments into Google Drive folders.

Posted by jpluimers on 2022/10/13

On the research list: [Wayback/Archive.is] ahochsteger/gmail2gdrive: Gmail2GDrive is a Google Apps Script which automatically stores and sorts Gmail attachments into Google Drive folders.

The script is in JavaScript and runs on Archive.is Apps Script – Google Apps Script.

Because of [Wayback/Archive.is] gmail – Way to automatically download or print attachments or URLs from emails with specific labels – Web Applications Stack Exchange.

–jeroen

Posted in Development, GMail, Google, JavaScript/ECMAScript, Power User, Scripting, Software Development | Leave a Comment »