December 2025
M	T	W	T	F	S	S
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31

Archive for the ‘bash’ Category

Easy Running of Scripts at Boot and Shutdown – SUSE Blog | SUSE Communities

Posted by jpluimers on 2019/01/01

Cool:

/etc/init.d/after.local

–jeroen

Posted in *nix, *nix-tools, bash, Development, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux, Tumbleweed | Leave a Comment »

Find the TTL for a domain and subdomain by getting to the authoritative nameserver first

Posted by jpluimers on 2018/11/15

Lets find the authoritative name server and TTL (time to live) for the example.org domain and www.example.org subdomain.

Notes:

you can also do this using the web-page wrapper around dig [WayBack] Dig web interface

web-pages return most information in one go are [WayBack] intodns.com/example.org and [WayBack] dnsinspect.com: Report for example.org.

the steps below are partially based on a more in depth domain consistency check at [WayBack] Debugging nameservers using dig +norec

1a: get parents of name servers

First start with a root server (dig: getting the list of root servers) to get parents of the name servers for example.org (don’t you love indirection!):

$ dig +norecurse +noall +authority @f.root-servers.net. example.org.
org.            172800  IN  NS  a0.org.afilias-nst.info.
org.            172800  IN  NS  a2.org.afilias-nst.info.
org.            172800  IN  NS  b0.org.afilias-nst.org.
org.            172800  IN  NS  b2.org.afilias-nst.org.
org.            172800  IN  NS  c0.org.afilias-nst.info.
org.            172800  IN  NS  d0.org.afilias-nst.org.

You can repeat this query for 2 more root servers to ensure they are in sync.

1b: get authoritative name servers from the parents

Now repeat with at least 3 of these to ensure they give matching results for the name servers for example.org:

$ dig +norecurse +noall +authority @b0.org.afilias-nst.info. example.org.
example.org.        86400   IN  NS  b.iana-servers.net.
example.org.        86400   IN  NS  a.iana-servers.net.
$ dig +norecurse +noall +authority @c0.org.afilias-nst.info. example.org.
example.org.        86400   IN  NS  a.iana-servers.net.
example.org.        86400   IN  NS  b.iana-servers.net.
$ dig +norecurse +noall +authority @a0.org.afilias-nst.info. example.org.
example.org.        86400   IN  NS  a.iana-servers.net.
example.org.        86400   IN  NS  b.iana-servers.net.

2a: getting the domain name servers from a public name server

A query to a public DNS server will also return a name server list, but then you would need to know that name server first. In addition, you can not ask for +authority; you have to ask for +answer NS in stead:

$ dig +norecurse +noall +answer NS @8.8.8.8 example.org.
example.org.        55312   IN  NS  a.iana-servers.net.
example.org.        55312   IN  NS  b.iana-servers.net.

The name servers on the list are not guaranteed to be authoritative, as this query returns an empty result:

$ dig +norecurse +noall +authority @8.8.8.8 example.org.

2b. ensuring the name servers are authoritative name servers

From the name servers returned, you can now check if the servers themselves return the same name servers. If so, then you are sure they are authoritative:

$ dig +norecurse +noall +authority @a.iana-servers.net. example.org.
example.org.        86400   IN  NS  a.iana-servers.net.
example.org.        86400   IN  NS  b.iana-servers.net.
$ dig +norecurse +noall +authority @b.iana-servers.net. example.org.
example.org.        86400   IN  NS  b.iana-servers.net.
example.org.        86400   IN  NS  a.iana-servers.net.

3: get the actual TTL

With the authoritative name servers, you can get the actual TTL:

$ dig +norecurse +noall +answer SOA @a.iana-servers.net. example.org.
example.org.        3600    IN  SOA sns.dns.icann.org. noc.dns.icann.org. 2017042729 7200 3600 1209600 3600
$ dig +norecurse +noall +multiline +answer SOA @a.iana-servers.net. example.org.
example.org.        3600 IN SOA sns.dns.icann.org. noc.dns.icann.org. (
                2017042729 ; serial
                7200       ; refresh (2 hours)
                3600       ; retry (1 hour)
                1209600    ; expire (2 weeks)
                3600       ; minimum (1 hour)
                )

I got the +multiline trick from [WayBack] HOWTO: Using dig(1) to Find DNS Time to Live (TTL) Values – A-Team Systems.

4: get the count down TTL from a local name server

You can repeat the above process with a non-authoritative name server a few times to see the TTL decrease:

$ dig +norecurse +noall +answer SOA example.org.
example.org.        322 IN  SOA sns.dns.icann.org. noc.dns.icann.org. 2017042729 7200 3600 1209600 3600
$ dig +norecurse +noall +answer SOA example.org.
example.org.        321 IN  SOA sns.dns.icann.org. noc.dns.icann.org. 2017042729 7200 3600 1209600 3600

This is for instance what is returned by [WayBack] example.org DNS information – who.is example.org DNS information. DNS records such SOA, TTL, MX, TXT and more.

Public DNS servers having multiple servers per IP can even run disperse TTL numbers, for instance Google DNS at 8.8.8.8 does this:

$ dig +norecurse +noall +answer SOA @8.8.8.8 example.org.
example.org.        13  IN  SOA sns.dns.icann.org. noc.dns.icann.org. 2017042729 7200 3600 1209600 3600
$ dig +norecurse +noall +answer SOA @8.8.8.8 example.org.
example.org.        1388    IN  SOA sns.dns.icann.org. noc.dns.icann.org. 2017042729 7200 3600 1209600 3600
$ dig +norecurse +noall +answer SOA @8.8.8.8 example.org.
example.org.        10  IN  SOA sns.dns.icann.org. noc.dns.icann.org. 2017042729 7200 3600 1209600 3600

Note that `+nssearch` does not work for me

Using +nssearch as per [WayBack] How to find what Authoritative Name Server provided the answer using dig? – Server Fault fails for me:

$ dig +nssearch example.org
SOA sns.dns.icann.org. noc.dns.icann.org. 2017042729 7200 3600 1209600 3600 from server 199.43.135.53 in 83 ms.
SOA sns.dns.icann.org. noc.dns.icann.org. 2017042729 7200 3600 1209600 3600 from server 199.43.133.53 in 144 ms.
;; connection timed out; no servers could be reached

This reveals this in the bold values:

The authoritative nameserver sns.dns.icann.org is not publicly accessible.
TTL 7200 (7200 seconds is 2 hours).

Future research

Authoritative answers might not be present in dig queries on some platforms. I need to dig deeper into [WayBack] terminal – Dig not returning authority section? – Ask Different to see why.

Glue records are always tricky to get right: [WayBack] How to check domain NS glue records using dig « Admins eHow

–jeroen

Posted in *nix, *nix-tools, bash, Development, dig, DNS, Internet, Power User, Scripting, Software Development | Leave a Comment »

How to Setup Chroot SFTP in Linux (Allow Only SFTP, not SSH)

Posted by jpluimers on 2018/11/07

I need to script this one day: [WayBack] How to Setup Chroot SFTP in Linux (Allow Only SFTP, not SSH)

–jeroen

Posted in *nix, Awk, bash, Communications Development, Development, Internet protocol suite, Power User, Scripting, Software Development, SSH, TCP | Leave a Comment »

aha (Ansi HTML Adapter) with clickable URIs

Posted by jpluimers on 2018/10/02

aha is great to generate HTML from ANSI text (i.e. the coloured output on a Linux console).

But it doesn’t generate clickable URIs (it can’t yet by itself as it only looks one character in the future).

The thread at https://github.com/theZiz/aha/issues/20 suggested a case-insensitive regex through sed but the exact suggestion failed for a few reasons I will explain below.

First the bash alias (requires both aha and perl):

	#!/usr/bin/env bash
	# based on https://github.com/theZiz/aha/issues/20#event-797466520
	aha-with-expanded-http-https-urls()
	{
	aha \| perl -C -Mutf8 -pe 's,([^"])((https?\|s?ftp\|ftps?\|file)://.*?)([\s]\|\"\s),$1<a href="$2">$2</a>$4,gi'
	}

view raw

testssh.sh-through-aha-with-expanded-http(s)-urls.bash

hosted with ❤ by GitHub

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, Perl, Power User, RegEx, Scripting, Software Development | Leave a Comment »

pure-bash-bible/README.md – book for doing things in bash without external tools

Posted by jpluimers on 2018/09/21

[WayBack] pure-bash-bible/README.md at master · dylanaraps/pure-bash-bible · GitHub:

The goal of this book is to document known and unknown methods of doing various tasks using only built-in bash features. Using the snippets from this bible can help remove unneeded dependencies from scripts and in most cases make them faster. I came across these tips and discovered a few while developing neofetch, pxltrm and other smaller projects.

The snippets below are linted using shellcheck and tests have been written where applicable. Want to contribute? Read the CONTRIBUTING.md. It outlines how the unit tests work and what is required when adding snippets to the bible.

See something incorrectly described, buggy or outright wrong? Open an issue or send a pull request. If the bible is missing something, open an issue and a solution will be found.

Via:

—jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

I am on a Mac that is bound to an AD Domain. The AD Domain has a 90 days pass…

Posted by jpluimers on 2018/08/27

[WayBack] I am on a Mac that is bound to an AD Domain. The AD Domain has a 90 days password expiration policy. When will my password expire? $ cat bin/is-passwor… – Kristian Köhntopp – Google+

For my link archive:

	#! /bin/bash —

	validFor=$(( 90 * 86400 ))
	domainPrefix='/Active Directory/DOMAIN/doma.in/"

	lastPW=$(dscl "/$domainPrefix" -read /Users/$USER \| awk '/SMBPasswordLastSet/ { print $NF }')
	unixPW=$(($lastPW / 10000000 – 11644473600 + $validFor))

	expireDate=$(date -r $unixPW)
	echo "Password expires $expireDate"

view raw

is-password-good.sh

hosted with ❤ by GitHub

Script copied to [WayBack] Kristian Köhntopp: I am on a Mac that is bound to an AD Domain. The AD Domain has a 90 days password expiration policy. When will my password expire? · GitHub

–jeroen

Posted in *nix, Apple, bash, Development, Mac, MacBook, MacBook Retina, MacBook-Air, MacBook-Pro, MacMini, Power User, Scripting, Software Development | Leave a Comment »

bash: generating random passwords on Mac OS X / MacOS or what Apple calls their operating system by now

Posted by jpluimers on 2018/08/14

After some fiddling, I put this function in ~/bash_profile on my MacBook:

# https://www.cyberciti.biz/tips/linux-unix-bsd-openssh-server-best-practices.html
# pass # digits as first argument; defaults to 20
generate_password() {
  local l=$1
  [ "$l" == "" ] && l=20
  env LC_CTYPE=C LC_ALL=C tr -dc A-Za-z0-9_ < /dev/urandom | head -c ${l} | xargs
}

The fiddling was working around this error:

tr: Illegal byte sequence

The cause is that Mac OS is not Linux, where tr happily accepts any byte sequence from /dev/urandom: it requires the C locale.

Originally, I started with the below function from [WayBack] Top 20 OpenSSH Server Best Security Practices – nixCraft (thanks +Joe C Hecht) which I undid from mixed tab/spaces:

genpasswd() {
  local l=$1
  [ "$l" == "" ] && l=20
  tr -dc A-Za-z0-9_ < /dev/urandom | head -c ${l} | xargs
}

Via: [WayBack] Joe C. Hecht – Google+ who mentions:

I might add that if you do move the SSH port, try to keep it in within the first 1024 ports for a tad bit more protection on some systems.

Background reading on Apple hating binary input:

Background reading on the language setting:

–jeroen

Posted in bash, Development, Scripting, Software Development | 2 Comments »

Let’s stop copying C / fuzzy notepad

Posted by jpluimers on 2017/12/07

Ah, C. The best lingua franca we have… because we have no other lingua francas. Linguae franca. Surgeons general? C is fairly old — 44 years, now! — and comes from a time when there were possibly more architectures than programming languages. It works well for what it is, and what it is is a relatively simple layer of indirection atop assembly. Alas, the popularity of C has led to a number of programming languages’ taking significant cues from its design, and parts of its design are… slightly questionable. I’ve gone through some common features that probably should’ve stayed in C and my justification for saying so. The features are listed in rough order from (I hope) least to most controversial. The idea is that C fans will give up when I call it “weakly typed” and not even get to the part where I rag on braces. Wait, crap, I gave it away.

Great re-read towards the end of the year: [WayBack] Let’s stop copying C / fuzzy notepad

Via: [WayBack] Old and busted: emacs vs vi. New and hot: Language war, everybody against everybody else. – Kristian Köhntopp – Google+

–jeroen

Posted in .NET, APL, Awk, bash, BASIC, C, C#, C++, COBOL, CoffeeScript, CommandLine, D, Delphi, Development, F#, Fortran, Go (golang), Java, Java Platform, JavaScript/ECMAScript, Pascal, Perl, PHP, PowerShell, PowerShell, Python, Ruby, Scala, Scripting, Software Development, TypeScript, VB.NET, VBScript | 3 Comments »

bash: `printf` supports `\e` just like `echo -e` does

Posted by jpluimers on 2017/11/07

Learned a few things when modifying https://github.com/gkotian/gautam_linux/blob/master/scripts/colours.sh

Note: `printf` supports emitting `ESC` (ASCII character `\033` aka `27` aka `0x1B`)as `\e` the same way that `echo` does

https://linux.die.net/man/1/printf
https://linux.die.net/man/1/echo

Format strings are at https://linux.die.net/man/3/printf
%-10s means left adjusted (aligned) string of length 10

–jeroen

via:

I was investigating how the colour definitions on my OpenSuSE system actually work internally so I added some extra output: ${TYPE} and ${COLOUR}.

Source: Show type and colour definition in addition to the rendered colour. by jpluimers · Pull Request #5 · gkotian/gautam_linux

Posted in *nix, *nix-tools, bash, bash, Development, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux | Leave a Comment »

cURL – POST an XML file as a stream

Posted by jpluimers on 2017/10/25

I hope I’m not alone on this but I find the cURL documentation hard to follow and short on examples.

My goal was to mimic some HTTP XML posting traffic a server gets from IoT devices. Google Chrome Postman (or Postman REST Client) reproduction is very easy and will send.

TL;DR

ensure you have an empty --header "Content-Type:" header: this ensures that cURL doesn’t add one and does not mess on how the content is being transferred.

use the --data or --data-binary command with an @ to post a file as body.

if you want --write-out then be sure you have a recent cURL version.

This is how the IoT or Postman will send.

Post headers like these:

Host:127.0.0.1:8080 Content-Length: 245 Connection:Keep-Alive

Content like this:

	<?xml version="1.0"?>
	<Root Attribute="value">
	<Branch>
	<Leaf>content</Leaf>
	</Branch>
	<Branch Attribute="value">
	<Bough Attribute="value">
	<Twig Attribute="value">
	<Leaf Attribute="value"/>
	</Twig>
	</Bough>
	</Branch>
	</Root>

view raw

httpPostSample.xml

hosted with ❤ by GitHub

The data is being streamed to the HTTP server even with the very limited set of headers.

I’ve been unable to come up with exact cURL statement that exactly matches the headers and way the content is being transferred.

This is what I tried (in all examples, %1 is the IPv4 address of the HTTP 1.1 server):

POST with the all the headers and the --data command:

curl --request POST --header "Host: %1:8080" --header "Content-Length: 245" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

This will hang the connection: somehow cURL will never notify the upload is done and the HTTP server keeps waiting. When you put --verbose or --trace-ascii - on the command-line you will see something like this before hanging: * upload completely sent off: 245 out of 245 bytes.

Note the trick to emit the ASCII trace to stdout using --trace-ascii with the minus sign: thanks to [WayBack] Daniel Stenberg for answering [WayBack] How can I see the request headers made by curl when sending a request to the server? – Stack Overflow.

You can do the same with --trace which dumps all characters (not only ASCII) including their HEX representation

POST with the all but the Content-Length headers and the --data command:

curl --request POST --header "Host: %1:8080" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

This will automatically add a Content-Length: 245 header and complete the transfer. But it will also add a Content-Type: application/x-www-form-urlencoded header causing the content not being posted as a body.

POST with a --form file= command:

curl --request POST --header "Host: %1:8080" --header "Connection: Keep-Alive" --form file=@httpPostSample.xml http://%1:8080/target

This will automatically ad a Content-Length: xxx header (way longer than 245) because it converts the request into a Content-Type: multipart/form-data; boundary=------------------------e1c0d47bac806954 one (the hex at the end differs) which is totally unlike what Postman does.

It is also unlike to what the HTTP server accepts.

POST with the all but the Content-Length headers and the --data-binary or --data-ascii command as suggested on Stack Overflow :

curl --request POST --header "Host: %1:8080" --header "Connection: Keep-Alive" --data-binary @httpPostSample.xml http://%1:8080/target

curl –request POST –header “Host: %1:8080” –header “Connection: Keep-Alive” –data-binary @httpPostSample.xml http://%1:8080/target

It turns out that --data-ascii is exactly the same as --data and that --data-binary just skips some new-line conversion when compared to --data or --data-ascii. Contrary to the --data-raw documentation that suggest it is equivalent to --data-binary it seems --data-raw behaves exactly like --data and --data-ascii. Odd.

So these are all stuck with the Content-Type: application/x-www-form-urlencoded and I thought I was running out of options.

Then I found [WayBack] soundmonster had posted an answer at [WayBack] http – What is the cURL command-line syntax to do a POST request? – Super User mentioning to add a Content-Type header.

So I changed the request to include the --header "Content-Type: text/xml; charset=UTF-8" header:

curl --request POST --header "Content-Type: text/xml; charset=UTF-8" --header "Host: %1:8080" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

This works. But: the Content-Type header is not present in the original request.

Finally it occurred to me: What if cURL would not insert a Content-Type header if I add an empty Content-Type header?.

That works!

curl --request POST --header "Content-Type:" --header "Host: %1:8080" --header "Connection: Keep-Alive" --data @httpPostSample.xml http://%1:8080/target

It posts exactly the same content as the IoT devices and Postman do.

Phew!

I tried to combine this with the --write-out (a.k.a. -w) option, but for older versions of cURL (I could reproduce with 7.34) that forces cURL back in to Content-Type: application/x-www-form-urlencoded mode so watch your cURL version!

Later I will put more research in chuncked transfer. Links that might help me:

–jeroen

Some of the references:

Posted in *nix, bash, cURL, Development, Encoding, Power User, Scripting, Software Development | Leave a Comment »

« Previous Entries

Next Entries »

	Lars Fosdal on Security alarm provider Woonve…
	Thomas Mueller on Question got closed in May 202…
	Thaddy de Koning on Formulier voor bewindvoerders…
	Thaddy de Koning on Formulier voor bewindvoerders…
	Thaddy de Koning on Formulier voor bewindvoerders…

The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

Twitter Updates

My Flickr Stream

Pages

All categories

Email Subscription

Archive for the ‘bash’ Category

Easy Running of Scripts at Boot and Shutdown – SUSE Blog | SUSE Communities

Find the TTL for a domain and subdomain by getting to the authoritative nameserver first

1a: get parents of name servers

1b: get authoritative name servers from the parents

2a: getting the domain name servers from a public name server

2b. ensuring the name servers are authoritative name servers

3: get the actual TTL

4: get the count down TTL from a local name server

Note that `+nssearch` does not work for me

Future research

How to Setup Chroot SFTP in Linux (Allow Only SFTP, not SSH)

aha (Ansi HTML Adapter) with clickable URIs

pure-bash-bible/README.md – book for doing things in bash without external tools

I am on a Mac that is bound to an AD Domain. The AD Domain has a 90 days pass…

bash: generating random passwords on Mac OS X / MacOS or what Apple calls their operating system by now

Let’s stop copying C / fuzzy notepad

bash: `printf` supports `\e` just like `echo -e` does

cURL – POST an XML file as a stream

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

My Flickr Stream

Pages

All categories

Email Subscription

Archive for the ‘bash’ Category

Rate this:

Share this:

1a: get parents of name servers

1b: get authoritative name servers from the parents

2a: getting the domain name servers from a public name server

2b. ensuring the name servers are authoritative name servers

3: get the actual TTL

4: get the count down TTL from a local name server

Note that +nssearch does not work for me

Future research

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Note that `+nssearch` does not work for me