The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,854 other subscribers

Archive for the ‘*nix-tools’ Category

Getting your local IPv4 addresses, netmasks and CIDRs

Posted by jpluimers on 2017/04/11

In order to scan some local networks for unknown hosts (yes, on some sites you need to perform archeology), I needed the local IPv4 addresses, netmasks and CIDRs on my Mac running OS X.

Part of that is using ifconfig to get local inet information which however uses hexadecimal network masks and delivers no CIDRs.

SoI was a bit premature when I wrote about “This could be done by creating bash functions mask2cdr and cdr2mask, but that’s a bit too convoluted right now” in Getting the IP addresses of gmail MX servers – via Super User – dig isn’t enough.

netmask to CIDR and CIDR to netmask conversion

I need mask2cdr now, so lets start with these two bash functions and their aliases:

Read the rest of this entry »

Posted in *nix, *nix-tools, Apple, bash, Development, Mac OS X / OS X / MacOS, OS X 10.10 Yosemite, OS X 10.11 El Capitan, OS X 10.9 Mavericks, Power User, Scripting, Software Development | Leave a Comment »

Starting a command with one more more spaces prevents it from going into the bash history…

Posted by jpluimers on 2017/04/07

Learned about the -www switch for ps: extra extra “unlimited” wide (a single -w gets you 132 columns).

ps auxwww

Actually, there is no need for www, ww just works:

ps auxww

–jeroen

Source: Starting a command with one more more spaces prevents it from going into the…

Posted in *nix, *nix-tools, Power User, ps | Leave a Comment »

Recursively convert WMA files to MP3 using ffmeg on a Mac OS X bash shell

Posted by jpluimers on 2017/04/05

A friend of mine made the mistake to capture some CDs using WMA files and throwing away the CDs. His old Nokia could play them, but not his new iOS and Android devices.

ffmeg and bash to the rescue:

find . -iname "*.wma" -execdir bash -c 'NAME="{}" && ffmpeg -y -i "$NAME" -ab 192k "${NAME/.wma/.mp3}" -map_metadata 0:s:0 && rm "$NAME"' \;

Tricks used:

Read the rest of this entry »

Posted in *nix, *nix-tools, Audio, bash, bash, Development, ffmpeg, Media, Power User, Scripting, Software Development | Leave a Comment »

ducks: How Do I Find The Largest Top 10 Files and Directories On a Linux / UNIX / BSD?

Posted by jpluimers on 2017/03/20

Explains how to find out top 10 files and directories under Unix / Linux using sort and du command in human-readable format.

Interesting, especially

alias 'ducks=du -cks * | sort -rn | head'

Source: How Do I Find The Largest Top 10 Files and Directories On a Linux / UNIX / BSD?

via: Joe C. Hecht and nixCraft.

–jeroen

Posted in *nix, *nix-tools, Apple, BSD, Linux, Mac, Mac OS X / OS X / MacOS, Power User | Leave a Comment »

sed: convert Google Drive urls to direct download ones

Posted by jpluimers on 2017/03/14

RegEx Fu

RegEx Fu

One of the things after moving most of my things from copy.com to Google Drive was the direct (public) download URLs that copy.com provides. DropBox has them as well, but Google Drive lacks them in the UI.

There is a URL format that does allow for direct download though:

While Google aims for Drive to be a competent Dropbox competitor, there’s one small but key feature that isn’t easy: sharing direct download links. Fortunately, you can create your own.

Source: Share Direct Links to Files in Google Drive and Skip the Web Viewer

You can do a similar replacement for Google Doc URLs: How to Create Direct Download Links for Files on Google Drive

The Google Drive conversion seems straightforward as they convert from either of

https://drive.google.com/file/d/FILE_ID/edit?usp=sharing
https://drive.google.com/file/d/FILE_ID/view
https://drive.google.com/open?id=FILE_ID

to

https://drive.google.com/uc?export=download&id=FILE_ID

There are tons of RegEx examples for doing the first conversion at Regex to modify Google Drive shared file URL – Stack Overflow, but

  1. they don’t cover the two conversions
  2. they use the non-greedy (.*?) capturing groups which are tricky, introduce question mark escaping issues in hash and many sed implementations fail to implement non-greedy

Since I’m a command-line person, I’ve opted for a sed conversion that wasn’t in the above list. I choose sed because it allows you to convert either a line or a complete file at one time.

There are a few indispensable resources to get my regex expressions right:

So here it goes, starting with fixing https://drive.google.com/open?id=FILE_ID as it’s the most simple replacement because the FILE_ID is at the end.

First of all, these code fragments below are part of bash functions as bash functions remove the quoting hell you have with bash aliases.

Where bash aliases have no parameters (i.e. the arguments are put after the end of the expansion), functions have parameters. So if you want to pass all function parameters to a command inside a function, you have to use “$@” to pass all parameters.

This fragment fixes https://drive.google.com/open?id=FILE_ID printing each fix on one line using the p for printing command in sed:

sed -n 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@p' "$@"

A few remarks:

The second fragment fixes https://drive.google.com/file/d/FILE_ID/edit?usp=sharing and https://drive.google.com/file/d/FILE_ID/view again printing each fix:

sed -n 's@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@p' "$@"

Some more remarks:

  • The FILE_ID is obtained from a capturing group during the match using \([^.]*\) and using the value in the replace with \1 as reference.
  • There is backslash escaping of the parentheses because that’s the sed way.
  • I’ve used a non-greedy \(.*?\) capturing group (sed can’t do that) but \([^.]*\)/ which matches any non-slash inside the capturing group until the first slash outside that group.

The final part is combing both replacement into one sed command:

sed 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@;s@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@' "$@"

Final remarks:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sed, sed script, Software Development | Leave a Comment »

Sniffers, Packet Capture – PFSenseDocs – cool, as it uses tcpdump/Wireshark format!

Posted by jpluimers on 2017/03/13

I hadn’t done a lot with pfSense in the past, which I regret a bit since I discovered this really cool feature: Sniffers, Packet Capture – PFSenseDocs.

The coolness isn’t so much that you can capture packets, but that it’s compatible with tcpdump and Wireshark (which has become available natively for Mac like 2 years ago).

Which means that you can download captures and open them in Wireshark.

So it’s as easy as 1,2,3:

  1. Set-up the capture on your router https://a.b.c.d/diag_packet_capture.php and start it
  2. Stop the capture and download the file
  3. Open the file in Wireshark or convert it to text using tshark

–jeroen

Posted in *nix, *nix-tools, Internet, Monitoring, pfSense, Power User, routers, tcpdump, Wireshark | Leave a Comment »

bash – how do I list the functions defined in my shell? – Stack Overflow

Posted by jpluimers on 2017/03/07

Nice: “you can get both aliases and functions with compgen -a -A function

It uses the compgen completion generator. Simply brilliant (:

Source: bash – how do I list the functions defined in my shell? – Stack Overflow

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

Generate Mozilla Security Recommended Web Server Configuration Files

Posted by jpluimers on 2017/03/06

In case you manually want to configure or have a web-server that’s not supported by certbot for letsencrypt (yet): Generate Mozilla Security Recommended Web Server Configuration Files.

At the time of writing, these were supported by the generator (* were not supported by certbot for letsencrypt yet):

–jeroen

via: Feature request: admin web interface over HTTPS · Issue #630 · pi-hole/pi-hole

Posted in *nix, *nix-tools, Apache2, Encryption, Let's Encrypt (letsencrypt/certbot), Power User, Security | Leave a Comment »

Getting the IP addresses of gmail MX servers – via Super User – dig isn’t enough

Posted by jpluimers on 2017/03/06

I needed the current IP-addresses of the gmail MX server (don’t ask the details; but it has to do with the brain-dead TP-LINK ER5120 configuration possibilities).

This is the command I finally used:

dig @8.8.8.8 +short MX gmail.com | sed "s/^[0-9]* //g" | sed "s/.$//" | xargs -I {} dig @8.8.8.8 +short {} | uniq | sort

Basically it’s a three stage sequence which had to work on OS X as well as Linux using a bash shell:

  1. Use the Google DNS servers (either 8.8.8.8 or 8.8.4.4)
  2. Get the FQDNs of MX records of gmail.com which are the mail servers for GMail.
  3. Translate these in IPv4 addresses
  4. Filter into a distinct list (just in case entries are duplicate: they aren’t yet, but might be)

The basics of the above are about using dig to get short (or terse) answers with as little (but still to the point) information as possible.
Read the rest of this entry »

Posted in *nix, *nix-tools, DNS, Power User | 1 Comment »

fixing a Mac home `brew update` that has permission errors (after that I could install plantuml)

Posted by jpluimers on 2017/01/26

I had this occurring on my system:

RetinaMBPro1TB:~ jeroenp$ brew update
error: unable to unlink old 'Library/ENV/pkgconfig/10.11/libcurl.pc' (Permission denied)
error: unable to unlink old 'Library/ENV/pkgconfig/10.11/libxml-2.0.pc' (Permission denied)
error: unable to unlink old 'Library/ENV/pkgconfig/10.11/sqlite3.pc' (Permission denied)
To restore the stashed changes to /usr/local run:
  'cd /usr/local && git stash pop'
Already up-to-date.

This is how I solved it:

RetinaMBPro1TB:~ jeroenp$ ls -al /usr/local | grep -w Library
drwxr-xr-x+ 11 jeroenp  admin   374 Mar  9 19:33 Library
RetinaMBPro1TB:~ jeroenp$ sudo chown -R $USER /usr/local/Library/
Password:
RetinaMBPro1TB:~ jeroenp$ brew update
To restore the stashed changes to /usr/local run:
  'cd /usr/local && git stash pop'
Updated Homebrew from d32996d to 638d755.
==> New Formulae
...
==> Updated Formulae
...
==> Renamed Formulae
...
==> Deleted Formulae
...
RetinaMBPro1TB:~ jeroenp$ 

The above solution is based on major python problems · Issue #48301 · Homebrew/homebrew

After that, I could install plantuml (which requires java, just so you know) so now I can create SVGs from it locally:

plantuml -tsvg PSO.network-diagram.PlantUML.txt

Note I had to edit the formula so it installs plantuml-8037 or higher (the git version back then installed plantuml-8031) as it fixed a namespace bug. Since plantuml releases often, be prepared to do some version fiddling.

–jeroen

Posted in *nix, *nix-tools, Apple, Development, Diagram, Home brew / homebrew, Java, Java Platform, Mac, Mac OS X / OS X / MacOS, Mac OS X 10.5 Leopard, Mac OS X 10.6 Snow Leopard, MacBook, MacBook Retina, MacBook-Air, MacBook-Pro, MacMini, OS X 10.10 Yosemite, OS X 10.8 Mountain Lion, OS X 10.9 Mavericks, PlantUML, Power User, Software Development, UML | Leave a Comment »