The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My work

  • My badges

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming

    20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now

    20140424-Windows-7-free-disk-space

    More Photos
  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,724 other followers

Archive for the ‘bash’ Category

sed: convert Google Drive urls to direct download ones

Posted by jpluimers on 2017/03/14

RegEx Fu

RegEx Fu

One of the things after moving most of my things from copy.com to Google Drive was the direct (public) download URLs that copy.com provides. DropBox has them as well, but Google Drive lacks them in the UI.

There is a URL format that does allow for direct download though:

While Google aims for Drive to be a competent Dropbox competitor, there’s one small but key feature that isn’t easy: sharing direct download links. Fortunately, you can create your own.

Source: Share Direct Links to Files in Google Drive and Skip the Web Viewer

You can do a similar replacement for Google Doc URLs: How to Create Direct Download Links for Files on Google Drive

The Google Drive conversion seems straightforward as they convert from either of

https://drive.google.com/file/d/FILE_ID/edit?usp=sharing
https://drive.google.com/file/d/FILE_ID/view
https://drive.google.com/open?id=FILE_ID

to

https://drive.google.com/uc?export=download&id=FILE_ID

There are tons of RegEx examples for doing the first conversion at Regex to modify Google Drive shared file URL – Stack Overflow, but

  1. they don’t cover the two conversions
  2. they use the non-greedy (.*?) capturing groups which are tricky, introduce question mark escaping issues in hash and many sed implementations fail to implement non-greedy

Since I’m a command-line person, I’ve opted for a sed conversion that wasn’t in the above list. I choose sed because it allows you to convert either a line or a complete file at one time.

There are a few indispensable resources to get my regex expressions right:

So here it goes, starting with fixing https://drive.google.com/open?id=FILE_ID as it’s the most simple replacement because the FILE_ID is at the end.

First of all, these code fragments below are part of bash functions as bash functions remove the quoting hell you have with bash aliases.

Where bash aliases have no parameters (i.e. the arguments are put after the end of the expansion), functions have parameters. So if you want to pass all function parameters to a command inside a function, you have to use “$@” to pass all parameters.

This fragment fixes https://drive.google.com/open?id=FILE_ID printing each fix on one line using the p for printing command in sed:

sed -n 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@p' "$@"

A few remarks:

The second fragment fixes https://drive.google.com/file/d/FILE_ID/edit?usp=sharing and https://drive.google.com/file/d/FILE_ID/view again printing each fix:

sed -n 's@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@p' "$@"

Some more remarks:

  • The FILE_ID is obtained from a capturing group during the match using \([^.]*\) and using the value in the replace with \1 as reference.
  • There is backslash escaping of the parentheses because that’s the sed way.
  • I’ve used a non-greedy \(.*?\) capturing group (sed can’t do that) but \([^.]*\)/ which matches any non-slash inside the capturing group until the first slash outside that group.

The final part is combing both replacement into one sed command:

sed 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@;s@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@' "$@"

Final remarks:

–jeroen

Posted in Software Development, Development, Power User, *nix, Scripting, bash, sed script, *nix-tools, bash, sed | Leave a Comment »

ShellCheck – shell script analysis tool

Posted by jpluimers on 2017/03/09

This is golden: ShellCheck – shell script analysis tool.

It checks your shell scripts, either on-line or off-line (brew install shellcheck for Mac, apt, etc for Linuces)

–jeroen

via: regex – Read file line by line with bash script – Stack Overflow

Posted in bash, Development, Scripting, Sh Shell, Software Development | Leave a Comment »

bash – how do I list the functions defined in my shell? – Stack Overflow

Posted by jpluimers on 2017/03/07

Nice: “you can get both aliases and functions with compgen -a -A function

It uses the compgen completion generator. Simply brilliant (:

Source: bash – how do I list the functions defined in my shell? – Stack Overflow

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

Determining the current shell in *n*x variants including ESXi

Posted by jpluimers on 2017/02/08

On most systems, I use bash as shell, but not all systems have it, for instance the shell.xs4all.nl server uses tcsh and ESXi 4+ uses a very limited ash shell from busybox (ESX 4 had bash though).

There is this huge script that covers many shell and operating system versions (even DOS, Windows) and interpreters (python, ruby, php, etc) what shell is this which I got through Stéphane Chazelas‘s answer in linux – determine shell in script during runtime – Unix & Linux Stack Exchange

I wanted a shorter thing that works in current Linux, BSD, OS X and ESXi versions.

Some very short scripts are less reliable, for instance echo $SHELL looks nice, but isn’t always set.

Similar for echo $0 which will fail for instance if it shows as sh but in fact is a different shell in disguise.

This works for bash, tcsh and busybox sh, is a bit more precise than getting $0. It’s based on HOWTO: Detect bash from shell script – Stack Overflow:

lsof -p $$ | awk '(NR==2) {print $1}'

But on ESXi it shows this because lsof doesn’t take any parameter there and just dumps all information:

----------+---------------------+---------------------+--------+------------------

It’s because lsof on ESXi always shows this header where Cartel and World aren’t exactly well documented:

Cartel | World name | Type | fd | Description
----------+---------------------+---------------------+--------+------------------

Empirically for non VM related processes, it looks like the Cartel is the PID and World name the command.

On Linux and BSD based systems, the header looks like this, so command and PID are reversed in ESXi:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

This command then works on both ESXi, OS X, Linux and BSD assuming you can word search for the PID and noting that PID/command will be reversed on ESXi as compared to OSX/Linux/BSD:

lsof -p $$ | grep -w $$ | awk '(NR==2) {print $1,$2}'

–jeroen

Posted in Apple, bash, BSD, Development, iMac, Mac, MacBook, MacBook Retina, MacBook-Air, MacBook-Pro, MacMini, OS X, OS X El Capitan, OS X Leopard, OS X Lion, OS X Maverick, OS X Mountain Lion, OS X Snow Leopard, OS X Tiger, OSX Yosemite, Power User, Scripting, Software Development | Leave a Comment »

pi-hole/pi-hole: A black hole for Internet advertisements (designed for Raspberry Pi)

Posted by jpluimers on 2017/01/31

pi-hole – A black hole for Internet advertisements (designed for Raspberry Pi)

Works on most Debian distributions as well. Hopefully on opensuse one day as well.

Source: pi-hole/pi-hole: A black hole for Internet advertisements (designed for Raspberry Pi)

Not exactly the nicest way of installing though:

curl -L install.pi-hole.net | bash

Source: Pi-Hole: A Black Hole For Internet Advertisements

Source: In the past year, a similar threat has begun to emerge on mobile devices:…

Note that any ad-block mechanism needs curation to white/black list some stuff. But: who does that and who watches the curators?

Source: In the past year, a similar threat has begun to emerge on mobile devices: So-called overlay malware that impersonates login pages from popular apps and… – Jeroen Wiert Pluimers – Google+

via:

Some more links for when you get this going:

Changing hostname

As all raspbian hosts advertise their hostname as raspberrypi it is confusing to set them apart, so I changed the hostname in these files:

/etc/hostname
/etc/hosts
/etc/wicd/wired-settings.conf
/etc/wicd/wireless-settings.conf

Then rebooted (probably could have done sudo /etc/init.d/hostname.sh) to force the new hostname to be used everywhere.

DHCP versus static IP

Note that pi-hole by default converts the DHCP assigned address on eth0 to a static one. This makes it harder to use pi-hole in these situations:

  • preparing a pi-hole on network A and deploying it on network B
  • using pi-hole on a DHCP based network where the DHCP server hands out fixed IP addresses based on MAC

To get going I:

  1. filed an issue Work with DHCP address instead of static IP address configuration. · Issue #629 · pi-hole/pi-hole
  2. plugged in the Edimax EW-7811Un 150Mbps 11n Wi-Fi USB Adapter  which appeared as wlan0 in the ifconfig list
  3. failed in getting wicd-curses to work: it would only detect half of the WiFi networks that iwlist wlan0 scan detects.
  4. used the steps at Setting WiFi up via the command line – Raspberry Pi Documentation to get WLAN going:
    1. perform sudo iwlist wlan0 scan | grep ESSID scan to get a list of networks and their (E)SSID names
    2. append the below fragment to the end of /etc/wpa_supplicant/wpa_supplicant.conf and correct the value for ssid to the ESSID (keep the double quotes around it) and psk to the password for that ESSID (also keep the double quotes around it)
    3. performed sudo ifdown wlan0  and sudo fup wlan0 to force a WiFi connection refresh
    4. waited 30 seconds for a DHCP address to appear in ifconfig for wlan0
network={
    ssid="The_ESSID_from_earlier"
    psk="Your_wifi_password"
}

 

 

–jeroen

Read the rest of this entry »

Posted in *nix, bash, Development, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux, Tumbleweed | Leave a Comment »

 
%d bloggers like this: