The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,862 other subscribers

Archive for the ‘bash’ Category

aha (Ansi HTML Adapter) with clickable URIs

Posted by jpluimers on 2018/10/02

aha is great to generate HTML from ANSI text (i.e. the coloured output on a Linux console).

But it doesn’t generate clickable URIs (it can’t yet by itself as it only looks one character in the future).

The thread at https://github.com/theZiz/aha/issues/20 suggested a case-insensitive regex through sed but the exact suggestion failed for a few reasons I will explain below.

First the bash alias (requires both aha and perl):


#!/usr/bin/env bash
# based on https://github.com/theZiz/aha/issues/20#event-797466520
aha-with-expanded-http-https-urls()
{
aha | perl -C -Mutf8 -pe 's,([^"])((https?|s?ftp|ftps?|file)://.*?)([\s]|\&quot;\s),$1<a href="$2">$2</a>$4,gi'
}

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, Perl, Power User, RegEx, Scripting, Software Development | Leave a Comment »

pure-bash-bible/README.md – book for doing things in bash without external tools

Posted by jpluimers on 2018/09/21

[WayBack] pure-bash-bible/README.md at master · dylanaraps/pure-bash-bible · GitHub:

The goal of this book is to document known and unknown methods of doing various tasks using only built-in bash features. Using the snippets from this bible can help remove unneeded dependencies from scripts and in most cases make them faster. I came across these tips and discovered a few while developing neofetchpxltrm and other smaller projects.

The snippets below are linted using shellcheck and tests have been written where applicable. Want to contribute? Read the CONTRIBUTING.md. It outlines how the unit tests work and what is required when adding snippets to the bible.

See something incorrectly described, buggy or outright wrong? Open an issue or send a pull request. If the bible is missing something, open an issue and a solution will be found.

Via:

jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

bash: `printf` supports `\e` just like `echo -e` does

Posted by jpluimers on 2017/11/07

Learned a few things when modifying https://github.com/gkotian/gautam_linux/blob/master/scripts/colours.sh

Note: `printf` supports emitting `ESC` (ASCII character `\033` aka `27` aka `0x1B`)as `\e` the same way that `echo` does

https://linux.die.net/man/1/printf
https://linux.die.net/man/1/echo

Format strings are at https://linux.die.net/man/3/printf
%-10s means left adjusted (aligned) string of length 10

–jeroen

via:

I was investigating how the colour definitions on my OpenSuSE system actually work internally so I added some extra output: ${TYPE} and ${COLOUR}.

Source: Show type and colour definition in addition to the rendered colour. by jpluimers · Pull Request #5 · gkotian/gautam_linux

Posted in *nix, *nix-tools, bash, bash, Development, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux | Leave a Comment »

nojhan/liquidprompt: A full-featured & carefully designed adaptive prompt for Bash & Zsh

Posted by jpluimers on 2017/10/05

Wow: nojhan/liquidprompt: A full-featured & carefully designed adaptive prompt for Bash & Zsh

This is really useful!

via:

Sort of tanslated from the first “via” (note that “mit Alles und Scharf” is hard to translate; it’s somewhere between “everything but the kitchen sink, but done right” and “right on the money”):

Bash Prompt Overkill: https://github.com/nojhan/liquidprompt is a Bash “Prompt doing it all right”-extension, which doesn’t care how much any feature costs as we have cores, gigabytes and SSD.

Liquid Prompt automagically recognises context and enables a plethora of features in the prompt when needed based on that context.

It’s like pixie dust for your prompt.

You can configure everything, but you don’t have to: the out of the box experience is already like pixie dust for your prompt.

It works on OS X too and is part of homebrew:

$ brew install liquidprompt
==> Using the sandbox
==> Downloading https://github.com/nojhan/liquidprompt/archive/v_1.11.tar.gz
==> Downloading from https://codeload.github.com/nojhan/liquidprompt/tar.gz/v_1.11
######################################################################## 100.0%
==> Caveats
Add the following lines to your bash or zsh config (e.g. ~/.bash_profile):
  if [ -f /usr/local/share/liquidprompt ]; then
    . /usr/local/share/liquidprompt
  fi
If you'd like to reconfigure options, you may do so in ~/.liquidpromptrc.
A sample file you may copy and modify has been installed to
  /usr/local/share/liquidpromptrc-dist
Don't modify the PROMPT_COMMAND variable elsewhere in your shell config;
that will break things.
==> Summary
🍺  /usr/local/Cellar/liquidprompt/1.11: 7 files, 125.6K, built in 3 seconds
[jeroenp:~/Versioned] 10s $

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, Apple, bash, bash, Development, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »

youtube-dl – saving both audio and video without keeping intermediate files seems impossible

Posted by jpluimers on 2017/09/14

Just in case someone has a better alternative than youtube-dl alias:

alias youtube-dl-audio-and-video='youtube-dl --keep-video --extract-audio --audio-quality 0 --audio-format mp3'

It extracts the audio and keeps the video.

The result is that also all intermediate downloads are being kept.

So even after studying the README extensively the only alternative seems to be a double download like this:

youtube-dl-audio-and-video() { youtube-dl --extract-audio --audio-quality 0 --audio-format mp3 $1; youtube-dl $1; }

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

OpenSuSE Tumbleweed – testing the password of any user with getent and openssl

Posted by jpluimers on 2017/06/21

For one of my VMs I forgot to note which of the initial password I had changed, so I wanted to check them.

Since I didn’t have a keyboard attached to the console and ssh wasn’t allowing root, I needed an alternative than actual login to test the passwords.

Luckily /etc/shadow, with getent and openssl came to the rescue.

Since getent varies per distribution, here is how it works on OpenSuSE:

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, bash, bash, Development, Encoding, Hashing, Linux, md5, openSuSE, Power User, Scripting, Security, SHA, SHA-256, SHA-512, Software Development, SuSE Linux | Leave a Comment »

opensuse – How to run my script after SuSE finished booting up? – Unix & Linux Stack Exchange

Posted by jpluimers on 2017/04/21

For future research: opensuse – How to run my script after SuSE finished booting up? – Unix & Linux Stack Exchange

Reason? Want to show the output of this as the last boot sequence line:

hostname
ip route
echo
ip address | grep -w 'UP\|flags\|inet\|inet6'
echo more detailed info through "ip address" and "ip route"
cat /etc/resolv.conf | grep nameserver

–jeroen

Posted in *nix, *nix-tools, bash, Linux, openSuSE, Power User, SuSE Linux, Tumbleweed | Leave a Comment »

Hosting Grumpydev Imageflair locally – part 2 – trying to get the text and images to display

Posted by jpluimers on 2017/04/12

Blank imageFlair

Blank imageFlair

Hosting Grumpydev Imageflair locally ended with two issues left: an empty image and my wish to include more complete StackExchange bits like the current StackExchange flair does.

I thought fixing the empty image would take a rainy day. It actually took a few rainy hours.

No text

The culprit is that I didn’t have the Microsoft Core Fonts for the Web installed. Which was no coincidence as the free download of those from Microsoft terminated in 2002. The upside is that because of their licenses, they are available as open source and most linux distributions have a script package that will download these fonts. OpenSuSE has fetchmsttfonts for this.

Alternatively, you can use the web.archive.org to download manually, but that’s a tad tedious. But if you love tedious: Free downloads – TrueType core fonts for the Web.

zypper install fetchmsttfonts

That installs the fonts in:

/usr/share/fonts/truetype

The drawback of having fetchmsttfonts is that the original Microsoft versions of these fonts are downloaded from corefonts.sourceforge.net each time the fetchmsttfonts package is updated, potentially overwriting newer versions of the fonts in that directory. If you don’t want that, use the trick at  (not yet archived at the WayBack machine) font handling – install fetchmsttfonts, copy fonts, rpm -e fethmsttfonts, copy fonts back.

Having the fonts installed, I thought the only thing I needed to fix were the multiple references in config.php from that pointed to Arial.TTF. I took the poor man’s approach and just did this being in the directory of config.php:

cp /usr/share/fonts/truetype/arial.ttf Arial.TTF

Filled Imageflair

That didn’t work either: still no text showed.

So I decided to run imageFlair.php from the command line after setting $imageflair_debug = true; in config.php which then resulted in all sorts of warnings like

PHP Warning: imagettftext(): Could not find/open font

After reading I decided to build a small php-gd.tester.php script containing phpinfo(); and gd_info showing these portions for PHP GD (non-relevant bits stripped):

Additional .ini files parsed /etc/php5/conf.d/ctype.ini,
/etc/php5/conf.d/dom.ini,
/etc/php5/conf.d/gd.ini,
/etc/php5/conf.d/iconv.ini,
/etc/php5/conf.d/json.ini,
/etc/php5/conf.d/mysql.ini,
/etc/php5/conf.d/mysqli.ini,
/etc/php5/conf.d/pdo.ini,
/etc/php5/conf.d/pdo_mysql.ini,
/etc/php5/conf.d/pdo_sqlite.ini,
/etc/php5/conf.d/sqlite3.ini,
/etc/php5/conf.d/tokenizer.ini,
/etc/php5/conf.d/xmlreader.ini,
/etc/php5/conf.d/xmlwriter.ini

gd

GD Support enabled
GD headers Version 2.1.1
GD library Version 2.1.1
FreeType Support enabled
FreeType Linkage with freetype
FreeType Version 2.6.3
GIF Read Support enabled
GIF Create Support enabled
JPEG Support enabled
libJPEG Version 8
PNG Support enabled
libPNG Version 1.6.21
WBMP Support enabled
XPM Support enabled
libXpm Version 30411
XBM Support enabled
WebP Support enabled

And the gd_info dump:


<?php
echo "<h1>gd_info</h1>";
$gdInfo = gd_info();
echo "<table>";
foreach($gdInfo as $key=>$value) {
echo "<tbody>";
echo "<tr>";
echo "<td class='e'>" . $key . "</td>";
echo "<td class='v'>" . $value . "</td>";
echo "</tbody>";
}
echo "</table>";
?>

GD Version 2.1.1
FreeType Support 1
FreeType Linkage with freetype
T1Lib Support
GIF Read Support 1
GIF Create Support 1
JPEG Support 1
PNG Support 1
WBMP Support 1
XPM Support 1
XBM Support 1
WebP Support 1
JIS-mapped Japanese Font Support

Too bad though: no information on where it sources the fonts from.

No image

Having no solution for the font rendering yet, I focussed at the lack of profile picture.

In the past, the images were generated with gravatar information in the JSON, but now that is empty. See for instance the output of http://superuser.com/users/flair/1.json versus his image http://superuser.com/users/flair/1.png


{
"id": 1,
"gravatarHtml": {
},
"profileUrl": "http:\/\/superuser.com\/users\/1\/jeff-atwood",
"displayName": "Jeff Atwood",
"reputation": "14,561",
"badgeHtml": "<span title=\"24 gold badges\"><span class=\"badge1\">&#9679;<\/span><span class=\"badgecount\">24<\/span><\/span><span title=\"79 silver badges\"><span class=\"badge2\">&#9679;<\/span><span class=\"badgecount\">79<\/span><\/span><span title=\"109 bronze badges\"><span class=\"badge3\">&#9679;<\/span><span class=\"badgecount\">109<\/span><\/span>"
}

view raw

1.json

hosted with ❤ by GitHub

That reveals quite a change in the JSON that imageFlair expects. Actually there is a lot of HTML in there.

So decided to try http://superuser.com/users/flair/1 in addition to http://superuser.com/users/flair/1.json with this result:


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"&gt;
<html xmlns="http://www.w3.org/1999/xhtml"&gt;
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
.valuable-flair .userInfo .username a, .valuable-flair .mod-flair
{
color: #1086A4;
}
</style>
<link rel="stylesheet" href="//sstatic.net/flair-Default.css" />
</head>
<body>
<div class="valuable-flair">
<div class="gravatar">
<a title="See my profile on Super User" target="_blank" href="http://superuser.com/users/1/jeff-atwood"><div class="gravatar-wrapper-50"><img src="https://www.gravatar.com/avatar/51d623f33f8b83095db84ff35e15dbe8?s=50&amp;d=identicon&amp;r=PG&quot; alt="" width="50" height="50"></div></a>
</div>
<div class="userInfo">
<span class="username"><img src="http://superuser.com/favicon.ico&quot; width="16" /><a href="http://superuser.com/users/1/jeff-atwood&quot; target="_blank">Jeff Atwood</a><span class="mod-flair" title="moderator">&#9830;</span></span>
<br />
<span class="reputation-score" title="reputation score">14,561</span>
<br />
<span title="24 gold badges"><span class="badge1">&#9679;</span><span class="badgecount">24</span></span><span title="79 silver badges"><span class="badge2">&#9679;</span><span class="badgecount">79</span></span><span title="109 bronze badges"><span class="badge3">&#9679;</span><span class="badgecount">109</span></span>
</div>
</div>
</body>
</html>

view raw

1.html

hosted with ❤ by GitHub

Two downsides here:

  1. This doesn’t work for the combined stackexchange flair: http://stackexchange.com/users/flair/1.png works, but http://stackexchange.com/users/flair/1 gives a 404.
  2. Rendering HTML servers side to PNG requires a lot of work.

Time for another rainy day (:

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, Development, Linux, openSuSE, PHP, Pingback, Power User, Scripting, Software Development, Stackoverflow, SuSE Linux | Leave a Comment »

Recursively convert WMA files to MP3 using ffmeg on a Mac OS X bash shell

Posted by jpluimers on 2017/04/05

A friend of mine made the mistake to capture some CDs using WMA files and throwing away the CDs. His old Nokia could play them, but not his new iOS and Android devices.

ffmeg and bash to the rescue:

find . -iname "*.wma" -execdir bash -c 'NAME="{}" && ffmpeg -y -i "$NAME" -ab 192k "${NAME/.wma/.mp3}" -map_metadata 0:s:0 && rm "$NAME"' \;

Tricks used:

Read the rest of this entry »

Posted in *nix, *nix-tools, Audio, bash, bash, Development, ffmpeg, Media, Power User, Scripting, Software Development | Leave a Comment »

sed: convert Google Drive urls to direct download ones

Posted by jpluimers on 2017/03/14

RegEx Fu

RegEx Fu

One of the things after moving most of my things from copy.com to Google Drive was the direct (public) download URLs that copy.com provides. DropBox has them as well, but Google Drive lacks them in the UI.

There is a URL format that does allow for direct download though:

While Google aims for Drive to be a competent Dropbox competitor, there’s one small but key feature that isn’t easy: sharing direct download links. Fortunately, you can create your own.

Source: Share Direct Links to Files in Google Drive and Skip the Web Viewer

You can do a similar replacement for Google Doc URLs: How to Create Direct Download Links for Files on Google Drive

The Google Drive conversion seems straightforward as they convert from either of

https://drive.google.com/file/d/FILE_ID/edit?usp=sharing
https://drive.google.com/file/d/FILE_ID/view
https://drive.google.com/open?id=FILE_ID

to

https://drive.google.com/uc?export=download&id=FILE_ID

There are tons of RegEx examples for doing the first conversion at Regex to modify Google Drive shared file URL – Stack Overflow, but

  1. they don’t cover the two conversions
  2. they use the non-greedy (.*?) capturing groups which are tricky, introduce question mark escaping issues in hash and many sed implementations fail to implement non-greedy

Since I’m a command-line person, I’ve opted for a sed conversion that wasn’t in the above list. I choose sed because it allows you to convert either a line or a complete file at one time.

There are a few indispensable resources to get my regex expressions right:

So here it goes, starting with fixing https://drive.google.com/open?id=FILE_ID as it’s the most simple replacement because the FILE_ID is at the end.

First of all, these code fragments below are part of bash functions as bash functions remove the quoting hell you have with bash aliases.

Where bash aliases have no parameters (i.e. the arguments are put after the end of the expansion), functions have parameters. So if you want to pass all function parameters to a command inside a function, you have to use “$@” to pass all parameters.

This fragment fixes https://drive.google.com/open?id=FILE_ID printing each fix on one line using the p for printing command in sed:

sed -n 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@p' "$@"

A few remarks:

The second fragment fixes https://drive.google.com/file/d/FILE_ID/edit?usp=sharing and https://drive.google.com/file/d/FILE_ID/view again printing each fix:

sed -n 's@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@p' "$@"

Some more remarks:

  • The FILE_ID is obtained from a capturing group during the match using \([^.]*\) and using the value in the replace with \1 as reference.
  • There is backslash escaping of the parentheses because that’s the sed way.
  • I’ve used a non-greedy \(.*?\) capturing group (sed can’t do that) but \([^.]*\)/ which matches any non-slash inside the capturing group until the first slash outside that group.

The final part is combing both replacement into one sed command:

sed 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@;s@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@' "$@"

Final remarks:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sed, sed script, Software Development | Leave a Comment »