The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,762 other followers

Archive for the ‘sed script’ Category

sed in a bash script: backslash escape anything that looks suspicious

Posted by jpluimers on 2019/02/26

Did I ever tell I dislike regular expressions and old-skool shells?

They’re not good for anything but basic commands, so if you try any scripts in them, you’re basically lost.

If you disagree, please read [WayBack] Don’t write Shell scripts. I would recommend Python, but I tried “pip search mysql”…. – Kristian Köhntopp – Google+) and [WayBack] How did this shit ever work? by the same author.

On the other hand: on many system, the baseline isn’t much more than a shell and a very limited tool set.

With nx like systems that usually comes down to sed and a shell like bash.

Since I wanted to modify an openssh hardening script to cover more permutations that was using sed in a bash script, I had not much choice but to bite the bullet.

TL;DR:

When you use any of the below characters, prepend them with a backslash as they have a bash meaning in addition to a sed meaning.

  • ? becomes \?
  • ( becomes \(
  • ) becomes \)
  • | becomes \|

The script

Hopefully by now it’s [Archive.is] been merged into https://github.com/comotion/gone/blob/github/modules/ssh. If not, it’s at https://github.com/jpluimers/gone/blob/jpluimers-ssh-hardening-patch/modules/ssh.

The diff: [Archive.is] https://github.com/jpluimers/gone/commit/329bf12a320704080e68eee90f4c099e92d8388d?diff=unified

The relevant portion (which also uses backslashes as line continuation and wrap a command over multiple lines [WayBack]):

sed -i \
-e 's/#\?MaxAuthTries *[0-9]*.*/MaxAuthTries 2/' \
-e 's/#\?PermitRootLogin *\(yes\|no\).*/PermitRootLogin no/' \
-e 's/#\?UsePrivilegeSeparation *\(yes\|no\|sandbox\).*/UsePrivilegeSeparation sandbox/' \
-e 's/#\?StrictModes *\(yes\|no\).*/StrictModes yes/' \
-e 's/#\?IgnoreRhosts *\(yes\|no\).*/IgnoreRhosts yes/' \
-e 's/#\?PermitEmptyPasswords *\(yes\|no\).*/PermitEmptyPasswords no/' \
-e 's/#\?ChallengeResponseAuthentication *\(yes\|no\).*/ChallengeResponseAuthentication yes/' \
-e 's/#\?KerberosAuthentication *\(yes\|no\).*/KerberosAuthentication no/' \
-e 's/#\?GSSAPIAuthentication *\(yes\|no\).*/GSSAPIAuthentication no/' \
-e 's/#\?GatewayPorts *\(yes\|no\).*/GatewayPorts no/' \
-e 's/#\?X11Forwarding *\(yes\|no\).*/X11Forwarding no/' \
-e 's/#\?PrintMotd *\(yes\|no\).*/PrintMotd no/' \
-e 's/#\?PrintLastLog *\(yes\|no\).*/PrintLastLog yes/' \
-e 's/#\?TCPKeepAlive *\(yes\|no\).*/TCPKeepAlive no/' \
-e 's/#\?PermitUserEnvironment *\(yes\|no\).*/PermitUserEnvironment no/' \
-e 's/^\(HostKey .*ssh_host_dsa_key\)/#\1/' \
sshd_config

–jeroen

 

 

 

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sed, sed script, Software Development | Leave a Comment »

sed: convert Google Drive urls to direct download ones

Posted by jpluimers on 2017/03/14

RegEx Fu

RegEx Fu

One of the things after moving most of my things from copy.com to Google Drive was the direct (public) download URLs that copy.com provides. DropBox has them as well, but Google Drive lacks them in the UI.

There is a URL format that does allow for direct download though:

While Google aims for Drive to be a competent Dropbox competitor, there’s one small but key feature that isn’t easy: sharing direct download links. Fortunately, you can create your own.

Source: Share Direct Links to Files in Google Drive and Skip the Web Viewer

You can do a similar replacement for Google Doc URLs: How to Create Direct Download Links for Files on Google Drive

The Google Drive conversion seems straightforward as they convert from either of

https://drive.google.com/file/d/FILE_ID/edit?usp=sharing
https://drive.google.com/file/d/FILE_ID/view
https://drive.google.com/open?id=FILE_ID

to

https://drive.google.com/uc?export=download&id=FILE_ID

There are tons of RegEx examples for doing the first conversion at Regex to modify Google Drive shared file URL – Stack Overflow, but

  1. they don’t cover the two conversions
  2. they use the non-greedy (.*?) capturing groups which are tricky, introduce question mark escaping issues in hash and many sed implementations fail to implement non-greedy

Since I’m a command-line person, I’ve opted for a sed conversion that wasn’t in the above list. I choose sed because it allows you to convert either a line or a complete file at one time.

There are a few indispensable resources to get my regex expressions right:

So here it goes, starting with fixing https://drive.google.com/open?id=FILE_ID as it’s the most simple replacement because the FILE_ID is at the end.

First of all, these code fragments below are part of bash functions as bash functions remove the quoting hell you have with bash aliases.

Where bash aliases have no parameters (i.e. the arguments are put after the end of the expansion), functions have parameters. So if you want to pass all function parameters to a command inside a function, you have to use “$@” to pass all parameters.

This fragment fixes https://drive.google.com/open?id=FILE_ID printing each fix on one line using the p for printing command in sed:

sed -n 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@p' "$@"

A few remarks:

The second fragment fixes https://drive.google.com/file/d/FILE_ID/edit?usp=sharing and https://drive.google.com/file/d/FILE_ID/view again printing each fix:

sed -n 's@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@p' "$@"

Some more remarks:

  • The FILE_ID is obtained from a capturing group during the match using \([^.]*\) and using the value in the replace with \1 as reference.
  • There is backslash escaping of the parentheses because that’s the sed way.
  • I’ve used a non-greedy \(.*?\) capturing group (sed can’t do that) but \([^.]*\)/ which matches any non-slash inside the capturing group until the first slash outside that group.

The final part is combing both replacement into one sed command:

sed 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@;s@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@' "$@"

Final remarks:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sed, sed script, Software Development | Leave a Comment »

RosettaCode: cool way to improve your coding skills

Posted by jpluimers on 2015/01/22

Wow: I feel like having lived under a stone for 8 years, as RosettaCode has been alive since it was founded in 2007 by Mike Mol.

The idea is that you solve a task and learn from that, or learn by seeing how others have solved tasks or draft tasks.

So in a sense it is similar to the Rosetta stone: it has different languages phrasing the same tasks.

There are already a whole bunch of languages on RosettaCode (of which a few are in the categories below), and you can even suggest or add your own languages.

When you want to solve tasks, be sure to look at the list unimplemented tasks by language that leads to automatic reports by language (for instance two of the languages I use most often: C# and Delphi).

I’m sure there are lots of programming chrestomathy sites, even beyond the ones, and it feels very similar to programming kata sites.

–jeroen

Posted in .NET, APL, Awk, bash, Batch-Files, C, C#, C++, COBOL, CommandLine, Delphi, Development, Fortran, FreePascal, Java, JavaScript/ECMAScript, Lazarus, Object Pascal, Office VBA, Pascal, Perl, PHP, PowerShell, PowerShell, Prism, Scripting, sed script, Sh Shell, Software Development, Turbo Prolog, VB.NET, VBS, VBScript, Visual Studio and tools, Web Development | Leave a Comment »

SDB:System upgrade – openSUSE

Posted by jpluimers on 2014/05/01

SDB:System upgrade – openSUSE.

Every once in a while, I feel like I have been living under a stupid rock.

This was one of these cases. About 5 years ago, openSUSE added this feature:

openFATE – #305634: Debian-like dist-upgrade live system full version upgrade.

It is awesome, requires a bit of twiddling and afterwards you have a system that:

  • was updated in-place while live
  • is the most current openSUSE version

I don’t know of any drawbacks yet (will soon try it). If I find ones, I will post a new entry here.

The basic steps are like these: Read the rest of this entry »

Posted in *nix, Development, Linux, openSuSE, Power User, Scripting, sed script, Software Development, SuSE Linux | Leave a Comment »

 
%d bloggers like this: