The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,839 other subscribers

sed in a bash script: backslash escape anything that looks suspicious

Posted by jpluimers on 2019/02/26

Did I ever tell I dislike regular expressions and old-skool shells?

They’re not good for anything but basic commands, so if you try any scripts in them, you’re basically lost.

If you disagree, please read [WayBack] Don’t write Shell scripts. I would recommend Python, but I tried “pip search mysql”…. – Kristian Köhntopp – Google+) and [WayBack] How did this shit ever work? by the same author.

On the other hand: on many system, the baseline isn’t much more than a shell and a very limited tool set.

With nx like systems that usually comes down to sed and a shell like bash.

Since I wanted to modify an openssh hardening script to cover more permutations that was using sed in a bash script, I had not much choice but to bite the bullet.

TL;DR:

When you use any of the below characters, prepend them with a backslash as they have a bash meaning in addition to a sed meaning.

  • ? becomes \?
  • ( becomes \(
  • ) becomes \)
  • | becomes \|

The script

Hopefully by now it’s [Archive.is] been merged into https://github.com/comotion/gone/blob/github/modules/ssh. If not, it’s at https://github.com/jpluimers/gone/blob/jpluimers-ssh-hardening-patch/modules/ssh.

The diff: [Archive.is] https://github.com/jpluimers/gone/commit/329bf12a320704080e68eee90f4c099e92d8388d?diff=unified

The relevant portion (which also uses backslashes as line continuation and wrap a command over multiple lines [WayBack]):

sed -i \
-e 's/#\?MaxAuthTries *[0-9]*.*/MaxAuthTries 2/' \
-e 's/#\?PermitRootLogin *\(yes\|no\).*/PermitRootLogin no/' \
-e 's/#\?UsePrivilegeSeparation *\(yes\|no\|sandbox\).*/UsePrivilegeSeparation sandbox/' \
-e 's/#\?StrictModes *\(yes\|no\).*/StrictModes yes/' \
-e 's/#\?IgnoreRhosts *\(yes\|no\).*/IgnoreRhosts yes/' \
-e 's/#\?PermitEmptyPasswords *\(yes\|no\).*/PermitEmptyPasswords no/' \
-e 's/#\?ChallengeResponseAuthentication *\(yes\|no\).*/ChallengeResponseAuthentication yes/' \
-e 's/#\?KerberosAuthentication *\(yes\|no\).*/KerberosAuthentication no/' \
-e 's/#\?GSSAPIAuthentication *\(yes\|no\).*/GSSAPIAuthentication no/' \
-e 's/#\?GatewayPorts *\(yes\|no\).*/GatewayPorts no/' \
-e 's/#\?X11Forwarding *\(yes\|no\).*/X11Forwarding no/' \
-e 's/#\?PrintMotd *\(yes\|no\).*/PrintMotd no/' \
-e 's/#\?PrintLastLog *\(yes\|no\).*/PrintLastLog yes/' \
-e 's/#\?TCPKeepAlive *\(yes\|no\).*/TCPKeepAlive no/' \
-e 's/#\?PermitUserEnvironment *\(yes\|no\).*/PermitUserEnvironment no/' \
-e 's/^\(HostKey .*ssh_host_dsa_key\)/#\1/' \
sshd_config

More on sshd hardening

In case I have to revisit the script again, here are some more links on ssh and hardening from my blog posts:

–jeroen

 

 

 

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sed, sed script, Software Development | Leave a Comment »

WoL (Wake on LAN) from various routers

Posted by jpluimers on 2019/02/25

Until recently, I hardly used Wake on LAN, so I never noticed that many routers nowadays can send WoL requests themselves.

A few links:

And a few ones from my previous WoL related posts:

–jeroen

Posted in Ethernet, Network-and-equipment, Power User, Wake-on-LAN (WoL) | Leave a Comment »

How to hide an entire drive from prying eyes on Windows 10 | Windows Central

Posted by jpluimers on 2019/02/25

For my link archive: 3 ways to hide drive letters.

TL;DR:

  1. Using diskman.msc (Disk Management) by removing drive letters or/and changing the mount point to be in another drive.
  2. Using regedit.exe (or other Registry Editor like reg.exe) for a bitmap of drive letters to add a value named [WayBackNoDrives to
    • globally to HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer
    • locally for the current user to HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer
  3. Using secpol.msc (Local Security Policy editor, or find it through gpedit.msc) to add a security policy similar to the registry.

Source: [WayBackHow to hide an entire drive from prying eyes on Windows 10 | Windows Central

After setting NoDrive you have to reboot (logoff/logon isn’t sufficient).

For the NoDrive bitmap: these list below has the values to add for each drive to hide, but it’s easier to use the [WayBackNT Drive Calculator – The ‘NoDrives’ Registry Key Value Calculator which calculates the Decimal version of the value needed.

Read the rest of this entry »

Posted in Power User, Windows, Windows 10 | Leave a Comment »

If you are looking for good Travis engineers, look at the #TravisAlums hashtag on Twitter

Posted by jpluimers on 2019/02/23

Idera is laying of a lot of really good Travis engineers.

If you want to hire them, then follow the [Archive.is] #TravisAlums hashtag on Twitter

Via: [WayBackJosé León Serna on Twitter: “Idera acquired Embarcadero Technologies in 2016 and fired almost all its R&D (150+ people in Spain, Russia, US, etc) so not sure why #TravisAlums are surprised, that’s what they do, that’s their business model.”

–jeroen

Posted in Continuous Integration, Development, Software Development, TravisCI | Leave a Comment »

Ubiquity UniFi SDN – Server status “UniFi Controller is starting up… Please wait a moment”

Posted by jpluimers on 2019/02/22

When your UniFi Cloud Key web interface shows this [Archive.is] after upgrading:

 

UniFi Controller is starting up…

Please wait a moment

 

Then you just ran into a bug which seems to occur most with 5.9.29 of the UniFi software: [Archive.is] “UniFi Controller is starting up…” “Please wait a moment” “5.9.29” – Google Search

The easiest way is to restore from a backup: this usually works.

For that you need ssh access, which usually is with the ubnt user. But you could have made life more complicated when you followed these:

Steps from [WayBack] UniFi Controller is starting up… Please Wait A Moment – Ubiquiti Networks Community (thanks [Archive.is] About mrfoxdk – Ubiquiti Networks Community!):

  1. SSH to the Unifi CloudKey
  2. dpkg -P unifi
  3. Open a browser and browse the URL of CloudKey
  4. Login
  5. Press “Install” under UniFi
  6. Wait for the process to complete, and then wait a bit for the service to be brought online.
  7. Recover the configuration backup from the latest backup

Removing and reinstalling the UniFi software, then restore from backup

Step 2 above will purge the unifi software of the cloud key as per dpkg --help:

Read the rest of this entry »

Posted in *nix, Power User, Ubiquiti, WiFi | Leave a Comment »

Many Mac OS X / MacOX / whatever versions: Unable to modify the volume with the keyboard – Ask Different

Posted by jpluimers on 2019/02/22

This has happened to me on most Macs with most Apple Mac OS X / MacOS / whatever versions: the built in sound controls for internal speakers and head phones fail to work (keyboard shortcuts and UI both fail).

The solution at [WayBackmavericks – Unable to modify the volume with the keyboard – Ask Different works, but be sure to require the kernel module steps:

open up a Terminal window and run:

sudo killall coreaudiod
sudo kextunload /System/Library/Extensions/AppleHDA.kext 
sudo kextload /System/Library/Extensions/AppleHDA.kext

–jeroen

Posted in Apple, Mac, Mac OS X / OS X / MacOS, MacBook, MacBook Retina, MacBook-Air, MacMini, macOS 10.12 Sierra, OS X 10.10 Yosemite, OS X 10.11 El Capitan, OS X 10.9 Mavericks, Power User | Leave a Comment »

When Windows 10 doesn’t recognise your DVD device any more

Posted by jpluimers on 2019/02/22

Yes, some people still use DVD devices. My mentally retarded brother does every now and then, so it was a big problem that one day he could not put play a photo DVD any more.

This happened:

The selected device was gone. Reboots (including cold ones) didn’t help.

What helped was selecting the “Dual Channel PCI IDE Controller”, remove it, then reboot so Windows would try to re-install the drivers. After that the DVD drive appeared again.

Later I found these two links with similar solutions:

–jeroen

 

Posted in Power User, Windows, Windows 10 | Leave a Comment »

Application shutdown: wait for all threads to terminate or not?

Posted by jpluimers on 2019/02/21

A while ago, I ran into a problem that an anonymous thread would run longer than the main thread of the application.

This caused all sorts of trouble, so in this case I decided to fix it for that particular thread.

There are various opinions if this should be done for all threads or not. Like always, it depends, so it is good to mention a few:

This particular case resulted into the memory manager shutting down earlier than the anonymous thread, but the anonymous thread was still using memory allocation functions, resulting into a few things of which you do not want the first and second to happen on a continuous integration system:

  1. Error messages during shutdown, which is unwanted on a headless system:
    ---------------------------
    MyIntegrationTests.exe: MM Operation after uninstall.
    ---------------------------
    FastMM has detected a GetMem call after FastMM was uninstalled.
    ---------------------------
    OK 
    ---------------------------

    or

    ---------------------------
    MyIntegrationTests.exe: MM Operation after uninstall.
    ---------------------------
    FastMM has detected a FreeMem call after FastMM was uninstalled.
    ---------------------------
    OK 
    ---------------------------

    either of them followed by

    ---------------------------
    Error
    ---------------------------
    Runtime error 203 at 00408EFF
    ---------------------------
    OK 
    ---------------------------

    or

    ---------------------------
    Error
    ---------------------------
    Runtime error 204 at 0040AFE9
    ---------------------------
    OK 
    ---------------------------

    The errors are mappings of:

    203, { reOutOfMemory }
    204, { reInvalidPtr }
  2. The MyIntegrationTests_MemoryManager_EventLog.txt to rapidly grow to 100s of megabytes.
  3. The MyIntegrationTests_MemoryManager_EventLog.txt not to be truncated.

This particular case was easy to fix by adding a global (but implementation section contained) boolean indicating if the thread was already finished:

unit DebugInformationLoaderUnit;

interface

implementation

uses
  JclDebug;

var
  LoadDebugInformationAsyncFinished: Boolean = False;

procedure LoadDebugInformationAsync;
begin
  TThread.CreateAnonymousThread(
    procedure
    begin
      TThread.NameThreadForDebugging('LoadDebugInforoamtionAsync');
      DebugInfoAvailable(MainInstance);
      LoadDebugInformationAsyncFinished := True;
    end).Start;
end;

initialization
  LoadDebugInformationAsync;

finalization
  while not LoadDebugInformationAsyncFinished do
  begin
    Sleep(1);
  end;
end.

In addition, I did this to suppress message boxes outside Delphi:

program MyIntegrationTests;

...

{$Include FastMM4Options.inc}

uses
  FastMM4 in '..\..\..\Shared\FastMM4.pas',
  System.Classes,
...;

{$R *.RES}

begin
  TThread.NameThreadForDebugging(ParamStr(0));

  SuppressMessageBoxes := SuppressMessageBoxes // follow pattern in FastMM4.FinalizeMemoryManager
    {$ifdef RequireIDEPresenceForLeakReporting}
        and DelphiIsRunning
    {$endif}
    {$ifdef RequireDebuggerPresenceForLeakReporting}
        and ((DebugHook <> 0)
        {$ifdef PatchBCBTerminate}
        or (Assigned(pCppDebugHook) and (pCppDebugHook^ <> 0))
        {$endif PatchBCBTerminate}
        )
    {$endif}
    ;
  {$WARN SYMBOL_PLATFORM OFF} NoErrMsg := {$WARN SYMBOL_PLATFORM ON} SuppressMessageBoxes; // Set RTL message boxes as well;

  ...
end.

–jeroen

Posted in Conference Topics, Conferences, Delphi, Development, Event, Software Development | Leave a Comment »

How to collect HAProxy metrics

Posted by jpluimers on 2019/02/21

For my link archive:

[WayBackHow to collect HAProxy metrics

Once you’ve figured out what to monitor, it’s time to collect HAProxy metrics! Use either HAProxy’s built-in tools or third-party programs to get the info you need.

Note that the heading of the listen configuration for the built-in statistics page now should be like michael-sqlbot explains in [WayBackHAProxy 1.7 Statistics Setup – Server Fault:

listen stats
    bind :9000

He posted more HAProxy insights, for instance [WayBackunderstanding HAProxy Frontend and Backend current session stats – Server Fault.

–jeroen

 

 

 

Posted in *nix, HAProxy, Power User | Leave a Comment »

IP over Avian Carriers

Posted by jpluimers on 2019/02/21

From the geek fun department: [WayBackIP over Avian Carriers – Wikipedia.

I learned through this slightly after the fight to keep HTTP status code 418 (I’m a teapot) which is part of RFC2324 released on April 1st, 1998.

The IP over Avian Carriers is part of three RFCs, all released on April 1st in various years:

–jeroen

via: Http-statuscode ‘I’m a teapot’ is voorlopig veilig – IT Pro – .Geeks – Tweakers

Posted in Communications Development, Development, Fun, Geeky, HTTP, Internet protocol suite, Software Development, TCP | Leave a Comment »