The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘sed’ Category

Next Entries »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 3

Posted by jpluimers on 2021/04/28

Yesterday’s installment ended with a list of power related vim-cmd vmsvc commands:

Usage: power.getstate vmid
Usage: power.hibernate vmid
Usage: power.off vmid
Usage: power.on vmid
Usage: power.reboot vmid
Usage: power.reset vmid
Usage: power.shutdown vmid
Usage: power.suspend vmid
Usage: power.suspendResume vmid

Getting vmid values

These all have a vmid parameter, so let’s create a small statement that gives the list of vmid on an ESXi system:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"

On my system the relevant VMs it returns are these:

10
5

The various power commands

The outcome of the various power commands are not as simple as one might think as they often do not just depend on the current power state of the VM, but also on either VMware Tools or open-vm-tools to be running.

Basically these tools do the same, but their origin is different: open-vm-tools is often included as part of Linux distributions; VMware Tools is often installed separately, see:

I have made a table for this hoping it makes reading easier, the explanations are by empirical usage, as the documentation in the help dump (see [Wayback] delimited vim-cmd help for each vmsvc command.txt) does not seem to match behaviour.

Command Explanation
vim-cmd vmsvc/power.getstate vmid Gets the power state of the VM, returning a line Retrieved runtime info followed by lines indicating the power state:

  • Powered off
  • Powered on
  • Suspended
vim-cmd vmsvc/power.hibernate vmid When neither VMware Tools or open-vm-tools to be running, you get an error:

(vim.fault.ToolsUnavailable) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = 
   msg = "Received SOAP response fault from []: standbyGuest
Cannot complete operation because VMware Tools is not running in this virtual machine."
}

When a hibernate/suspend task is already running, you get an error:

(vim.fault.TaskInProgress) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   task = 'vim.Task:haTask-12-vim.VirtualMachine.suspend-1006072588'
   msg = "Received SOAP response fault from []: standbyGuest
Another task is already in progress."
}

Otherwise, depends on the power state of the VM if vmware tools:

  • Powered off: keeps the power state of the VM as Powered off and shows an error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "poweredOff"
   msg = "Received SOAP response fault from []: standbyGuest
The attempted operation cannot be performed in the current state (Powered off)."
}
  • Powered on: sets the power state of the VM to Suspended and shows no output, unless neither VMware Tools, nor open-vm-tools can be communicated with, then you get this error:
    (vmodl.fault.SystemError) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   reason = "Invalid fault"
   msg = "Received SOAP response fault from []: standbyGuest
vim.fault.GenericVmConfigFault"
}
  • Suspended: keeps the power state of the VM as Suspended and shows an error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "suspended"
   msg = "Received SOAP response fault from []: standbyGuest
The attempted operation cannot be performed in the current state (Suspended)."
}
vim-cmd vmsvc/power.off vmid Depends on the power state of the VM and either VMware Tools or open-vm-tools to be running:

  • Powered off: keeps the power state of the VM as Powered off and shows two lines
    Powering off VM:
Power off failed
  • Powered on: powers off the VM (bypasses any VMware Tools or open-vm-tools) and sets power state of the VM to Powered off showing one line
    Powering off VM:
  • Suspended: Hardware powers off the VM (bypasses any VMware Tools or open-vm-tools) and sets power state of the VM to Powered off showing one line
    Powering off VM:
vim-cmd vmsvc/power.on vmid Depends on the power state of the VM:

  • Powered on: keeps power state of the VM as Powered off showing two lines
    Powering on VM:
Power on failed
  • Powered off: keeps the power state of the VM as Powered off and shows one line
    Powering ofn VM:
  • Suspended: keeps the power state of the VM as Suspended showing two lines
    Powering on VM:
Power on failed

This also undoes a vim-cmd vmsvc/power.hibernate vmid, vim-cmd vmsvc/power.shutdown vmid or vim-cmd vmsvc/power.suspend vmid.
vim-cmd vmsvc/power.reboot vmid Depends on the power state of the VM and either VMware Tools or open-vm-tools to be running:

  • Powered off: keeps the power state of the VM as Powered off and shows the error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "poweredOff"
   msg = "Received SOAP response fault from []: rebootGuest
The attempted operation cannot be performed in the current state (Powered off)."
}
  • Powered on and either VMware Tools or open-vm-tools are running: uses VMware Tools or open-vm-tools to reboot the VM: keeps the power state of the VM as Powered on.
  • Powered on, but neither VMware Tools nor open-vm-tools are running: keeps the power state of the VM as Powered on and shows the error
    (vim.fault.ToolsUnavailable) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = 
   msg = "Received SOAP response fault from []: rebootGuest
Cannot complete operation because VMware Tools is not running in this virtual machine."
}
  • Suspended: keeps the power state of the VM as Suspended and shows the error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "suspended"
   msg = "Received SOAP response fault from []: rebootGuest
The attempted operation cannot be performed in the current state (Suspended)."
}
vim-cmd vmsvc/power.reset vmid Depends on the power state of the VM:

  • Powered off: keeps the power state of the VM as Powered off and shows two lines
    Reset VM:
Reset failed
  • Powered on: keeps the power state of the VM as Powered on, bypasses running VMware Tools or open-vm-tools (basically like a hardware reset button) and shows one line
    Reset VM:
  • Suspended: keeps the power state of the VM as Suspended and shows two lines
    Reset VM:
Reset failed
vim-cmd vmsvc/power.shutdown vmid Depends on the power state of the VM and either VMware Tools or open-vm-tools to be running:

  • Powered off: keeps the power state of the VM as Powered off and shows the error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "poweredOff"
   msg = "Received SOAP response fault from []: shutdownGuest
The attempted operation cannot be performed in the current state (Powered off)."
}
  • Powered on and either VMware Tools or open-vm-tools are running: uses VMware Tools or open-vm-tools to shutdown the VM and sets the power state of the VM to Powered off without showing any output.
  • Powered on, but neither VMware Tools nor open-vm-tools are running: keeps the power state of the VM as Powered on and shows the error
    (vim.fault.ToolsUnavailable) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = 
   msg = "Received SOAP response fault from []: shutdownGuest
Cannot complete operation because VMware Tools is not running in this virtual machine."
}
  • Suspended: keeps the power state of the VM as Suspended and shows the error
    (vim.fault.InvalidPowerState) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   requestedState = "poweredOn", 
   existingState = "suspended"
   msg = "Received SOAP response fault from []: shutdownGuest
The attempted operation cannot be performed in the current state (Suspended)."
}
vim-cmd vmsvc/power.suspend vmid When neither VMware Tools or open-vm-tools to be running, you get an error:

(vim.fault.ToolsUnavailable) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = 
   msg = "Received SOAP response fault from []: standbyGuest
Cannot complete operation because VMware Tools is not running in this virtual machine."
}

When a hibernate/suspend task is already running, you get an error:

(vim.fault.TaskInProgress) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   task = 'vim.Task:haTask-12-vim.VirtualMachine.suspend-1006072588'
   msg = "Received SOAP response fault from []: standbyGuest
Another task is already in progress."
}

Otherwise, depends on the power state of the VM:

  • Powered off: keeps the power state of the VM as Powered off and shows two lines
    Suspending VM:
Suspend failed
  • Powered on: keeps the power state of the VM as Powered on, and shows one line
    Suspending VM:

    unless neither VMware Tools, nor open-vm-tools can be communicated with, then you get this error:

    (vmodl.fault.SystemError) {
   faultCause = (vmodl.MethodFault) null, 
   faultMessage = , 
   reason = "Invalid fault"
   msg = "Received SOAP response fault from []: standbyGuest
vim.fault.GenericVmConfigFault"
}
  • Suspended: keeps the power state of the VM as Suspended and shows two lines
    Suspending VM:
Suspend failed
vim-cmd vmsvc/power.suspendResume vmid Depends on the power state of the VM:

  • Powered off: keeps the power state of the VM as Powered off and shows two lines
    Suspend/Resuming the VM:
Suspend/Resume failed
  • Powered on: keeps the power state of the VM as Powered on, and shows one line
    Suspend/Resuming the VM:
  • Suspended: keeps the power state of the VM as Suspended and shows two lines
    Suspend/Resuming the VM:
Suspend/Resume failed

Note that in the VMware web console, this is shown as “Invoke FSR” which is a Fast Suspend Resume, which I think has to do with vMotion.

Notes:

  • there is no command vim-cmd vmsvc/power.startup vmid (to undo vim-cmd vmsvc/power.shutdown vmid), use vim-cmd vmsvc/power.on vmid in stead.
  • there is no command vim-cmd vmsvc/power.resume vmid (to undo vim-cmd vmsvc/power.suspend vmid), use vim-cmd vmsvc/power.on vmid in stead.
  • there is no command vim-cmd vmsvc/power.wakeup vmid (to undo vim-cmd vmsvc/power.hibernate vmid), use vim-cmd vmsvc/power.on vmid in stead.

Running the various power commands on all relevant VMs

This will be the topic for the next installment.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | 1 Comment »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 2

Posted by jpluimers on 2021/04/27

Last week ended up to be a kind of VMware ESXi heavey, and this week will be similar. So it is time for following up on VMware ESXi console: viewing all VMs, suspending and waking them up: part 1.

That one ended with

Then we get this to print some of the capture groups:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/Vmid:\1 Guest:\6 Version:\7 Name:\3 Datastore:\4 File:\5 Annotation:\8/p"

With this output:

Vmid:10 Guest:windows9_64Guest Version:vmx-14 Name:X9SRI-3F-W10P-EN-MEDIA Datastore:[EVO860_500GB] File:VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx Annotation:
Vmid:5 Guest:centos64Guest Version:vmx-11 Name:PPB Local_Virtual Machine_v4.0 Datastore:[EVO860_500GB] File:VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx Annotation:PowerPanel Business software(Local) provides the service which communicates

Figuring out power state for each VM

This will be in the next installment, as by now this already has become a big blog-post (:

I basically went the vim-cmd vmsvc way instead of the esxcli vm way. My motivation was the easier to understand ID values. They are the basis of virtually all vim-cmd vmsvc based commands:

# vim-cmd vmsvc --help
Commands available under vmsvc/:
acquiremksticket                 get.snapshotinfo                 
acquireticket                    get.spaceNeededForConsolidation  
createdummyvm                    get.summary                      
destroy                          get.tasklist                     
device.connection                getallvms                        
device.connusbdev                gethostconstraints               
device.ctlradd                   message                          
device.ctlrremove                power.getstate                   
device.disconnusbdev             power.hibernate                  
device.diskadd                   power.off                        
device.diskaddexisting           power.on                         
device.diskextend                power.reboot                     
device.diskremove                power.reset                      
device.getdevices                power.shutdown                   
device.nvdimmadd                 power.suspend                    
device.nvdimmremove              power.suspendResume              
device.toolsSyncSet              queryftcompat                    
devices.createnic                reload                           
get.capability                   setscreenres                     
get.config                       snapshot.create                  
get.config.cpuidmask             snapshot.dumpoption              
get.configoption                 snapshot.get                     
get.datastores                   snapshot.remove                  
get.disabledmethods              snapshot.removeall               
get.environment                  snapshot.revert                  
get.filelayout                   snapshot.setoption               
get.filelayoutex                 tools.cancelinstall              
get.guest                        tools.install                    
get.guestheartbeatStatus         tools.upgrade                    
get.managedentitystatus          unregister                       
get.networks                     upgrade                          
get.runtime

My “goto” for getting information is [Wayback] “vim-cmd vmsvc” site:vmware.com – Google Search, and a few sample pages are here:

  1. [Wayback] Performing common virtual machine-related tasks with command-line utilities (2012964) (showing that there are many tasks only vim-cmd vmsvc can do, but esxcli vm cannot)
  2. [Wayback] Powering on a virtual machine from the command line when the host cannot be managed using vSphere Client (1038043) (showing how to combine vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and vim-cmd vmsvc/power.on)
  3. [Wayback] Determine the power status of a virtual machine on an ESX or ESXi host (1003737) (showing vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and ps –auxwww | grep –i VM_NAME)
  4. [Wayback] Collecting information about tasks in VMware ESXi/ESX (1013003) (showing the relation between VMs and tasks using  vim-cmd vimsvc/task_list, vim-cmd vmsvc/getallvms and vim-cmd vimsvc/task_info)
  5. [Wayback] Unable to Power off a Virtual Machine in an ESXi host (1014165) (focussing on vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate, vim-cmd vmsvc/power.shutdown and vim-cmd vmsvc/power.off)
  6. [Wayback] Reloading a vmx file without removing the virtual machine from inventory (1026043) (showing vim-cmd vmsvc/getallvms and vim-cmd vmsvc/reload)
  7. [Wayback] Investigating virtual machine file locks on ESXi hosts (10051) (trying to show how to combine vim-cmd vmsvc/getallvms, grep, awk, find and xargs to find vmdk files, but fails because of parsing errors)

The pattern above is that most of the vim-cmd vmsvc examples are for power state and tasks. Not fully sure why, but my guess is it is what most people use it for. That kind of use what this series of posts also focuses on too, but certainly not the only use. Read the first numbered entry above to get a full grasp of what is possible. I hope to find time in the future to show some more examples outside the power and task realms.

Basically the only time you need to check out esxcli with VMs is when you cannot shut down a VM in a normal way. These links explain what to do in that case:

So let’s go back to basics, and start with getting info on all vim-cmd vmsvc commands.

Help on all vim-cmd vmsvc commands

Executing vim-cmd help vmsvc (preferred) or vim-cmd help vmsvc --help gives you all commands prepended with the line Commands available under vmsvc/:.

Executing vim-cmd help vmsvc/command prints the help for a single command (but vim-cmd help vmsvc/command -help first prints an error, then the help).

Here are the steps how I got the help help for all commands.

First I needed a list of all commands. This is already a multi-stage process, so below the full command I will explain the bits.

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort
  1. vim-cmd help vmsvc gives all the commands (two per line!) prepended by the line Commands available under vmsvc/:.
  2. sed '1d' stripts that line.
  3. xargs -n 1 -r echo does a lot of things:
    1. It parses the sed '1d' input line by line, splits each line into parts, combines all the parts, then executes echo with the combined parts
    2. The -n 1 ensures each invocation of echo takes only a single one of the combined parts
    3. -r is just a protection: if there is no input, then echo is never executed, resulting in empty output
  4. sort will sort all the combined output of all the echo invocations to undo the horizontal combination of parts that xargs did

Now getting the help is doing more of the above, with some more bits to explain:

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} vim-cmd help vmsvc/{}
  1. Normally, xargs will execute each command by appending the parameter inserting a space in front of each parameter
  2. -I {} will force xargs to put each argument just as is in the place where {} is used in the argument
  3. This executes vim-cmd help vmsvc/command in stead of vim-cmd help vmsvc/ command

The result is a long blob of text that is very hard to read as there are no separators between the commands. I saved it as a [Wayback] vim-cmd help for each vmsvc command.txt gist.

With a sh -c shell trick, you can add some more information and separation to the output by embedding :

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} sh -c 'echo "-----" ; echo "help for vim-cmd help vmsvc/{}" ; echo ; vim-cmd help vmsvc/{}'

I have added the output to the [Wayback] delimited vim-cmd help for each vmsvc command.txt gist.

Commands taking a vmid parameter

Now that we know how to output all help, we can filter on it.

An interesting one is to filder only commands taking a vmid parameter:

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} vim-cmd help vmsvc/{} | grep -iw vmid

On VMware ESXi 6.7, this gets you the list:

Usage: acquiremksticket vmid
Usage: acquireticket vmid ticketType
Usage: destroy vmid
Usage: device.connection vmid deviceKey connect
Usage: device.connusbdev vmid usbid
Usage: device.ctlradd vmid ctlr_type bus_number
Usage: device.ctlrremove vmid ctlr_type bus_number
Usage: device.disconnusbdev vmid usbid
Usage: device.diskadd vmid size controller_numer unit_number datastore [ctlr_type]
Usage: device.diskaddexisting vmid disk_file controller_number unit_number [ctlr_type]
Usage: device.diskextend vmid new_size controller_numer unit_number [ctlr_type]
Usage: device.diskremove vmid controller_number unit_number delete_file [controller_type]
Usage: device.getdevices vmid
Usage: device.nvdimmadd vmid size
Usage: device.nvdimmremove vmid deviceKey
Usage: device.toolsSyncSet vmid new state
Usage: devices.createnic vmid adapter-type network-id [network-type]
Usage: get.capability vmid
Usage: get.config vmid
Usage: get.config.cpuidmask vmid
Usage: get.configoption vmid
Usage: get.datastores vmid
Usage: get.disabledmethods vmid
Usage: get.environment vmid
Usage: get.filelayout vmid
Usage: get.filelayoutex vmid
Usage: get.guest vmid
Usage: get.guestheartbeatStatus vmid
Usage: get.managedentitystatus vmid
Usage: get.networks vmid
Usage: get.runtime vmid
Usage: get.snapshotinfo vmid
Usage: get.spaceNeededForConsolidation vmid
Usage: get.summary vmid
Usage: get.tasklist vmid
Usage: message vmid [messageId] [messageChoice]
Usage: power.getstate vmid
Usage: power.hibernate vmid
Usage: power.off vmid
Usage: power.on vmid
Usage: power.reboot vmid
Usage: power.reset vmid
Usage: power.shutdown vmid
Usage: power.suspend vmid
Usage: power.suspendResume vmid
Usage: queryftcompat vmid [faultToleranceType]
Usage: reload vmid
Usage: setscreenres vmid width height
Usage: snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory] [quiesced]
Usage: snapshot.get vmid
Usage: snapshot.remove vmid snapshotId [removeChildren]
Usage: snapshot.removeall vmid
Usage: snapshot.revert vmid snapshotId suppressPowerOn
Usage: snapshot.setoption [OPTIONS] vmid
Usage: tools.cancelinstall vmid
Usage: tools.install vmid
Usage: tools.upgrade vmid [args]
Usage: unregister vmid
Usage: upgrade vmid [vm_hwversion]

In the above list, the bold entries have to do with power, that is what this series is supposed to center around, so more on that tomorrow.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Awk, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, fgrep, Power User, Scripting, sed, sed script, sh, Sh Shell, Software Development, sort, Virtualization, VMware, VMware ESXi, xargs | Leave a Comment »

Audacity error when starting “The application “Audacity.app” can’t be opened.”

Posted by jpluimers on 2020/06/29

For quite a while, I had the error “The application “Audacity.app” can’t be opened.” and Audacity still opening. Clicking the OK button often enough made me hunt for the solution:

This is what I did:

$ cd ~/Library/Application\ Support/audacity
$ grep Temp audacity.cfg
TempDir=/Users/jeroenp/Library/Application Support/audacity/SessionData

Then I checked if the directory existed:

Read the rest of this entry »

Posted in *nix, *nix-tools, Apple, Audacity, Audio, Mac, Mac OS X / OS X / MacOS, MacBook, MacBook Retina, MacBook-Air, MacBook-Pro, Media, Power User, sed | Leave a Comment »

bash + sed: quadruple backslash for proper escaping within an alias

Posted by jpluimers on 2020/06/09

This is part of a bash alias where I had to use quadruple backslash in order to escape it for sed:

# The sed with quad //// is to prevent 'unterminated substitute in regular expression':
alias x='... | sed "s/=.*/ \\\\/"'

This is needed because bash will escape \\\\ into \\ which sed then escapes to \.

The easiest way to find this is to replace the sed with echo to see the expansion.

References:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sed, Software Development | 2 Comments »

Linux command shows a list of all subdirectories that contain a least one .xml file – Thomas Mueller

Posted by jpluimers on 2019/09/09

Via [WayBack] So I don’t forget: The following Linux command shows a list of all subdirectories that contain a least one .xml file… – Thomas Mueller (dummzeuch) – Google+:

[WayBack]Find all subdirectories containing xml files on Linux command line

find . -type f -name '*.xml' | sed -r 's|/[^/]+$||' | sort | uniq

The sed bit is explained in [WayBack] Get list of subdirectories which contain a file matching a string:

The sed command consists of a single substitute. It looks for matches to the regular expression /[^/]+$ and replaces anything matching that with nothing. The dollar sign means the end of the line. [^/]+' means one or more characters that are not slashes. Thus, /[^/]+$ means all characters from the final slash to the end of the line. In other words, this matches the file name at the end of the full path. Thus, the sed command removes the file name, leaving unchanged the name of directory that the file was in.

–jeroen

Posted in *nix, *nix-tools, Power User, sed, sort, uniq | Leave a Comment »

sed in a bash script: backslash escape anything that looks suspicious

Posted by jpluimers on 2019/02/26

Did I ever tell I dislike regular expressions and old-skool shells?

They’re not good for anything but basic commands, so if you try any scripts in them, you’re basically lost.

If you disagree, please read [WayBack] Don’t write Shell scripts. I would recommend Python, but I tried “pip search mysql”…. – Kristian Köhntopp – Google+) and [WayBack] How did this shit ever work? by the same author.

On the other hand: on many system, the baseline isn’t much more than a shell and a very limited tool set.

With nx like systems that usually comes down to sed and a shell like bash.

Since I wanted to modify an openssh hardening script to cover more permutations that was using sed in a bash script, I had not much choice but to bite the bullet.

TL;DR:

When you use any of the below characters, prepend them with a backslash as they have a bash meaning in addition to a sed meaning.

  • ? becomes \?
  • ( becomes \(
  • ) becomes \)
  • | becomes \|

The script

Hopefully by now it’s [Archive.is] been merged into https://github.com/comotion/gone/blob/github/modules/ssh. If not, it’s at https://github.com/jpluimers/gone/blob/jpluimers-ssh-hardening-patch/modules/ssh.

The diff: [Archive.is] https://github.com/jpluimers/gone/commit/329bf12a320704080e68eee90f4c099e92d8388d?diff=unified

The relevant portion (which also uses backslashes as line continuation and wrap a command over multiple lines [WayBack]):

sed -i \
-e 's/#\?MaxAuthTries *[0-9]*.*/MaxAuthTries 2/' \
-e 's/#\?PermitRootLogin *\(yes\|no\).*/PermitRootLogin no/' \
-e 's/#\?UsePrivilegeSeparation *\(yes\|no\|sandbox\).*/UsePrivilegeSeparation sandbox/' \
-e 's/#\?StrictModes *\(yes\|no\).*/StrictModes yes/' \
-e 's/#\?IgnoreRhosts *\(yes\|no\).*/IgnoreRhosts yes/' \
-e 's/#\?PermitEmptyPasswords *\(yes\|no\).*/PermitEmptyPasswords no/' \
-e 's/#\?ChallengeResponseAuthentication *\(yes\|no\).*/ChallengeResponseAuthentication yes/' \
-e 's/#\?KerberosAuthentication *\(yes\|no\).*/KerberosAuthentication no/' \
-e 's/#\?GSSAPIAuthentication *\(yes\|no\).*/GSSAPIAuthentication no/' \
-e 's/#\?GatewayPorts *\(yes\|no\).*/GatewayPorts no/' \
-e 's/#\?X11Forwarding *\(yes\|no\).*/X11Forwarding no/' \
-e 's/#\?PrintMotd *\(yes\|no\).*/PrintMotd no/' \
-e 's/#\?PrintLastLog *\(yes\|no\).*/PrintLastLog yes/' \
-e 's/#\?TCPKeepAlive *\(yes\|no\).*/TCPKeepAlive no/' \
-e 's/#\?PermitUserEnvironment *\(yes\|no\).*/PermitUserEnvironment no/' \
-e 's/^\(HostKey .*ssh_host_dsa_key\)/#\1/' \
sshd_config

More on sshd hardening

In case I have to revisit the script again, here are some more links on ssh and hardening from my blog posts:

–jeroen

 

 

 

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sed, sed script, Software Development | Leave a Comment »

15 Useful ‘sed’ Command Tips and Tricks for Daily Linux System Administration Tasks

Posted by jpluimers on 2017/10/06

In this article we will review sed, the well-known stream editor, and share 15 tips to use it in order to accomplish the goals mentioned earlier, and more.

I like it because 15 Useful ‘sed’ Command Tips and Tricks for Daily Linux System Administration Tasks has a lot of screenshots showing what each expression actually does.

–jeroen

Posted in *nix, *nix-tools, Power User, sed | Leave a Comment »

sed: convert Google Drive urls to direct download ones

Posted by jpluimers on 2017/03/14

RegEx Fu

RegEx Fu

One of the things after moving most of my things from copy.com to Google Drive was the direct (public) download URLs that copy.com provides. DropBox has them as well, but Google Drive lacks them in the UI.

There is a URL format that does allow for direct download though:

While Google aims for Drive to be a competent Dropbox competitor, there’s one small but key feature that isn’t easy: sharing direct download links. Fortunately, you can create your own.

Source: Share Direct Links to Files in Google Drive and Skip the Web Viewer

You can do a similar replacement for Google Doc URLs: How to Create Direct Download Links for Files on Google Drive

The Google Drive conversion seems straightforward as they convert from either of

https://drive.google.com/file/d/FILE_ID/edit?usp=sharing
https://drive.google.com/file/d/FILE_ID/view
https://drive.google.com/open?id=FILE_ID

to

https://drive.google.com/uc?export=download&id=FILE_ID

There are tons of RegEx examples for doing the first conversion at Regex to modify Google Drive shared file URL – Stack Overflow, but

  1. they don’t cover the two conversions
  2. they use the non-greedy (.*?) capturing groups which are tricky, introduce question mark escaping issues in hash and many sed implementations fail to implement non-greedy

Since I’m a command-line person, I’ve opted for a sed conversion that wasn’t in the above list. I choose sed because it allows you to convert either a line or a complete file at one time.

There are a few indispensable resources to get my regex expressions right:

So here it goes, starting with fixing https://drive.google.com/open?id=FILE_ID as it’s the most simple replacement because the FILE_ID is at the end.

First of all, these code fragments below are part of bash functions as bash functions remove the quoting hell you have with bash aliases.

Where bash aliases have no parameters (i.e. the arguments are put after the end of the expansion), functions have parameters. So if you want to pass all function parameters to a command inside a function, you have to use “$@” to pass all parameters.

This fragment fixes https://drive.google.com/open?id=FILE_ID printing each fix on one line using the p for printing command in sed:

sed -n 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@p' "$@"

A few remarks:

The second fragment fixes https://drive.google.com/file/d/FILE_ID/edit?usp=sharing and https://drive.google.com/file/d/FILE_ID/view again printing each fix:

sed -n 's@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@p' "$@"

Some more remarks:

  • The FILE_ID is obtained from a capturing group during the match using \([^.]*\) and using the value in the replace with \1 as reference.
  • There is backslash escaping of the parentheses because that’s the sed way.
  • I’ve used a non-greedy \(.*?\) capturing group (sed can’t do that) but \([^.]*\)/ which matches any non-slash inside the capturing group until the first slash outside that group.

The final part is combing both replacement into one sed command:

sed 's@https://drive.google.com/open?id=@https://drive.google.com/uc?export=download\&id=@;s@https://drive.google.com/file/d/\([^.]*\)/.*@https://drive.google.com/uc?export=download\&id=\1@' "$@"

Final remarks:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, sed, sed script, Software Development | Leave a Comment »

Next Entries »