The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,947 other followers

Archive for the ‘Awk’ Category

Exporting your Twitter content, converting to Markdown and getting the image alt-texts (thanks @isotopp/@HBeckPDX/@weiglemc for the info and @kcgreenn/@dreamjar for the comic!)

Posted by jpluimers on 2022/11/12

This is fine #Twitter

This is fine #Twitter (illustration inspired by KC Green; creation video below)

(Edit 20221114: script for high-res images; more tweets from Jan) (Edit 20221116: hat-tip to Sam) (Edit 20221120: archiving t.co links by Michele Weigle) (Edit 20221122: added article by Johan van der Knijff) (20221128 Tapue export tool by Mike Hucka)

Time to be prepared:

The below will help you exporting your Twitter content (Tweets, DMs, media), perform some conversions on them and optionally delete (parts of) your content.

Important: keep your Twitter account afterwards (to prevent someone from creating a new account with the same handle).

Read the rest of this entry »

Posted in *nix, *nix-tools, Awk, Development, Lightweight markup language, MarkDown, Mastodon, Power User, SocialMedia, Software Development, Twitter | Leave a Comment »

Some notes on replacing parts of a text file with template text using sed on a Busybox system

Posted by jpluimers on 2022/03/17

Note before you think about putting stuff in /etc/rc.local.d/local.sh: that script will not be executed when UEFI booting.

In a very lightweight Busybox system, I wanted to modify some configuration files automatically using fragments stored in template files.

The system has diff, but no patch.

The basic idea is to use sed to insert the template files into certain spots of the configuration file when certain marker texts are not present. So I want the opposite of [Wayback] Hey Stephen Wood: Try patch instead of sed in shell scripts.

Basically the idea is a poor-man’s patch, described in Too bad: ESXi busybox has diff, but not patch « The Wiert Corner – irregular stream of stuff.

Some links that might help me with this:

One alternative would have been to use ed (which is part of the normal Busybox), but ESXi Busybox omits ed like it omits patch.

Too bad that sed commands are too different from ed commands, as I could have used diff -e on another system based on ideas here:

I might give it one more go, as vi is sort of derived from ed via ex (see vi: Creation – Wikipedia), which means that vi “colon mode” (officially command mode: [Wayback] Vim documentation: cmdline) is very similar to ed.

Another alternative would be awk, but I have done so little work with it awk, that I’m hesitating to use a new tool. Some links:

And finally, ash could be used:

The kind of modifications I am after

Below are a few links with examples of the kind of modifications I want to make. Most patch just /etc/rc.local.d/local.sh, but some others introduce other changes as well.

Note that especially with networking settings, local.sh commands might not have any effect (for instance when having slow DHCP or other network issues), see for instance [Wayback/Archive.is] I’m running ESXi 5.5 and my persistent route in local.sh is not taking effect after boot. : vmware.

There is a very convoluted way around using local.sh by using the VIB authoring tool as described in [Wayback] How to create persistent firewall rules on ESXi. It requires lowering the software acceptance level to Community Supported (esxcli software acceptance set --level=CommunitySupported), which gives you a hard time installing ESXi updates.

I got that VIB idea from [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN, as:

The local.sh file gets overwritten often with upgrades so it would mean another step during the process.

From the same thread comes [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN

set the sticky bit on your separate xml-file – then it will be backed up and persist through reboot: chmod +t

run backup manually before the first reboot: /sbin/auto-backup.sh  because backup runs only once per hour

Within vSphere, one could use [Wayback] Configure ESXi Hosts with Host Profiles, but a standalone ESXi box is not part of vSphere, so that won’t work.

ESXi 7 and up

ESXi 7 makes the above harder as for instance user root cannot change file rights any more, so eventually I might revert to a VM that auto-boots when ESXi comes up, then patches the right files in place over PowerCLI (read-only) or SSH.

Need to give this some thought later:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Awk, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, PowerCLI, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Cryptosense Discovery

Posted by jpluimers on 2022/03/15

This is cool: [Wayback] Cryptosense Discovery:

Free tool that discovers security configuration errors in SSH and TLS servers and explains how to fix them. Supports STARTTLS and can also scan HTTPS, POP3, IMAP and SMTP servers.

It gives you a list of servers a target domain uses (for purposes like web, email, etc) that can have external encryption enabled, then allows you to test these.

The list by default has only servers within that target domain enabled, but you can optionally include other servers (for instance if a domain uses a third party for their SMTP handling).

Basically it is the web-counterpart of a tool like testssl.sh (which I have written about before).

Found while checking out how to test the MX security of a domain using [Wayback] testssl.sh as I forgot the syntax, which in retrospect is dead easy as per [Wayback] tls – How to use testssl.sh on an SMTP server? – Information Security Stack Exchange (thanks [Wayback] Z.T.!):

testssl.sh --mx <domain name>

works fine.

testssl.sh -t smtp <ip>:25

and

testssl.sh -t smtp <ip>:587

also work fine.

Note that not specifying the port assumes port 443, despite specifying protocol smtp. That doesn’t work.

Also, you might try discovery.cryptosense.com which does the same thing only better

That website is made by the cool people at [Wayback] Cryptosense.

Both are a lot easier than the alternatives described in [Wayback] Blog · How to test SMTP servers using the command-line · Halon MTA: using nslookup and dig for determining the affected hosts, using nc or telnet for testing basic connectivity, using [Wayback/Archive.is] openssl s_client to test TLS, and [Wayback/Archive.is] smtpping for measuring throughput.

In addition to the above tools mentioned in the blog, I’ve also used sendEmail (note case sensitivity), ehlo-size, and swaks.

This is what I tested:

–jeroen

Posted in *nix, *nix-tools, Awk, bash, bash, Communications Development, Development, DNS, Encryption, grep, HTTPS/TLS security, Internet, Internet protocol suite, Power User, Scripting, Security, SMTP, Software Development, SSH, ssh/sshd, TCP, testssl.sh, TLS | Leave a Comment »

VMware ESXi console: viewing all VMs, suspending and waking them up: part 2

Posted by jpluimers on 2021/04/27

Last week ended up to be a kind of VMware ESXi heavey, and this week will be similar. So it is time for following up on VMware ESXi console: viewing all VMs, suspending and waking them up: part 1.

That one ended with

Then we get this to print some of the capture groups:

vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/Vmid:\1 Guest:\6 Version:\7 Name:\3 Datastore:\4 File:\5 Annotation:\8/p"

With this output:

Vmid:10 Guest:windows9_64Guest Version:vmx-14 Name:X9SRI-3F-W10P-EN-MEDIA Datastore:[EVO860_500GB] File:VM/X9SRI-3F-W10P-EN-MEDIA/X9SRI-3F-W10P-EN-MEDIA.vmx Annotation:
Vmid:5 Guest:centos64Guest Version:vmx-11 Name:PPB Local_Virtual Machine_v4.0 Datastore:[EVO860_500GB] File:VM/PPB-Local_Virtual-Machine_v4.0/PPB Local_Virtual Machine_v4.0.vmx Annotation:PowerPanel Business software(Local) provides the service which communicates

Figuring out power state for each VM

This will be in the next installment, as by now this already has become a big blog-post (:

I basically went the vim-cmd vmsvc way instead of the esxcli vm way. My motivation was the easier to understand ID values. They are the basis of virtually all vim-cmd vmsvc based commands:

# vim-cmd vmsvc --help
Commands available under vmsvc/:
acquiremksticket                 get.snapshotinfo                 
acquireticket                    get.spaceNeededForConsolidation  
createdummyvm                    get.summary                      
destroy                          get.tasklist                     
device.connection                getallvms                        
device.connusbdev                gethostconstraints               
device.ctlradd                   message                          
device.ctlrremove                power.getstate                   
device.disconnusbdev             power.hibernate                  
device.diskadd                   power.off                        
device.diskaddexisting           power.on                         
device.diskextend                power.reboot                     
device.diskremove                power.reset                      
device.getdevices                power.shutdown                   
device.nvdimmadd                 power.suspend                    
device.nvdimmremove              power.suspendResume              
device.toolsSyncSet              queryftcompat                    
devices.createnic                reload                           
get.capability                   setscreenres                     
get.config                       snapshot.create                  
get.config.cpuidmask             snapshot.dumpoption              
get.configoption                 snapshot.get                     
get.datastores                   snapshot.remove                  
get.disabledmethods              snapshot.removeall               
get.environment                  snapshot.revert                  
get.filelayout                   snapshot.setoption               
get.filelayoutex                 tools.cancelinstall              
get.guest                        tools.install                    
get.guestheartbeatStatus         tools.upgrade                    
get.managedentitystatus          unregister                       
get.networks                     upgrade                          
get.runtime                      

My “goto” for getting information is [Wayback] “vim-cmd vmsvc” site:vmware.com – Google Search, and a few sample pages are here:

  1. [Wayback] Performing common virtual machine-related tasks with command-line utilities (2012964) (showing that there are many tasks only vim-cmd vmsvc can do, but esxcli vm cannot)
  2. [Wayback] Powering on a virtual machine from the command line when the host cannot be managed using vSphere Client (1038043) (showing how to combine vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and vim-cmd vmsvc/power.on)
  3. [Wayback] Determine the power status of a virtual machine on an ESX or ESXi host (1003737) (showing vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate and ps –auxwww | grep –i VM_NAME)
  4. [Wayback] Collecting information about tasks in VMware ESXi/ESX (1013003) (showing the relation between VMs and tasks using  vim-cmd vimsvc/task_list, vim-cmd vmsvc/getallvms and vim-cmd vimsvc/task_info)
  5. [Wayback] Unable to Power off a Virtual Machine in an ESXi host (1014165) (focussing on vim-cmd vmsvc/getallvms, vim-cmd vmsvc/power.getstate, vim-cmd vmsvc/power.shutdown and vim-cmd vmsvc/power.off)
  6. [Wayback] Reloading a vmx file without removing the virtual machine from inventory (1026043) (showing vim-cmd vmsvc/getallvms and vim-cmd vmsvc/reload)
  7. [Wayback] Investigating virtual machine file locks on ESXi hosts (10051) (trying to show how to combine vim-cmd vmsvc/getallvms, grep, awk, find and xargs to find vmdk files, but fails because of parsing errors)

The pattern above is that most of the vim-cmd vmsvc examples are for power state and tasks. Not fully sure why, but my guess is it is what most people use it for. That kind of use what this series of posts also focuses on too, but certainly not the only use. Read the first numbered entry above to get a full grasp of what is possible. I hope to find time in the future to show some more examples outside the power and task realms.

Basically the only time you need to check out esxcli with VMs is when you cannot shut down a VM in a normal way. These links explain what to do in that case:

So let’s go back to basics, and start with getting info on all vim-cmd vmsvc commands.

Help on all vim-cmd vmsvc commands

Executing vim-cmd help vmsvc (preferred) or vim-cmd help vmsvc --help gives you all commands prepended with the line Commands available under vmsvc/:.

Executing vim-cmd help vmsvc/command prints the help for a single command (but vim-cmd help vmsvc/command -help first prints an error, then the help).

Here are the steps how I got the help help for all commands.

First I needed a list of all commands. This is already a multi-stage process, so below the full command I will explain the bits.

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort
  1. vim-cmd help vmsvc gives all the commands (two per line!) prepended by the line Commands available under vmsvc/:.
  2. sed '1d' stripts that line.
  3. xargs -n 1 -r echo does a lot of things:
    1. It parses the sed '1d' input line by line, splits each line into parts, combines all the parts, then executes echo with the combined parts
    2. The -n 1 ensures each invocation of echo takes only a single one of the combined parts
    3. -r is just a protection: if there is no input, then echo is never executed, resulting in empty output
  4. sort will sort all the combined output of all the echo invocations to undo the horizontal combination of parts that xargs did

Now getting the help is doing more of the above, with some more bits to explain:

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} vim-cmd help vmsvc/{}
  1. Normally, xargs will execute each command by appending the parameter inserting a space in front of each parameter
  2. -I {} will force xargs to put each argument just as is in the place where {} is used in the argument
  3. This executes vim-cmd help vmsvc/command in stead of vim-cmd help vmsvc/ command

The result is a long blob of text that is very hard to read as there are no separators between the commands. I saved it as a [Wayback] vim-cmd help for each vmsvc command.txt gist.

With a sh -c shell trick, you can add some more information and separation to the output by embedding :

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} sh -c 'echo "-----" ; echo "help for vim-cmd help vmsvc/{}" ; echo ; vim-cmd help vmsvc/{}'

I have added the output to the [Wayback] delimited vim-cmd help for each vmsvc command.txt gist.

Commands taking a vmid parameter

Now that we know how to output all help, we can filter on it.

An interesting one is to filder only commands taking a vmid parameter:

vim-cmd help vmsvc | sed '1d' | xargs -n 1 -r echo | sort | xargs -n 1 -r -I {} vim-cmd help vmsvc/{} | grep -iw vmid

On VMware ESXi 6.7, this gets you the list:

Usage: acquiremksticket vmid
Usage: acquireticket vmid ticketType
Usage: destroy vmid
Usage: device.connection vmid deviceKey connect
Usage: device.connusbdev vmid usbid
Usage: device.ctlradd vmid ctlr_type bus_number
Usage: device.ctlrremove vmid ctlr_type bus_number
Usage: device.disconnusbdev vmid usbid
Usage: device.diskadd vmid size controller_numer unit_number datastore [ctlr_type]
Usage: device.diskaddexisting vmid disk_file controller_number unit_number [ctlr_type]
Usage: device.diskextend vmid new_size controller_numer unit_number [ctlr_type]
Usage: device.diskremove vmid controller_number unit_number delete_file [controller_type]
Usage: device.getdevices vmid
Usage: device.nvdimmadd vmid size
Usage: device.nvdimmremove vmid deviceKey
Usage: device.toolsSyncSet vmid new state
Usage: devices.createnic vmid adapter-type network-id [network-type]
Usage: get.capability vmid
Usage: get.config vmid
Usage: get.config.cpuidmask vmid
Usage: get.configoption vmid
Usage: get.datastores vmid
Usage: get.disabledmethods vmid
Usage: get.environment vmid
Usage: get.filelayout vmid
Usage: get.filelayoutex vmid
Usage: get.guest vmid
Usage: get.guestheartbeatStatus vmid
Usage: get.managedentitystatus vmid
Usage: get.networks vmid
Usage: get.runtime vmid
Usage: get.snapshotinfo vmid
Usage: get.spaceNeededForConsolidation vmid
Usage: get.summary vmid
Usage: get.tasklist vmid
Usage: message vmid [messageId] [messageChoice]
Usage: power.getstate vmid
Usage: power.hibernate vmid
Usage: power.off vmid
Usage: power.on vmid
Usage: power.reboot vmid
Usage: power.reset vmid
Usage: power.shutdown vmid
Usage: power.suspend vmid
Usage: power.suspendResume vmid
Usage: queryftcompat vmid [faultToleranceType]
Usage: reload vmid
Usage: setscreenres vmid width height
Usage: snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory] [quiesced]
Usage: snapshot.get vmid
Usage: snapshot.remove vmid snapshotId [removeChildren]
Usage: snapshot.removeall vmid
Usage: snapshot.revert vmid snapshotId suppressPowerOn
Usage: snapshot.setoption [OPTIONS] vmid
Usage: tools.cancelinstall vmid
Usage: tools.install vmid
Usage: tools.upgrade vmid [args]
Usage: unregister vmid
Usage: upgrade vmid [vm_hwversion]

In the above list, the bold entries have to do with power, that is what this series is supposed to center around, so more on that tomorrow.

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, Awk, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, fgrep, Power User, Scripting, sed, sed script, sh, Sh Shell, Software Development, sort, Virtualization, VMware, VMware ESXi, xargs | Leave a Comment »

How to Setup Chroot SFTP in Linux (Allow Only SFTP, not SSH)

Posted by jpluimers on 2018/11/07

I need to script this one day: [WayBackHow to Setup Chroot SFTP in Linux (Allow Only SFTP, not SSH)

–jeroen

Posted in *nix, Awk, bash, Communications Development, Development, Internet protocol suite, Power User, Scripting, Software Development, SSH, TCP | Leave a Comment »

 
%d bloggers like this: