One of the places explaining a more and more frequent error on Windows 7 installations is [Wayback/Archive.is] “You must provide a value expression on the right-hand side of the ‘-‘ operator.” · Issue #29 · shiftkey/chocolatey-beyondcompare:
Archive for the ‘Scripting’ Category
Chocolatey on Windows 7: “You must provide a value expression on the right-hand side of the ‘-‘ operator.”
Posted by jpluimers on 2022/06/08
Posted in Chocolatey, CommandLine, Development, Microsoft Surface on Windows 7, Power User, PowerShell, PowerShell, Scripting, Software Development, Windows, Windows 10, Windows 7, Windows 8, Windows 8.1, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, Windows Server 2008 R2 | Leave a Comment »
Wake-on-LAN from a Windows machine
Posted by jpluimers on 2022/06/02
Before digging into Wake-on-LAN on Windows machines, I’ll interlude with basically the canonical wakonlan available on many non Windows machines
On Linux and BSD machines, there is the Perl script wakeonlan (steadily at version 0.41) at [Wayback/Archive.is] jpoliv/wakeonlan: Perl script for waking up computers via Wake-On-LAN magic packets (script: [Wayback/Archive.is] wakeonlan/wakeonlan at master · jpoliv/wakeonlan) with this help:
Posted in *nix, *nix-tools, Development, Perl, Power User, Scripting, Software Development, Windows | Leave a Comment »
A twitter call to say nice things about technology sparked interesting threads
Posted by jpluimers on 2022/05/27
A while ago [Archive.is] Adam Jacob on Twitter: “Let’s say nice things about technology today. I’ll start. If it wasn’t for @lkanies and @puppetize, there is no way we would have been able to adapt as an industry to the rise of the cloud. Quote tweet me with your own.” sparked some interesting threads.
First posts are below; click on them to see the full threads.
Posted in Chrome, Configuration Management, Development, DevOps, Firefox, History, IaC - Infrastructure as Code, Infocom and Z-machine, Infrastructure, KVM Kernel-based Virtual Machine, LSI/3ware, Open Source, PDP-11, Power User, PowerShell, Puppet, Python, Qemu, Rust, Safari, Scripting, Software Development, UCSD Pascal, Vagrant, Veewee, Virtualization, Web Browsers, Xen | Leave a Comment »
Filezilla SFTP: figuring out the cause of “Connection timed out after 20 seconds of inactivity” part 2: about ((non-)?(interactive|login) ?){2} bash shells
Posted by jpluimers on 2022/05/12
Last year, I wrote about Filezilla: figuring out the cause of “Connection timed out after 20 seconds of inactivity” about sftp connection problems.
The solution there was to exclude part of bashrc with an if [Wayback] statement so bash would skip it during sftp, but not during ssh login:
[WayBack] linux – Use .bashrc without breaking sftp – Server Fault
- From answer 1 (thanks [WayBack] Mike):
Try doing this instead
if [ "$SSH_TTY" ] then source .bashc_real fi- From Answer 2 (thanks [WayBack] Insyte):
A good trick for testing the cleanliness of your login environment is to
sshin with a command, which simulates the same wayscp/sftpconnect. For example:ssh myhost /bin/truewill show you exactly whatscp/sftpsees when they connect.
That caused some scripts not to be run when switching user, for instance by doing sudo su -.
The reason for that was that I forgot to put enough research in part of Answer 2, so I quote a few bits more of it (highlights and code markup mine):
… it’s worth pointing out that you can accomplish this carefully selecting which startup files to put the verbose stuff in. From the
bashman page:When
bashis invoked as an interactive login shell, or as a non-interactive shell with the--loginoption, it first reads and executes commands from the file/etc/profile, if that file exists. After reading that file, it looks for~/.bash_profile,~/.bash_login, and~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The--noprofileoption may be used when the shell is started to inhibit this behavior.When an interactive shell that is not a login shell is started,
bashreads and executes commands from~/.bashrc, if that file exists. This may be inhibited by using the--norcoption. The--rcfilefile option will force bash to read and execute commands from file instead of~/.bashrc.The
sftp/scptools start an interactive non-login shell, so.bashrcwill be sourced.
For further reading, there is the underlying bash manual as a PDF file [Wayback] and html document tree [Wayback]. Note it is large (the PDF is 190 pages).
I find the easiest way to navigate around bash documentation through these links:
- [Wayback] Top (Bash Reference Manual) (especially read the Bash Features section there)
- [Wayback] Indexes (Bash Reference Manual) (there are five of them, centered around various core topics)
Types of shell invocations: to login or to non-login and to interactive or to non-interactive
Basically, from the above answer there are [Archive.is] 4 types of shells (confirmed by these parts of the bash documentation: [Wayback] Section 6.1: Invoking-Bash and [Wayback] Section 6.2: Bash-Startup-Files):
- interactive login
- interactive non-login
- non-interactive login
- non-interactive non-login
And there are various means the shells can start (ssh, local console, …). The "$SSH_TTY" trick only checks interactive login via ssh, but fails to detect others.
So I did some digging for the correct information to log, which including the above are:
- [Wayback] Section 4.3.1: The-Set-Builtin
-hLocate and remember (hash) commands as they are looked up for execution. This option is enabled by default.-mJob control is enabled (see Job Control). All processes run in a separate process group. When a background job completes, the shell prints a line containing its exit status.-BThe shell will perform brace expansion (see Brace Expansion). This option is on by default.-HEnable ‘!’ style history substitution (see History Interaction). This option is on by default for interactive shells.
Note that in addition to this, there is the non-settable option
i: The current shell is interactive (see the-iin section 6.1 below). - [Wayback] Section 4.3.2: The-Shopt-Builtin
login_shellThe shell sets this option if it is started as a login shell (see Invoking Bash). The value may not be changed.
- [Wayback] Section 6.1: Invoking-Bash
There are several single-character options that may be supplied at invocation which are not available with the
setbuiltin.-iForce the shell to run interactively. Interactive shells are described in Interactive Shells.
…
A login shell is one whose first character of argument zero is ‘-’, or one invoked with the –login option.
- [Wayback] Section 6.2: Bash-Startup-Files explains about these shell invocation types:
- interactive login shell
- interactive non-login shell
- non-interactive shell
- [Wayback] Section 6.3.1: Is-this-Shell-Interactive
To determine within a startup script whether or not Bash is running interactively, test the value of the ‘-’ special parameter. It contains
iwhen the shell is interactive. For example:case "$-" in *i*) echo This shell is interactive ;; *) echo This shell is not interactive ;; esac
Alternatively, startup scripts may examine the variable
PS1; it is unset in non-interactive shells, and set in interactive shells. Thus:if [ -z "$PS1" ]; then echo This shell is not interactive else echo This shell is interactive fi
From theory to practice
After reading the above documentation links, I put the below code in the global .bashrc (which of course caused trouble with sftp, so I commented it out later):
echo "Option flags: '$-'" echo "PS1: '$PS1'" echo "shopt login_shell: '$(shopt login_shell)'" echo "Parameter zero: '$0'" [ "$SSH_TTY" ] ; echo "[ \"\$SSH_TTY\" ] outcome: $?"
And the output after these commands:
-
ssh user@hostOption flags: 'himBH' PS1: '\u@\h:\w> ' shopt login_shell: 'login_shell on' Parameter zero: '-bash' [ "$SSH_TTY" ] outcome: 0Verdict:
interactive,login -
ssh user@hostfollowed by
sudo su -Option flags: 'himBH' PS1: '\[\]\h:\w #\[\] ' shopt login_shell: 'login_shell on' Parameter zero: '-bash' [ "$SSH_TTY" ] outcome: 1Verdict:
interactive,login -
ssh user@hostfollowed by
bashOption flags: 'himBH' PS1: '\u@\h:\w> ' shopt login_shell: 'login_shell off' Parameter zero: 'bash' [ "$SSH_TTY" ] outcome: 0Verdict:
interactive,non-login -
ssh user@hostfollowed by
sudo su -then by
bashOption flags: 'himBH' PS1: '\[\]\h:\w #\[\] ' shopt login_shell: 'login_shell off' Parameter zero: 'bash' [ "$SSH_TTY" ] outcome: 1Verdict:
interactive,non-login -
ssh user@host /bin/trueOption flags: 'hBc' PS1: '' shopt login_shell: 'login_shell off' Parameter zero: 'bash' [ "$SSH_TTY" ] outcome: 1Verdict:
non-interactive,non-login
The final one is what for instance sftp will see. It excludes the non-interactive mark in the shopt option flags.
Modifications to my .bashrc file
Since the [Wayback] test for "$SSH_TTY" is inconsistent with the login being interactive, I modified the .bashrc section
if [ "$SSH_TTY" ] then source .bashc_real fi
to become
if [[ $- =~ i ]] then # only during interactive login shells source .bashc_real fi
I know the [[...]] over test shorthand [...] is a bashism, see [Wayback] if statement – Is double square brackets [[ ]] preferable over single square brackets [ ] in Bash? – Stack Overflow for why I like it.
More relevant documentation
I based the above changes not only on the mentioned StackOverflow post, but also doing some more Googling revealing these useful documentation and question/answer links:
- [Wayback] Section 3.2.5.2: Conditional Constructs;
[[ expression ]]
[[…]][[ expression ]]Return a status of
0or1depending on the evaluation of the conditional expression expression. Expressions are composed of the primaries described below in Bash Conditional Expressions. Word splitting and filename expansion are not performed on the words between the[[and]]; tilde expansion, parameter and variable expansion, arithmetic expansion, command substitution, process substitution, and quote removal are performed. Conditional operators such as ‘-f’ must be unquoted to be recognized as primaries.…
An additional binary operator, ‘
=~’, is available, with the same precedence as ‘==’ and ‘!=’. When it is used, the string to the right of the operator is considered a POSIX extended regular expression and matched accordingly (using the POSIXregcompandregexecinterfaces usually described inregex(3)). The return value is0if the string matches the pattern, and1otherwise. If the regular expression is syntactically incorrect, the conditional expression’s return value is2. - [Wayback] Section 4.1: Bourne Shell Builtins;
testor[...](Bash Reference Manual)
test expr
Evaluate a conditional expression expr and return a status of
0(true) or1(false). Each operator and operand must be a separate argument. Expressions are composed of the primaries described below in Bash Conditional Expressions.testdoes not accept any options, nor does it accept and ignore an argument of--as signifying the end of options.When the
[form is used, the last argument to the command must be a]. - [Wayback] bash – What are the special dollar sign shell variables? – Stack Overflow (thanks [Wayback] kojiro!):
$1,$2,$3, … are the positional parameters."$@"is an array-like construct of all positional parameters,{$1, $2, $3 ...}."$*"is the IFS expansion of all positional parameters,$1 $2 $3 ....$#is the number of positional parameters.$-current options set for the shell.$$pid of the current shell (not subshell).$_most recent parameter (or the abs path of the command to start the current shell immediately after startup).$IFSis the (input) field separator.$?is the most recent foreground pipeline exit status.$!is the PID of the most recent background command.$0is the name of the shell or shell script.
Most of the above can be found under Special Parameters in the Bash Reference Manual. There are all the environment variables set by the shell.
For a comprehensive index, please see the Reference Manual Variable Index.
- [Wayback] bash – Differentiate Interactive login and non-interactive non-login shell – Ask Ubuntu (thanks [Wayback] terdon)
Briefly (see here for more details), with examples:
- interactive login shell: You log into a remote computer via, for example
ssh. Alternatively, you drop to a tty on your local machine (Ctrl+Alt+F1) and log in there. - interactive non-login shell: Open a new terminal.
- non-interactive non-login shell: Run a script. All scripts run in their own subshell and this shell is not interactive. It only opens to execute the script and closes immediately once the script is finished.
- non-interactive login shell: This is extremely rare, and you’re unlikey to encounter it. One way of launching one is
echo command | ssh server. Whensshis launched without a command (sosshinstead ofssh commandwhich will runcommandon the remote shell) it starts a login shell. If thestdinof thesshis not a tty, it starts a non-interactive shell. This is whyecho command | ssh serverwill launch a non-interactive login shell. You can also start one withbash -l -c command.
If you want to play around with this, you can test for the various types of shell as follows:
- Is this shell interactive?Check the contents of the
$-variable. For interactive shells, it will includei:## Normal shell, just running a command in a terminal: interacive $ echo $- himBHs ## Non interactive shell $ bash -c 'echo $-' hBc - Is this a login shell?There is no portable way of checking this but, for bash, you can check if the
login_shelloption is set:## Normal shell, just running a command in a terminal: interacive $ shopt login_shell login_shell off ## Login shell; $ ssh localhost $ shopt login_shell login_shell on
Putting all this together, here’s one of each possible type of shell:
## Interactive, non-login shell. Regular terminal $ echo $-; shopt login_shell himBHs login_shell off ## Interactive login shell $ bash -l $ echo $-; shopt login_shell himBHs login_shell on ## Non-interactive, non-login shell $ bash -c 'echo $-; shopt login_shell' hBc login_shell off ## Non-interactive login shell $ echo 'echo $-; shopt login_shell' | ssh localhost Pseudo-terminal will not be allocated because stdin is not a terminal. hBs login_shell on - interactive login shell: You log into a remote computer via, for example
- [Wayback] Difference between Login Shell and Non-Login Shell? – Unix & Linux Stack Exchange
A login shell is the first process that executes under your user ID when you log in for an interactive session. The login process tells the shell to behave as a login shell with a convention: passing argument 0, which is normally the name of the shell executable, with a
-character prepended (e.g.-bashwhereas it would normally bebash. Login shells typically read a file that does things like setting environment variables:/etc/profileand~/.profilefor the traditional Bourne shell,~/.bash_profileadditionally for bash†,/etc/zprofileand~/.zprofilefor zsh†,/etc/csh.loginand~/.loginfor csh, etc.When you log in on a text console, or through SSH, or with
su -, you get an interactive login shell. When you log in in graphical mode (on an X display manager), you don’t get a login shell, instead you get a session manager or a window manager.It’s rare to run a non-interactive login shell, but some X settings do that when you log in with a display manager, so as to arrange to read the profile files. Other settings (this depends on the distribution and on the display manager) read
/etc/profileand~/.profileexplicitly, or don’t read them. Another way to get a non-interactive login shell is to log in remotely with a command passed through standard input which is not a terminal, e.g.ssh example.com <my-script-which-is-stored-locally(as opposed tossh example.com my-script-which-is-on-the-remote-machine, which runs a non-interactive, non-login shell).When you start a shell in a terminal in an existing session (screen, X terminal, Emacs terminal buffer, a shell inside another, etc.), you get an interactive, non-login shell. That shell might read a shell configuration file (
~/.bashrcfor bash invoked asbash,/etc/zshrcand~/.zshrcfor zsh,/etc/csh.cshrcand~/.cshrcfor csh, the file indicated by theENVvariable for POSIX/XSI-compliant shells such as dash, ksh, and bash when invoked assh,$ENVif set and~/.mkshrcfor mksh, etc.).When a shell runs a script or a command passed on its command line, it’s a non-interactive, non-login shell. Such shells run all the time: it’s very common that when a program calls another program, it really runs a tiny script in a shell to invoke that other program. Some shells read a startup file in this case (bash runs the file indicated by the
BASH_ENVvariable, zsh runs/etc/zshenvand~/.zshenv), but this is risky: the shell can be invoked in all sorts of contexts, and there’s hardly anything you can do that might not break something.† I’m simplifying a little, see the manual for the gory details.
If you want to avoid the [[...]] bashishm, then read [Wayback] Bashism: How to make bash scripts work in dash – Greg’s Wiki.
–jeroen
Posted in *nix, *nix-tools, ash/dash, bash, bash, Communications Development, Conference Topics, Conferences, Development, Event, Internet protocol suite, Power User, Scripting, SFTP, Software Development, SSH, TCP | Leave a Comment »
Zypper: list info on all patterns, so you can find out which pattern provides a package
Posted by jpluimers on 2022/05/11
I wanted to know which pattern provides [WayBack] etckeeper which is in the [WayBack] openSUSE Software package etckeeper.
It seems no built-in search query can do that, so I built one my own.
Since the result takes quite a while to produce, the output is a pattern.txt that you can manually search.
This is the command:
zypper search -t pattern | grep "|" | tail -n +2 | perl -pe 's/^.*? \| //' | perl -pe 's/ *\| .*$//' | xargs -I {} sh -c "zypper info -t pattern {}" > patterns.txt
The content is like this (the 2017 date shows I wrote this a long time ago):
Posted in *nix, *nix-tools, bash, bash, Development, etckeeper, Linux, Perl, Power User, Scripting, sed, Software Development | Leave a Comment »
Remotely controlling most Harman Kardon systems
Posted by jpluimers on 2022/05/03
There is a Harman Kardon Remote app on [Wayback/Archive.is] Android and [Wayback/Archive.is] iOS.
It can control Harman Kardon AVR and BDS devices, and the communication appears to be simple XML over HTTP according to [Wayback/Archive.is] H/K AVR 151 Remote Control API – Adam Parsons.
It got reverse engineered into two libraries, both based on string concatenating XML HTTP requests:
- a PHP based library at [Wayback/Archive.is] KarimGeiger/HKAPI: API wrapper for Harman Kardon AVR with network capabilities and HK Remote support.
- a JavaScript library at [Wayback/Archive.is] andrei4002/homebridge-harman-kardon-avr: not working. wip
Note:
- the Harman Kardon Remote app only runs on my local IPv4 subnet (it won’t work routing from one IPv4 to another IPv4), hopefully the API works better
- the app does auto-detect (wich often fails) of on-line Harman Kardon AVR/BDS devices, I wonder how that is implemented
- there is this [Wayback] Harman Kardon AVR – Home Assistant plugin
- it works on some JBL devices too, as they are based on Harman Kardon internals: [Wayback] Harmon/Kardon AVR – Configuration – Home Assistant Community
I just tested it with a unit test and it work on my JBL AVR101 (same API as the Harman/Kardon)
- that the Harman Kardon remoting protocol is quite unstable (and even worse than the above Remote apps) that can only be solved by cold-rebooting your Harman Kardon device (observed this both on a BDS 277 and BDS 580), so be sure to have a remote controlled power outlet for that.
- You often have to cold-reboot these anyway because they stop listening to the IR-remote control as well or listen to HDMI commands (like ARC or CEC), so I already had such a switch in place.
- Same for Samsung TVs that over time will not wake up using CEC and require cold-rebooting
- various devices and firmware versions have different ideas of the
sleepcommand and eventually can perform a full shutdown, see the comments at [Wayback/Archive.is] Add list of known devices · KarimGeiger/HKAPI@6036d4f. - the Harkan Kardon Remote protocol is totally different from the Harman Kardon Controller protocol used by the Harman Kardon Controller apps on [Wayback/Archive.is] Android and [Wayback/Archive.is] iOS.
- For people interested: the Harman Kardon Controller protocol is for the Harman Kardon Omni speakers and use a JSON/REST API, with:
- documentation at [Wayback] REST API Specification (including PubNub JSON format) — Harman Developer 1.0 documentation.
- implementation in Groovy at [Wayback/Archive.is] tylerfreckmann/smartthings: harman smartthings integration
- via [Wayback] Suport for Harman Kardon Omni speakers – Feature Requests – Home Assistant Community
- For people interested: the Harman Kardon Controller protocol is for the Harman Kardon Omni speakers and use a JSON/REST API, with:
–jeroen
Posted in Development, PHP, Scripting, Software Development | Leave a Comment »
Kris on Twitter is a bit radical against shell scripts. Learn why.
Posted by jpluimers on 2022/04/13
I say to people: only use shell interactively, don’t write scripts. Never. Not one.
But Kris, they ask, why so radical?
Because of this:
is the literal English Google Translation of the German text
Ich sage den Leuten: benutzt Shell nur interaktiv, schreibt keine Scripte. Nie. Nicht eines.
Aber Kris, fragen sie, wieso so Radikal?
Deswegen:
then links to [Wayback/Archive] Jan Schaumann on Twitter: “TIL zgrep(1) is a shell script. BSD basically does “zcat | grep”, but GNU does “gzip -dc | sed”. How did I learn that? The fun way! CVE-2022-1271, arbitrary-file-write and code execution vulnerability in GNU zgrep / gzip. …”:
Posted in *nix, *nix-tools, Apple, ash/dash, ash/dash development, bash, bash, BSD, Development, Mac, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »
When some virtual machines cannot run VMware Tools: Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly)
Posted by jpluimers on 2022/04/12
An interesting set of scripts from [Wayback/Archive.is] No Joke IT: Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly).
If all ESXi virtual machines support running of VMware Tools, then the solution is a plain /sbin/shutdown.sh && /sbin/poweroff (see [Wayback/Archive.is] No Joke IT: Shut down ESXi 5.1 guest VMs and the host (free edition) via SSH – the easy way!).
Code is in the repository at [Wayback/Archive.is] sixdimensionalarray/esxidown: A shell script to shutdown VMware ESXi host servers, with these two main files:
- [Wayback/Archive.is] esxidown/async.sh at master · sixdimensionalarray/esxidown
#!/bin/sh # Runs a shell command asynchronously. if [ "$1" != "" ]; then nohup sh $1 > /dev/null 2>&1 & fi - [Wayback/Archive.is] esxidown/esxidown.sh at master · sixdimensionalarray/esxidown with this interesting code fragment right before the script finishes:
# exit maintenance mode immediately before server has a chance to shutdown/power off # NOTE: it is possible for this to fail, leaving the server in maintenance mode on reboot! echo "Exiting maintenance mode..." if [ $TEST -eq 0 ]; then esxcli system maintenanceMode set -e false -t 0 fi
Note: the No Joke IT web-site has vanished, so only the [Wayback] and [Archive.is] links of it still work. The github code was still there at the time of writing.
Via: [Wayback] Solved: Read only Files – VMware Technology Network VMTN
Related: Some notes on replacing parts of a text file with template text using sed on a Busybox system.
–jeroen
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Chocolatey 1.0.0 got released last week (chocolatey/choco · GitHub)
Posted by jpluimers on 2022/03/24
Last week finally there was the stable [Wayback/Archive] Release version 1.0.0 · chocolatey/choco · GitHub.
So I fixed the Wikipedia page
- from: [Wayback/Archive] Chocolatey – Wikipedia (only mentioning 0.12.1 preview release)
- to: [Wayback/Archive] Chocolatey – Wikipedia (now including the 1.0.0 stable release)
It was a few days after the 11th birthday “Celebration”: [Wayback/Archive] Chocolatey Software Blog | This One Goes To 11! Celebrating 11 Years Of Chocolatey. Not a really festive post, though it does have a really nice overview of 11 years of Chocolatey history and clearly showing the momentum of it has been a few years behind us.
The thing is: hardly anybody noticed the celebration nor the 1.0.0 release. Being at various 0.* versions for like a decade makes people not follow sudden version bumps closely. I only noticed when updating a bunch of testing VMs of which one had a problem, so I inspected the logs and saw the 1.0.0 version.
So these recent tweets did not gain much attention:
- 20220321: [Wayback/Archive] Chocolatey on Twitter: “Join us in ONE HOUR to mark a milestone in Chocolatey history! No excuses. Take the dog out. Grab a coffee. Sit down. Watch us on Twitch. 4pm UTC. “
- 20220320: [Wayback/Archive] Chocolatey on Twitter: “Join the Chocolatey Team TOMORROW, 21 March, and mark a milestone in Chocolatey history! On Twitch. 4pm UTC. And join our Discord server for a little thank you to our Community. “
- 20220317: [Wayback/Archive] Chocolatey on Twitter: “ICYMI we have a … special event marking a milestone in Chocolatey history this Monday, 21 March at 4pm UTC. Join @pauby, @gep13 and the @chocolateynuget Team. And join our Discord server for a little thank you to our Community. “
Anyway: the release notes indicate a few things scheduled for 2.0.0. Given the sudden 0.12.0 -> 1.0.0 bump, I have no clue far (or near!) in the future that will be.
It is kind of both a saddening and relieved feeling: like for instance Stack Overflow/Stack Exchange (both in the same age cohort as Chocolatey), Chocolatey is just there and mostly works.
–jeroen
Posted in .NET, Batch-Files, C#, Chocolatey, CommandLine, Development, Power User, PowerShell, PowerShell, Scripting, Software Development, Windows | Leave a Comment »
Some notes on replacing parts of a text file with template text using sed on a Busybox system
Posted by jpluimers on 2022/03/17
Note before you think about putting stuff in
/etc/rc.local.d/local.sh: that script will not be executed when UEFI booting.
In a very lightweight Busybox system, I wanted to modify some configuration files automatically using fragments stored in template files.
The system has diff, but no patch.
The basic idea is to use sed to insert the template files into certain spots of the configuration file when certain marker texts are not present. So I want the opposite of [Wayback] Hey Stephen Wood: Try patch instead of sed in shell scripts.
Basically the idea is a poor-man’s patch, described in Too bad: ESXi busybox has diff, but not patch « The Wiert Corner – irregular stream of stuff.
Some links that might help me with this:
- [Wayback] regex – Using sed to delete all lines between two matching patterns – Stack Overflow (thanks users [Wayback] Lri and [Wayback] Akito)
- [Wayback/Archive.is] akito-libbash/setup.bash at dd91364083f13d1132d68489172bbce664b9c9c0 · theAkito/akito-libbash an actual example close to what I am after (it appends, I want to keep the end of the file in tact).
- [Wayback] bash – removing lines between two patterns (not inclusive) with sed – Stack Overflow
- [Wayback] shell – How to append multiple lines to a file – Unix & Linux Stack Exchange
- [Wayback] text processing – How to insert the content of a file into another file before a pattern (marker)? – Unix & Linux Stack Exchange
One alternative would have been to use ed (which is part of the normal Busybox), but ESXi Busybox omits ed like it omits patch.
Too bad that sed commands are too different from ed commands, as I could have used diff -e on another system based on ideas here:
- [Wayback] How can I create a sed command line from diff? – Super User
- [Wayback] Using diff -e Option to Create a Baseline diff File – Alvin Bunk
I might give it one more go, as vi is sort of derived from ed via ex (see vi: Creation – Wikipedia), which means that vi “colon mode” (officially command mode: [Wayback] Vim documentation: cmdline) is very similar to ed.
Another alternative would be awk, but I have done so little work with it awk, that I’m hesitating to use a new tool. Some links:
And finally, ash could be used:
- [Wayback] text processing – Script matching literal pattern over multiple lines? – Unix & Linux Stack Exchange
The kind of modifications I am after
Below are a few links with examples of the kind of modifications I want to make. Most patch just /etc/rc.local.d/local.sh, but some others introduce other changes as well.
- [Wayback] Shell script to automatically power on a specific VM which is powered off – VirtuallyVTrue
- [Wayback] Enable the SSH shell permanently in VMWare ESXi 6.7.0 and above
- [Wayback] Executing Commands During Boot Up In ESXi 5.1
- [Wayback] How to clone ESXi setup – VMware Technology Network VMTN
- [Wayback] Article Detail: Many current.png.xxxx on the /tmp on ESXi host causes hostd to crash and disconnect (2031839)
- [Wayback] ESXi booting faster than your SAN
- [Wayback] Shell script to automatically power on a specific VM which is powered off – VirtuallyVTrue
- [Wayback/Archive.is] Help getting GhettoVCB & cron jobs working on ESXi 6. : vmware
- [Wayback/Archive.is] Want to modify Roots CronTab : vmware
- [Wayback/Archive.is] Script Share: ESXi 6.7 re-scan iSCSI after FreeNAS VM has booted : freenas
- [Archive.is] Wayback: VMware KB: Changing the port used by SSH on an ESXi 5.0 host
- [Wayback/Archive.is] Creating custom firewall rules in VMware ESXi 5.x (2008226)
- [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN
- [Wayback] Execute ESXCLI commands during ESXi startup – The Virtualist
Note that especially with networking settings, local.sh commands might not have any effect (for instance when having slow DHCP or other network issues), see for instance [Wayback/Archive.is] I’m running ESXi 5.5 and my persistent route in local.sh is not taking effect after boot. : vmware.
There is a very convoluted way around using local.sh by using the VIB authoring tool as described in [Wayback] How to create persistent firewall rules on ESXi. It requires lowering the software acceptance level to Community Supported (esxcli software acceptance set --level=CommunitySupported), which gives you a hard time installing ESXi updates.
I got that VIB idea from [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN, as:
The
local.shfile gets overwritten often with upgrades so it would mean another step during the process.
From the same thread comes [Wayback] Solved: Re: Persistent firewall rule – VMware Technology Network VMTN
set the sticky bit on your separate xml-file – then it will be backed up and persist through reboot:
chmod +trun backup manually before the first reboot:
/sbin/auto-backup.shbecause backup runs only once per hour
Within vSphere, one could use [Wayback] Configure ESXi Hosts with Host Profiles, but a standalone ESXi box is not part of vSphere, so that won’t work.
ESXi 7 and up
ESXi 7 makes the above harder as for instance user root cannot change file rights any more, so eventually I might revert to a VM that auto-boots when ESXi comes up, then patches the right files in place over PowerCLI (read-only) or SSH.
Need to give this some thought later:
- [Wayback] virtualization – ESXi 7.X file permissions – how to buypass new security measures – Server Fault
- [Archive.is] The root account can no longer change permissions or executable files in ESXi 7.0.x (78689)
- [Wayback/Archive.is] PowerCLI, ESXi 6.5 free and Start-VM: Current license or ESXi version prohibits execution of the requested operation : homelab
PowerCLI commands on the free version are limited to commands that are “read-only”, so you can only find out information rather than perform actions, if that makes sense.
So you can find out if a VM is turned on, find it’s uptime, but you can not turn it on or reboot it
This presumably is to prevent automation without a license…
Having a quick look at the VMware KB, it looks like PowerCLI was limited on the free version to “read-only” operations from 5.0 (I.e. when it went from ESX to ESXI) along with vCLI and vSphere-Perl, so for some time from the looks of it
–jeroen
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Awk, BusyBox, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, PowerCLI, Scripting, sed, sed script, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »





