The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘bash’ Category

Some bash parameter propagation links that hopefully will work with ash/dash too

Posted by jpluimers on 2021/10/27

For my link archive; I started with [Wayback] dash get all parameters quoted – Google Search:

–jeroen

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »

Multiple commands in one sudo: use “sudo sh -c ‘apt update && apt upgrade -y'”

Posted by jpluimers on 2021/09/03

So I won’t forget: [WayBack] @nixcraft on Twitter: Instead of typing the following on your Ubuntu/Debian/Mint Linux desktop: sudo apt update sudo apt upgrade -y Do to save typing and time at the CLI (add to your shell startup): alias update=”sudo sh -c ‘apt update && apt upgrade -y'” See for more info:

[WayBack] How to run multiple commands in sudo under Linux or Unix – nixCraft:

sudo syntax to run multiple commands

The syntax is:
sudo sh -c 'command1 && command2'
sudo -- sh -c 'command1 && command2'
sudo -u userNameHere -- sh -c 'command1; command2'
sudo -- sh -c 'command1; command2'
sudo -- bash -c 'command1; command2'
sudo -i -- 'command1; command2; command3'
sudo -i -- sh -c 'command1 && command2 && command3'

UNDERSTANDING SUDO COMMAND OPTIONS

  1. -- : A — signals the end of options and disables further option processing for sudo command.
  2. sh -c : Run sh shell with given commands
  3. 'apt-get update && sudo apt-get -y upgrade' First update repo and apply upgrades if update was successful.

A note about using sudo command in a shell script

Here is a sample shell script that shows how to use or run multiple commands with sudo:

#!/bin/bash
echo "Running commands as a root user..."
sudo -- -sh -c <<EOF
apt-get update
apt-get -y upgrade
apt-get -y install nginx 
apt-get -y remove nano
apt-get clean
echo "All done."
EOF

A note about using sudo with bash shell aliases

The syntax is as follows for shell aliases:

alias foo="sudo -- sh -c 'cmd1 && cmd2'"
alias bar='sudo -- sh -c "cmd1 && cmd2"'

–jeroen

Posted in *nix, *nix-tools, bash, Power User | Leave a Comment »

firewalld: show interfaces with their zone details and show zones in use

Posted by jpluimers on 2021/08/26

A while ago openSUSE switched to firewalld as a fronte-end for iptables. Tumbleweed was first in 2018, so I wrote a reminder: On my research list: migrate from OpenSuSE SuSEfirewall2 to firewalld « The Wiert Corner – irregular stream of stuff.

The core concept of firewalld is zones, which some people find hard to understand: [Archive.is/WayBack] Firewalld on Leap 15 – why is it so complicated ? : openSUSE.

Another concept is interfaces and how they bind to zones. [Wayback] Masquerading and Firewalls | Security Guide | openSUSE Leap 15.2 shows more of that.

The final concept is services that bind one or more aspects (like ports or addresses) to a service name [Wayback] Documentation – Manual Pages – firewalld.service | firewalld.

Other interesting bits of information:

Below are some examples on what I learned, especially finding details about active interfaces and the zones they are bound to.

All of them are based on:

  • the xargs shell trick (I known you can do some of them without the trick, but I try to use common patterns in my solution so I do not have to remember which boundary case fails
  • the echo -n trick to skip the newline output
  • the [WayBack] firewall-cmd options (which kind of care commands)
    • --get-active-zones:

      Print currently active zones altogether with interfaces and sources used in these zones. Active zones are zones, that have a binding to an interface or source. The output format is:

      zone1
        interfaces: interface1 interface2 ..
        sources: source1 ..
      zone2
        interfaces: interface3 ..
      zone3
        sources: source2 ..

      If there are no interfaces or sources bound to the zone, the corresponding line will be omitted.

    • --list-interfaces:

      List interfaces that are bound to zone zone as a space separated list. If zone is omitted, default zone will be used.

    • --get-zone-of-interface=<zone>:

      Print the name of the zone the interface is bound to or no zone.

    • --info-zone=<zone> (which shows far more information than the manual indicates):

      Print information about the zone zone. The output format is:

      zone
        interfaces: interface1 ..
        sources: source1 ..
        services: service1 ..
        ports: port1 ..
        protocols: protocol1 ..
        forward-ports: forward-port1 ..
        source-ports: source-port1 ..
        icmp-blocks: icmp-type1 ..
        rich rules: rich-rule1 ..

Two more notes before the examples:

  1. My first hunch was to use --list-all-zones, but that shows details of all un-used zones as well.
  2. I am not fully sure about the --list-interfaces to list *all* interfaces. I might replace this later with ls /sys/class/net (see [WayBack] linux – List only the device names of all available network interfaces – Super User).

Other useful commands

Besides lising zones and interfaces, you might be interested in services and ports:

# firewall-cmd --list-services
dhcpv6-client ssh
# firewall-cmd --list-ports

List used zones

The first only shows the zone names

# firewall-cmd --list-interfaces | xargs -I {} sh -c 'firewall-cmd --get-zone-of-interface={}'
public

The second both zones and interfaces:

# firewall-cmd --get-active-zones 
public
  interfaces: ens192

When there are no bound interfaces

OpenSuSE by default does not bind interfaces to zones; it means any interface uses the default zone. That means the --list-interfaces commands in this blog post fail.

You can check this behaviour by running this command:

# ls /sys/class/net | xargs -I {} sh -c 'echo -n "interface {} has zone " ; firewall-cmd --get-zone-of-interface={} | xargs -I [] sh -c "echo [] ; firewall-cmd --info-zone=[]"'
interface eth0 has zone no zone
interface lo has zone no zone
interface wlan0 has zone no zone

Alternatives:

  1. Finding the default zone
    # firewall-cmd --get-default-zone
    public
    
  2. Details of the default zone
    # firewall-cmd --info-zone=$(firewall-cmd --get-default-zone)
    public
      target: default
      icmp-block-inversion: no
      interfaces: 
      sources: 
      services: dhcpv6-client ssh
      ports: 
      protocols: 
      masquerade: no
      forward-ports: 
      source-ports: 
      icmp-blocks: 
      rich rules: 

You can see that here the public zone is marked default which means it binds to any interface that is not bound to a specific zone.

List used zone details

# firewall-cmd --list-interfaces | xargs -I {} sh -c 'firewall-cmd --get-zone-of-interface={} | xargs -I [] sh -c "firewall-cmd --info-zone=[]"'
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources: 
  services: dhcpv6-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

List interfaces and their zones:

# firewall-cmd --list-interfaces | xargs -I {} sh -c 'echo -n "interface {} has zone " ; firewall-cmd --get-zone-of-interface={}'
interface ens192 has zone public

List interfaces and their zone details:

# firewall-cmd --list-interfaces | xargs -I {} sh -c 'echo -n "interface {} has zone " ; firewall-cmd --get-zone-of-interface={} | xargs -I [] sh -c "echo [] ; firewall-cmd --info-zone=[]"'
interface ens192 has zone public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources: 
  services: dhcpv6-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

Verifying if dns service is available, then allow it on public

Verify if a DNS is in the enabled services:

# firewall-cmd --list-services
dhcpv6-client ssh

Here no DNS service is enabled, so I need to figure out if any DNS service is available to be enabled.

This lists all the services that can be enabled in a zone:

# firewall-cmd --get-services

On my system, this returned the following list:

RH-Satellite-6 amanda-client amanda-k5-client amqp amqps apcupsd audit bacula bacula-client bb bgp bitcoin bitcoin-rpc bitcoin-testnet bitcoin-testnet-rpc bittorrent-lsd ceph ceph-mon cfengine cockpit condor-collector ctdb dhcp dhcpv6 dhcpv6-client distcc dns dns-over-tls docker-registry docker-swarm dropbox-lansync elasticsearch etcd-client etcd-server finger freeipa-4 freeipa-ldap freeipa-ldaps freeipa-replication freeipa-trust ftp ganglia-client ganglia-master git grafana gre http https imap imaps ipp ipp-client ipsec irc ircs iscsi-target isns jenkins kadmin kdeconnect kerberos kibana klogin kpasswd kprop kshell ldap ldaps libvirt libvirt-tls lightning-network llmnr managesieve matrix mdns memcache minidlna mongodb mosh mountd mqtt mqtt-tls ms-wbt mssql murmur mysql nfs nfs3 nmea-0183 nrpe ntp nut openvpn ovirt-imageio ovirt-storageconsole ovirt-vmconsole plex pmcd pmproxy pmwebapi pmwebapis pop3 pop3s postgresql privoxy prometheus proxy-dhcp ptp pulseaudio puppetmaster quassel radius rdp redis redis-sentinel rpc-bind rsh rsyncd rtsp salt-master samba samba-client samba-dc sane sip sips slp smtp smtp-submission smtps snmp snmptrap spideroak-lansync spotify-sync squid ssdp ssh steam-streaming svdrp svn syncthing syncthing-gui synergy syslog syslog-tls telnet tentacle tftp tftp-client tile38 tinc tor-socks transmission-client upnp-client vdsm vnc-server wbem-http wbem-https wsman wsmans xdmcp xmpp-bosh xmpp-client xmpp-local xmpp-server zabbix-agent zabbix-server

I was searching to see if dns was available, so I split the string with tr, then searced with grep:

# firewall-cmd --get-services | tr " " "\n" | grep dns
dns
dns-over-tls
mdns

To get details, use the firewall-cmd --info-service=servicename like this:

# firewall-cmd --get-services | tr " " "\n" | grep dns | xargs -I [] sh -c "firewall-cmd --info-service=[]"
dns
  ports: 53/tcp 53/udp
  protocols: 
  source-ports: 
  modules: 
  destination: 
  includes: 
dns-over-tls
  ports: 853/tcp
  protocols: 
  source-ports: 
  modules: 
  destination: 
  includes: 
mdns
  ports: 5353/udp
  protocols: 
  source-ports: 
  modules: 
  destination: ipv4:224.0.0.251 ipv6:ff02::fb
  includes: 

So for named (bind), I need the dns service to be enabled:

# firewall-cmd --zone=public --add-service=dns --permanent
success

Now a –list-services will not show dns as we changed the --permanent configuration, not the current configuration:

# firewall-cmd --list-services
dhcpv6-client ssh

So you need to --reload the --permanent settings:

# firewall-cmd --list-services --permanent
dhcpv6-client dns ssh
# firewall-cmd --reload
success
# firewall-cmd --list-services
dhcpv6-client dns ssh

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, iptables, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux, Tumbleweed, xargs | Leave a Comment »

linux – How can I find all hardlinked files on a filesystem? – Super User

Posted by jpluimers on 2021/08/25

[WayBack] linux – How can I find all hardlinked files on a filesystem? – Super User

use the following line (for sure you have to replace /PATH/FOR/SEARCH/ with whatever you want to search):

find /PATH/FOR/SEARCH/ -xdev -printf '%i\t%n\t%p\n' | fgrep -f <(find . -xdev -printf '%i\n' | sort -n | uniq -d) | sort -n

this scans the filesystem only once, shows inode, number of hardlinks and path of files with more than one hardlink and sorts them according to the inode.

if you are annoyed by error messages for folders you aren’t allowed to read, you can expand the line to this:

find /PATH/FOR/SEARCH/ -xdev -printf '%i\t%n\t%p\n' 2> /dev/null | fgrep -f <(find . -xdev -printf '%i\n' 2> /dev/null | sort -n | uniq -d) | sort -n

It uses these commands:

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, fgrep, find, Power User, Scripting, Software Development | 1 Comment »

bash – Search for a previous command with the prefix I just typed – Unix & Linux Stack Exchange

Posted by jpluimers on 2021/08/18

[WayBack] bash – Search for a previous command with the prefix I just typed – Unix & Linux Stack Exchange answered by [WayBack] John1024:

What you are looking for is Ctrl-R.

Type Ctrl-R and then type part of the command you want. Bash will display the first matching command. Keep typing CtrlR and bash will cycle through previous matching commands.

To search backwards in the history, type Ctrl-S instead. (If Ctrl-S doesn’t work that way for you, that likely means that you need to disable XON/XOFF flow control: to do that, run stty -ixon.)

This is documented under “Searching” in man bash.

Comment by [WayBack] HongboZhu:

Ctrl-Q to quit the frozen state, if you already hit Ctrl-S without turning off flow control first and got your terminal frozen.

A far more elaborate answer with many other tips is from [WayBack] Peter Cordes:

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

-r argument to pipe (no argument for MacOS)- If no input is given to xargs, don’t let xargs run the utility – Unix & Linux Stack Exchange

Posted by jpluimers on 2021/07/28

TL;DR

There is a non-standard -r option to xargs that allows it to skip executing when there are no arguments at all.

On some operating systems, the -r is default.

MacOS has no -r, but does not execute xargs if there are no arguments given.

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development, xargs | Leave a Comment »

Filezilla: figuring out the cause of “Connection timed out after 20 seconds of inactivity”

Posted by jpluimers on 2021/07/21

On one of my Raspberry Pi boxes, somehow I could not access files over SFTP (SSH File Transfer Protocol) via FileZilla.

I would consistently get this error:

"Connection timed out after 20 seconds of inactivity"

Figuring the exact cause took a while.

TL;DR: SFTP uses an interactive non-login shell, then interprets the output from that shell. For that kind of shell, ensure few or none scripts run that output text.

These links finally got me to the cause

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Communications Development, Conference Topics, Conferences, Development, Event, Internet protocol suite, Power User, Scripting, SFTP, Software Development, SSH, TCP | Leave a Comment »

Busybox sh (actually ash derivative dash): checking exit codes

Posted by jpluimers on 2021/04/20

Even if you include a double quotes "sh" in a Google search to force only sh (in the early days this was the Thompson shell, but nowadays usually a Bourne shell or derivative) results, almost all unix like scripting examples you find are based on bash (the Bourne again shell), so I was glad I dug a bit deeper into what the actual Busybox shell is.

I wanted to know which shell Busybox uses and what capabilities it has, as ESXi ships with this very slimmed down set of tools (called applets in Busybox speak).

It does not even include ssh: that gap is often filled by [Wayback] Dropbear SSH, which was used by ESXi and named dbclient (I think with ESXi 6.0 it was replaced with a more regular ssh implementation): [Wayback] How to compile a statically linked rsync binary for ESXi.

Busybox shell source code is at [Wayback] ash.c\shell – busybox – BusyBox: The Swiss Army Knife of Embedded Linux and indicates the shell is the ash (the Almquist shell) derivative dash (yes, you guessed it right: the Debian Almquist shell), ported from NetBSD and debianized:

 * Copyright (c) 1997-2005 Herbert Xu <herbert@gondor.apana.org.au>
 * was re-ported from NetBSD and debianized.
...
//config:   The most complete and most pedantically correct shell included with
//config:   busybox. This shell is actually a derivative of the Debian 'dash'
//config:   shell (by Herbert Xu), which was created by porting the 'ash' shell
//config:   (written by Kenneth Almquist) from NetBSD.

nx like systems have a shell hell similar to Windows DLL hell: there are too many, and their differences and be both subtle and frustrating. To get a feel, browse through Source: Comparison of command shells – Wikipedia (yes, some shells from other operating environments like DOS, OS/2, VMS and Windows, but the majority is nx).

Since ash is sufficiently different from bash (for example [Wayback] ash – exit code for a piped process), I always want to know what shell code (which often comes from bash as it is so ubiquitous) will work.

There is hardly any shell documentation at the Busybox site. There is [Wayback] BusyBox – The Swiss Army Knife of Embedded Linux, the source code at [Wayback] ash.c\shell – busybox – BusyBox: The Swiss Army Knife of Embedded Linux does not offer much either,

A manual page of it is at [Archive.is] ash(1) [minix man page]. There you see the age: back then, “exit status” is used where nowadays many people would use “exit code”. It does not explain how to check for specific exit codes.

Because ash is derived from the Bourne shell, this page was of great help for me to grasp exit code handing: [Wayback] Exit Codes – Shell Scripting Tutorial

A Bourne Shell Programming / Scripting Tutorial for learning about using the Unix shell.

Here two examples from that page to get me going:

#!/bin/sh
# Second attempt at checking return codes
grep "^${1}:" /etc/passwd > /dev/null 2>&1
if [ "$?" -ne "0" ]; then
  echo "Sorry, cannot find user ${1} in /etc/passwd"
  exit 1
fi
USERNAME=`grep "^${1}:" /etc/passwd|cut -d":" -f1`
NAME=`grep "^${1}:" /etc/passwd|cut -d":" -f5`
HOMEDIR=`grep "^${1}:" /etc/passwd|cut -d":" -f6`

echo "USERNAME: $USERNAME"
echo "NAME: $NAME"
echo "HOMEDIR: $HOMEDIR"

and

#!/bin/sh
# A Tidier approach

check_errs()
{
  # Function. Parameter 1 is the return code
  # Para. 2 is text to display on failure.
  if [ "${1}" -ne "0" ]; then
    echo "ERROR # ${1} : ${2}"
    # as a bonus, make our script exit with the right error code.
    exit ${1}
  fi
}

### main script starts here ###

grep "^${1}:" /etc/passwd > /dev/null 2>&1
check_errs $? "User ${1} not found in /etc/passwd"
USERNAME=`grep "^${1}:" /etc/passwd|cut -d":" -f1`
check_errs $? "Cut returned an error"
echo "USERNAME: $USERNAME"
check_errs $? "echo returned an error - very strange!"

This basically means that status code handling is the same as in bash, so constructs can be used like [Wayback] bash – How to check the exit status using an if statement – Stack Overflow:

$? is a parameter like any other. You can save its value to use before ultimately calling exit.

exit_status=$?
if [ $exit_status -eq 1 ]; then
    echo "blah blah blah"
fi
exit $exit_status

Read the rest of this entry »

Posted in *nix, *nix-tools, ash/dash, ash/dash development, bash, bash, BusyBox, Development, Power User, Scripting, Software Development, ssh/sshd | 1 Comment »

linux – How can I execute a series of commands in a bash subshell as another user using sudo? – Stack Overflow

Posted by jpluimers on 2021/03/31

Based on [WayBack] linux – How can I execute a series of commands in a bash subshell as another user using sudo? – Stack Overflow:

alias restart-spotlight-service-as-root="sudo bash -c 'echo stop;launchctl stop com.apple.metadata.mds;echo start;launchctl start com.apple.metadata.mds;echo started'"

The bold bits above sudo bash -c 'echo stop;launchctl stop com.apple.metadata.mds;echo start;launchctl start com.apple.metadata.mds;echo started' allow the commands between single quotes to executed in one new bash shell under sudo.

–jeroen

Posted in *nix, *nix-tools, Apple, bash, bash, Development, Mac OS X / OS X / MacOS, Power User, Scripting, Software Development | Leave a Comment »

explainshell.com: parse and explain just about any shell command

Posted by jpluimers on 2021/02/17

I bumped into the tremendously site [WayBack] explainshell.com – match command-line arguments to their help text only after documenting the relevant cURL options of yesterdays post on checking your CertBot domain expiration dates.

The site allows put in a shell command-line to see the help text that, including matches for each argument.

It works so well because it parses both the shell command-line and the man pages, then constructs a web-page linking the relevant man page content to the shell command-line in the correct shell command-line order.

The explainshell has a counterpart showthedocs (both are open source) for explaining other languages (on the one hand more extended as it goes much deeper into parsing for instance SQL, on the other hand more limited as it only supports a few languages). More on showthedocs later.

The links

The parsing results

The first bit below is just the text output, and the second bit the screenshot, of a relatively simple command like [WayBack] explainshell.com – curl -fsSL example.org:

curl(1) -fsSL example.org
transfer a URL
-f, --fail
       (HTTP)  Fail  silently  (no  output at all) on server errors. This is mostly done to better enable
       scripts etc to better deal with failed attempts. In normal cases  when  a  HTTP  server  fails  to
       deliver  a  document,  it  returns an HTML document stating so (which often also describes why and
       more). This flag will prevent curl from outputting that and return error 22.

       This method is not fail-safe and there are occasions where non-successful response codes will slip
       through, especially when authentication is involved (response codes 401 and 407).
-s, --silent
       Silent or quiet mode. Don't show progress meter or error messages.  Makes Curl mute.
-S, --show-error
       When used with -s it makes curl show an error message if it fails.
-L, --location
       (HTTP/HTTPS) If the server reports that the requested page  has  moved  to  a  different  location
       (indicated  with  a Location: header and a 3XX response code), this option will make curl redo the
       request on the new place. If used together with -i, --include or  -I,  --head,  headers  from  all
       requested pages will be shown. When authentication is used, curl only sends its credentials to the
       initial host. If a redirect takes curl to a different host, it won't  be  able  to  intercept  the
       user+password.  See  also  --location-trusted  on  how to change this. You can limit the amount of
       redirects to follow by using the --max-redirs option.

       When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will
       do  the  following  request  with a GET if the HTTP response was 301, 302, or 303. If the response
       code was any other 3xx code, curl will re-send the following request  using  the  same  unmodified
       method.
source manpages: curl

The screenshot is even more impressive:

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »