Posted by jpluimers on 2022/02/17
I always facepalm myself after looking up this: [Wayback] Can bash show a function’s definition? – Stack Overflow (thanks [Wayback] Benjamin Bannier!):
…
Use
type. Iffoobaris e.g. defined in your~/.profile:$ type foobar foobar is a function foobar { echo "I'm foobar" }…
typewill also expand aliases, which is a nice bonus :) – [Wayback] Esdras Lopez
–jeroen
Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »
Posted by jpluimers on 2022/02/09
As promised mid last year in “fixing” ESXi “rsync error: error allocating core memory buffers (code 22) at util2.c(106) [sender=3.1.2]”, I would follow up on building a static rsync for ESXi one day.
So below a few links on how to do this, roughly in the order I found them (most via [Wayback] vmware rsync “3.1.2” static – Google Search):
Especially the last link has a great set of steps on how to build manually.
Boy I forgot how long ago CentOS 3.9 was: [Wayback] [CentOS-announce] CentOS 3.9 is released for i386 and x86_64 Read the rest of this entry »
Posted in *nix, *nix-tools, CentOS, Development, Linux, Power User, RedHat, Software Development | 2 Comments »
Posted by jpluimers on 2022/01/27
First the script that display messages for all virtual machines, vim-cmd-display-messages-for-all-VMs.sh:
#!/bin/sh vmids=`vim-cmd vmsvc/getallvms | sed -n -E -e "s/^([[:digit:]]+)\s+((\S.+\S)?)\s+(\[\S+\])\s+(.+\.vmx)\s+(\S+)\s+(vmx-[[:digit:]]+)\s*?((\S.+)?)$/\1/p"` for vmid in ${vmids} ; do powerState=`vim-cmd vmsvc/power.getstate ${vmid} | sed '1d'` name=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/\(vim.vm.ConfigInfo\) \{/,/files = \(vim.vm.FileInfo\) \{/ s/^ +name = "(.*)",.*?/\1/p'` vmPathName=`vim-cmd vmsvc/get.config ${vmid} | sed -n -E -e '/files = \(vim.vm.FileInfo\) \{/,/tools = \(vim.vm.ToolsConfigInfo\) \{/ s/^ +vmPathName = "(.*)",.*?/\1/p'` echo "Messages for VM with id ${vmid} which has power state ${powerState} (name = ${name}; vmPathName = ${vmPathName})." vim-cmd vmsvc/message ${vmid} done exit 0
It is very similar to vim-cmd-reload-all-VM-vmx-configurations.sh from Source: ESXi: reloading all virtual machines from their (potentially) vmx files.
No message” or are about “This virtual machine may have been moved or copied.“If there is no available message, then you always get the stock message No message., so this is something you can use as a check in scripts.
Posted in *nix, *nix-tools, ArchiveTeamWarrior, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/01/27
Based on [Wayback] domain name system – Overriding some DNS entries in BIND for internal networks – Server Fault and some further reading, there seem to be two ways used in these scenarios:
I wonder how that would interact best with Pi-Hole based solutions. Would it be best to have your local network use the Pi-Hole server, then have the Pi-Hole server obtain the DNS information it cannot resolve through one of the above solutions? Or would other solutions work better?
So here are a few links:
This document describes a method for expressing DNS response policy inside a specially constructed DNS zone, and for recursive name servers to use such policy to return modified results to DNS clients. The modified DNS results can stop access to selected HTTP servers, redirect users to “walled gardens”, block objectionable email, and otherwise defend against attack. These “DNS Firewalls” are widely used in fighting Internet crime and abuse.
Learn how you can install your own DNS server to protect your devices from phishing, malware and ransomware
Unbound is a validating, recursive, caching DNS resolver. It is designed to be fast and lean and incorporates modern features based on open standards. Late 2019, Unbound has been rigorously audited, which means that the code base is more resilient than ever. To help increase online privacy, Unbound supports DNS-over-TLS …
The 1.10.0 release has RPZ support and serve stale functionality according to draft draft-ietf-dnsop-serve-stale-10. And a number of other, smaller, features, and bug fixes.
A merged collection of hosts from reputable sources.
The public blocklists used by Pi-Hole are the following:
- [Wayback] StevenBlack
- [Wayback] MalwareDom
- [Wayback] Cameleon
- [Wayback] ZeusTracker
- Disconnect.me Tracking
- Disconnect.me Ads
- [Wayback] HostsFile
I wrote a [Wayback] script that will fetch the blocklists content, parse it and create a local zone file for unbound(8). That file will contain all the blocked domains and use the redirect answer to resolve those as invalid.
Pi-Hole seems not interested in RPZ: [Wayback] Implement Response Zone Policies (NXDOMAIN) for end-user performance increase – Feature Requests / Implemented – Pi-hole Userspace
Pi-Hole default blacklist is mentioned in [Wayback/Archive.is] pi-hole/basic-install.sh at master · pi-hole/pi-hole (look for adlistFile which defaults to [Wayback/Archive.is] StevenBlack/hosts: 🔒 Consolidating and extending hosts files from several well-curated sources. Optionally pick extensions for porn, social media, and other categories.).
Since I need this for ESXi: [Wayback/Archive.is] Let’s Encrypt SSL for ESXi
–jeroen
Posted in *nix, *nix-tools, bind-named, DNS, Internet, Linux, Power User | Leave a Comment »
Posted by jpluimers on 2022/01/25
In Determining the ESXi installation type (2014558) | VMware KB, I also showed how to backup the configuration and download it.
Sometimes you want an ISO 8601 time-stamped local tarball just in case you want to revert to it at a later stage.
First a small recap on how to get the tarball, download location and temporary location in the first place (it will be automatically deleted from the temporary location):
# vim-cmd hostsvc/firmware/sync_config # vim-cmd hostsvc/firmware/backup_config Bundle can be downloaded at : http://*/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz # find /scratch/downloads/ -name *.tgz /scratch/downloads/52aa233b-5db4-2298-5e1b-f510b2cd149f/configBundle-ESXi-X10SRH-CF.tgz
Goal is to get the download filename and save it to a different folder and embed the ISO 8601 timestamp in the filename.
Like many scripts, sed and regular expressions come to the rescue once more, just like in ESXi ash/dash/busybox shell getting current timestamp in UTC ISO8601 format without colons or dashes (which we will need anyway because of the ISO 8601 time stamp, and a bit of fiddling at regex101.com/r/NyrzKF
# SCRATCH_CONFIG_BUNDLE_NAME=$(vim-cmd hostsvc/firmware/backup_config | sed -n -E -e "s/^(Bundle can be downloaded at : http://*)(/downloads/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}/configBundle-.+?)(.tgz)$//scratch23/p") # echo "SCRATCH_CONFIG_BUNDLE_NAME: '${SCRATCH_CONFIG_BUNDLE_NAME}'" SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz' # date --utc -I'seconds' --reference "${SCRATCH_CONFIG_BUNDLE_NAME}" 2021-05-09T17:44:42UTC
Note:
sed regular, expression, [[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12} matches a GUID.date command, the --reference parameter must be last.backup_config commands in succession:(vim.fault.TooManyWrites) { faultCause = (vmodl.MethodFault) null, faultMessage = msg = "Received SOAP response fault from []: syncConfiguration fault.TooManyWrites.summary" }
You can see this in the hostd.log, which on my system is in /scratch/log/hostd.log where it says I can retry in 2031 seconds (slightly more than half an hour):
2021-05-09T19:34:13.420Z verbose hostd[A703B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab3] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp map. 2021-05-09T19:34:13.420Z info hostd[A703B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=vim-cmd-27-2ab3] Event 196 : User root@127.0.0.1 logged in as VMware-client/6.5.0 2021-05-09T19:34:13.420Z verbose hostd[A703B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab3] RecordOp ADD: sessionList["52f6d64e-0a35-c1d7-de97-624d234bc2a7"], ha-sessionmgr. Applied change to temp map. 2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-27-2ab5 user=root] Task Created : haTask--vim.host.FirmwareSystem.syncConfiguration-114207804 2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ADD: recentTask["haTask--vim.host.FirmwareSystem.syncConfiguration-114207804"], ha-taskmgr. Applied change to temp map. 2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ASSIGN: info, haTask--vim.host.FirmwareSystem.syncConfiguration-114207804. Applied change to temp map. 2021-05-09T19:34:13.423Z error hostd[9AC1B70] [Originator@6876 sub=Hostsvc.FirmwareSystem opID=vim-cmd-27-2ab5 user=root] Failed to sync configuration. Too many writes. Next sync possible in 2031 sec. 2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Default opID=vim-cmd-27-2ab5 user=root] AdapterServer caught exception: vim.fault.TooManyWrites 2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Vimsvc.TaskManager opID=vim-cmd-27-2ab5 user=root] Task Completed : haTask--vim.host.FirmwareSystem.syncConfiguration-114207804 Status error 2021-05-09T19:34:13.423Z verbose hostd[9AC1B70] [Originator@6876 sub=PropertyProvider opID=vim-cmd-27-2ab5 user=root] RecordOp ASSIGN: info, haTask--vim.host.FirmwareSystem.syncConfiguration-114207804. Applied change to temp map. 2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Activation [N5Vmomi10ActivationE:0x0987e6e0] : Invoke done [syncConfiguration] on [vim.host.FirmwareSystem:ha-firmwareSystem] 2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Throw vim.fault.TooManyWrites 2021-05-09T19:34:13.423Z info hostd[9AC1B70] [Originator@6876 sub=Solo.Vmomi opID=vim-cmd-27-2ab5 user=root] Result: --> (vim.fault.TooManyWrites) { --> faultCause = (vmodl.MethodFault) null, --> faultMessage = --> msg = "" --> }
Not few people have bumped into this, the only other I could find through [Wayback] “vim.fault.TooManyWrites” “syncConfiguration” – Google Search is [Archive.is] mal wieder purple Screen – VMware-Forum.
Figuring out the various parts of the SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/5271677d-97db-30dc-673d-b99e61bed251/configBundle-ESXi-X10SRH-CF.tgz' is like at regex101.com/r/J4yU72, regex101.com/r/uID9xs and regex101.com/r/o8a4Am:
CONFIG_BUNDLE_DIRECTORY_NAME=$(echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\1/p") CONFIG_BUNDLE_FILE_NAME=$( echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\2/p") CONFIG_BUNDLE_DOT_EXTENSION=$( echo "${SCRATCH_CONFIG_BUNDLE_NAME}" | sed -n -E -e "s/(\/scratch\/downloads\/[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}\/)(configBundle-.+?)(.tgz)$/\3/p") echo "CONFIG_BUNDLE_DIRECTORY_NAME: '${CONFIG_BUNDLE_DIRECTORY_NAME}'" echo "CONFIG_BUNDLE_FILE_NAME: '${CONFIG_BUNDLE_FILE_NAME}'" echo "CONFIG_BUNDLE_DOT_EXTENSION: '${CONFIG_BUNDLE_DOT_EXTENSION}'"
Output is like this:
SCRATCH_CONFIG_BUNDLE_NAME: '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/configBundle-ESXi-X10SRH-CF.tgz' CONFIG_BUNDLE_DIRECTORY_NAME: '/scratch/downloads/528f9f5a-0123-f022-2b4d-a5c2e595c51a/' CONFIG_BUNDLE_FILE_NAME: 'configBundle-ESXi-X10SRH-CF' CONFIG_BUNDLE_DOT_EXTENSION: '.tgz'
Full backup-config-to-ESXi_configuration_backup-directory.sh script:
Posted in *nix, *nix-tools, ash/dash, ash/dash development, cron/crontab, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/01/21
Since dd on Apple does not support progress parameters via [Wayback] macos dd progress – Google Search:
A nice way I found to get progress indication whilst still being able to benefit from the huge speed increase in using
/dev/rdiskXis to install a tool calledpv, also known as [WayBack] Pipe Viewer.Example:
sudo dd if=/dev/rdiskX bs=1m | pv -s 64G | sudo dd of=/dev/rdiskY bs=1mAnother way to achieve something similar would be to use brew to install
coreutils, which will come with a newer version of dd that supports the status option.Example:
gdd if=/dev/diskX of=/dev/diskY bs=1m status=progress
You just need to enter a controlT character from the keyboard while the
ddcommand is executing.By pressing the controlT character, you are sending the same
SIGINFOsignal to theddcommand that the commandpkill -INFO -x ddsends.
dditself doesn’t provide a progress bar. You may estimate the progress of the dd copy process by adding a pkill -INFO command though.Example:
dd if=/dev/zero of=/dev/null bs=64m count=1000 & while pkill -INFO -x dd; do sleep 1; done
signal siginfo is coupled to key-combination CTRL-T. No need to use kill, you can just type CTRL-T in the terminal window where dd is running.
–jeroen
Posted in *nix, *nix-tools, Apple, iMac, Mac, Mac OS X / OS X / MacOS, MacBook, MacBook Retina, MacBook-Air, MacBook-Pro, MacMini, Power User | Leave a Comment »
Posted by jpluimers on 2022/01/21
The proper way is not manually changing /etc/hostname, but running this::
hostnamectl set-hostname host
[Wayback] How to change system hostname in SUSE
Background information in [Wayback] linux – What’s the point of the hostnamectl command? – Unix & Linux Stack Exchange (with a great answer by [Wayback] slm, edited by me for Wayback machine links):
Posted in *nix, *nix-tools, Linux, openSuSE, Power User, SuSE Linux, Tumbleweed | Leave a Comment »
Posted by jpluimers on 2022/01/19
A few links and notes:
Run these commands to change the hostname in ESXi 5.x, ESXi 6.x,ESXi 7.x, using the command line:
- esxcli system hostname set –host=hostname
- esxcli system hostname set –fqdn=fqdn
Cause
The name resolution parameters were not properly configured during the installation of the ESX host.
This issue is caused by a mismatch between the FQDN that was configured as the PNID during the vCenter Server deployment and the hostname that is currently configured.
I had a mismatch happen because of the second entry: a host configured in a different domain than it was deployed to.
Here are the commands to list and change the hosts name, domain and fqdn:
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/01/18
Almost two years ago, GitHub – facebook/osquery: SQL powered operating system instrumentation, monitoring, and analytics published from the automatic blog queue.
It was in the midst of my rectum cancer treatment, so I was glad the blog queue back then was still about 18 months deep.
This meant I looked into osquery in 2018, which I remember because I needed it on MacOS as I did not want to remember the syntax for MacOS specific commands on getting system information. It also coincides with how much my repository fork was behind: [Wayback: jpluimers/osquery commits/Archive: jpluimers/osquery commits].
Fast forward to now, the breath of systems I’m involved with has widened, so I was glad to see that Kristian Köhntopp mentioned it:
So time to try it again (:
The links he mentioned:
osqueryis an operating system instrumentation framework for Windows, OS X (macOS), Linux, and FreeBSD. The tools make low-level operating system analytics and monitoring both performant and intuitive.
The high-performance and low-footprint distributed host monitoring daemon,osqueryd, allows you to schedule queries to be executed across your entire infrastructure. The daemon takes care of aggregating the query results over time and generates logs which indicate state changes in your infrastructure. You can use this to maintain insight into the security, performance, configuration, and state of your entire infrastructure.osqueryd‘s logging can integrate into your internal log aggregation pipeline, regardless of your technology stack, via a robust plugin architecture.The interactive query console,osqueryi, gives you a SQL interface to try out new queries and explore your operating system. With the power of a complete SQL language and dozens of useful tables built-in,osqueryiis an invaluable tool when performing incident response, diagnosing a systems operations problem, troubleshooting a performance issue, etc.
Main site: [Wayback/Archive] osquery | Easily ask questions about your Linux, Windows, and macOS infrastructure
Repository: [Wayback/Archive] osquery/osquery: SQL powered operating system instrumentation, monitoring, and analytics.
–jeroen
Posted in *nix, *nix-tools, Apple, Development, DevOps, Facebook, Infrastructure, Mac, Mac OS X / OS X / MacOS, Power User, SocialMedia, Software Development, Windows | Leave a Comment »
Posted by jpluimers on 2022/01/18
Since ESXi boots from RAM, most files in /etc are not persisted after modification.
The files that are persisted, are only persisted once every hour by auto-backup.sh, so better run auto-backup.sh by hand if you want to reboot after changing them.
The auto-backup.sh script is ran every hour at 1 minute past the hour as per below crontab.
Default ESXi crontab in
/var/spool/cron/crontabs/root:#min hour day mon dow command 1 1 * * * /sbin/tmpwatch.py 1 * * * * /sbin/auto-backup.sh 0 * * * * /usr/lib/vmware/vmksummary/log-heartbeat.py */5 * * * * /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh 00 1 * * * localcli storage core device purgeSchedules deciphered via [Wayback] Crontab.guru – The cron schedule expression editor:
- [Wayback] Every day at
01:01: “This module removes stale temporary files”1 1 * * * /sbin/tmpwatch.py- [Wayback] Every hour at
*:01: saves backup to/bootbank/state.tgz.1 * * * * /sbin/auto-backup.sh- [Wayback] Every hour at
*:00logs heartbeat messages to/var/log/vmksummary.loglike2021-02-23T19:00:02Z heartbeat: up 577d2h37m16s, 9 VMs; [[2802426 vmx 4194304kB] [6176344 vmx 4194304kB] [68997 vmx 8388608kB]] [[2802426 vmx 0%max] [6176344 vmx 0%max] [68997 vmx 0%max]]0 * * * * /usr/lib/vmware/vmksummary/log-heartbeat.py- [Wayback] Every
5thminute logs to/var/log/hostd-probe.log.*/5 * * * * /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh- [Wayback] Every day at
01:00: Removes storage devices which have not been seen in some time interval.00 1 * * * localcli storage core device purgeNote that
localclicommands are the same asesxcli; foresxcli, a runninghostdis required;localclican run withouthostd. See:
- [Wayback] VCDX #200 Blog of one VMware Infrastructure Designer: ESXi localcli
- [Wayback/Archive.is] Overview of vSphere Command-Line Interfaces – Getting Started with vSphere Command-Line Interfaces – VMware {code}
localclicommandslocalclicommands are equivalent toESXCLIcommands, but bypass the hoste daemon (hostd). Thelocalclicommands are only for situations when hostd is unavailable and cannot be restarted. After you run alocalclicommand, you must restarthostd. Run ESXCLI commands after the restart.Warning: If you use alocalclicommand, an inconsistent system state and potential failure can result.- [Wayback] VMware ESXi 6.5 – ESXCLI Command Reference | virten.net
–jeroen
Posted in *nix, *nix-tools, ash/dash, ash/dash development, Development, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Scripting, Software Development, Virtualization, VMware | Leave a Comment »
The Wiert Corner - irregular stream of stuff 