The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,484 other followers

Archive for August 26th, 2021

ESXi: finding unmounted VMFS volumes

Posted by jpluimers on 2021/08/26

Sometimes, especially when ESXi thinks a volume is bad, but the ESXi S.M.A.R.T. logging does not indicate so, it boots without mounting some of the VMFS volumes as datastores.

It took me a while to find the right command to list those, but I’m glad I found it:

# esxcfg-volume -l
Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs).
VMFS UUID/label: 552f5788-33e30274-8dba-001f29022aed/850EVO1TBR1B
Can mount: Yes
Can resignature: Yes
Extent name: naa.600605b00aa054a0ff0000210221eaf8:1 range: 0 - 953087 (MB)

Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
VMFS UUID/label: 5ad4af1b-f3ae285c-e0f4-0cc47aaa9742/IntelNVMe1TB-BTPY750500091P0H
Can mount: Yes
Can resignature: Yes
Extent name: eui.0000000001000000e4d25cd29be94e01:1 range: 0 - 976639 (MB)

Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
VMFS UUID/label: 5ad4aeea-6954841c-470e-0cc47aaa9742/IntelNVMe1TB-BTPY7425047S1P0H
Can mount: Yes
Can resignature: Yes
Extent name: eui.0000000001000000e4d25c0e8dc74e01:1 range: 0 - 976639 (MB)

Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs).
VMFS UUID/label: 552d3e82-ccee005a-b719-001f29022aed/850EVO1TBR1A
Can mount: Yes
Can resignature: Yes
Extent name: naa.600605b004b87fb01cc22b3487cbf9a9:1 range: 0 - 953087 (MB)

It is a similar esxcfg-* command with very long output:

# esxcfg-scsidevs -m
t10.ATA_____WDC_WD3200BEKT2D22F3T0________________________WD2DWXE708P5C200:1 /vmfs/devices/disks/t10.ATA_____WDC_WD3200BEKT2D22F3T0________________________WD2DWXE708P5C200:1 502124c5-cc47df0d-3ef7-a0369f0e1091  0  simplesata
t10.ATA_____SAMSUNG_MZHPV512HDGL2D00000______________S1X1NYAGB09589______:1  /vmfs/devices/disks/t10.ATA_____SAMSUNG_MZHPV512HDGL2D00000______________S1X1NYAGB09589______:1  5791a3e1-0b9368de-4965-0cc47aaa9742  0  Samsung512NVME
t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0HA01043L_____:1   /vmfs/devices/disks/t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0HA01043L_____:1   59a018dd-07a9931a-7135-0cc47aaa9742  0  Samsung850-2TB-S3D4NX0HA01043L
t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S2KMNCAGB04321L_____:1   /vmfs/devices/disks/t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S2KMNCAGB04321L_____:1   59a01b5c-c46ae5be-f4eb-0cc47aaa9742  0  Samsung850-2TB-S2KMNCAGB04321L
naa.5000c50087762d1b:1                                                       /vmfs/devices/disks/naa.5000c50087762d1b:1                                                       59a33f7b-66df7c00-11b0-0cc47aaa9742  0  ST6000VX0001-1SH-Z4D3DZZV
naa.600605b00aa054a0ff000021022683ae:1                                       /vmfs/devices/disks/naa.600605b00aa054a0ff000021022683ae:1                                       532cd010-6e8c01d1-45be-001f29022aed  0  Raid6SSD

or a totally different command than the other way around (listing all mounted datastores):

# esxcli storage vmfs extent list
Volume Name                     VMFS UUID                            Extent Number  Device Name                                                                 Partition
------------------------------  -----------------------------------  -------------  --------------------------------------------------------------------------  ---------
simplesata                      502124c5-cc47df0d-3ef7-a0369f0e1091              0  t10.ATA_____WDC_WD3200BEKT2D22F3T0________________________WD2DWXE708P5C200          1
Samsung512NVME                  5791a3e1-0b9368de-4965-0cc47aaa9742              0  t10.ATA_____SAMSUNG_MZHPV512HDGL2D00000______________S1X1NYAGB09589______           1
Samsung850-2TB-S3D4NX0HA01043L  59a018dd-07a9931a-7135-0cc47aaa9742              0  t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0HA01043L_____            1
Samsung850-2TB-S2KMNCAGB04321L  59a01b5c-c46ae5be-f4eb-0cc47aaa9742              0  t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S2KMNCAGB04321L_____            1
ST6000VX0001-1SH-Z4D3DZZV       59a33f7b-66df7c00-11b0-0cc47aaa9742              0  naa.5000c50087762d1b                                                                1
Raid6SSD                        532cd010-6e8c01d1-45be-001f29022aed              0  naa.600605b00aa054a0ff000021022683ae                                                1

Yes, this has to do with my post earlier today: NVMe and SATA health data on ESXi: some links to investigate.

You can mount the volume persistently (with the -M or --persistent-mount option) or transiently (with the -m or --mount option), where you can either refer to the volume by name or by uuid:

esxcfg-volume
-l|--list               List all volumes which have been
                        detected as snapshots/replicas.
-m|--mount              Mount a snapshot/replica volume, if 
                        its original copy is not online.
-u|--umount             Umount a snapshot/replica volume.
-r|--resignature        Resignature a snapshot/replica volume.
-M|--persistent-mount   Mount a snapshot/replica volume
                        persistently, if its original copy is
                        not online.
-U|--upgrade            Upgrade a VMFS3 volume to VMFS5.
-h|--help               Show this message.

So in my case, I mounted two of the volumes by hand (of which later one of the NVMe devices – IntelNVMe1TB-BTPY750500091P0H – died within warranty and the other – IntelNVMe1TB-BTPY7425047S1P0H – was causing read errors within warranty, so sent both for RMA):

  • # esxcfg-volume --persistent-mount IntelNVMe1TB-BTPY750500091P0H
    Persistently mounting volume IntelNVMe1TB-BTPY750500091P0H
  • # esxcfg-volume --persistent-mount IntelNVMe1TB-BTPY7425047S1P0H
    Persistently mounting volume IntelNVMe1TB-BTPY7425047S1P0H

Alternatively, you can use esxcli to do the same: [Wayback] Mount a Datastore with ESXCLI

Mount a Datastore with ESXCLI

The esxcli storage filesystem commands support mounting and unmounting volumes. You can also specify whether to persist the mounted volumes across reboots by using the --no-persist option.

Use the esxcli storage filesystem command to list mounted volumes, mount new volumes, and unmount a volume. Specify one of the connection options listed in Connection Options for vCLI Host Management Commands in place of <conn_options>.

Procedure:

  1. List all volumes that have been detected as snapshots.

    esxcli <conn_options> storage filesystem list
  2. Run esxcli storage filesystem mount with the volume label or volume UUID.

    esxcli <conn_options> storage filesystem volume mount --volume-label=<label>|--volume-uuid=<VMFS-UUID>

    Note

    This command fails if the original copy is online.

What to do next

You can later run esxcli storage filesystem volume unmount to unmount the snapshot volume.

esxcli <conn_options> storage filesystem volume unmount --volume-label=<label>|--volume-uuid=<VMFS-UUID>

References:

Knowing the command made me – through [Wayback] “esxcfg-volume -l” – Google Search – found out many ran in the same issue, including myself (:

–jeroen

Posted in ESXi6, ESXi6.5, ESXi6.7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »

More firewalld links

Posted by jpluimers on 2021/08/26

Last week I posted firewalld: show interfaces with their zone details and show zones in use as a follow up on On my research list: migrate from OpenSuSE SuSEfirewall2 to firewalld « The Wiert Corner – irregular stream of stuff.

I explained that firewalld is based on zones, and showed some statements on how to retrieve information.

In the mean time, I collected a bunch more firewalld related links, so here they are:

–jeroen

Posted in *nix, *nix-tools, firewalld, Power User | Leave a Comment »

firewalld: show interfaces with their zone details and show zones in use

Posted by jpluimers on 2021/08/26

A while ago openSUSE switched to firewalld as a fronte-end for iptables. Tumbleweed was first in 2018, so I wrote a reminder: On my research list: migrate from OpenSuSE SuSEfirewall2 to firewalld « The Wiert Corner – irregular stream of stuff.

The core concept of firewalld is zones, which some people find hard to understand: [Archive.is/WayBack] Firewalld on Leap 15 – why is it so complicated ? : openSUSE.

Another concept is interfaces and how they bind to zones. [Wayback] Masquerading and Firewalls | Security Guide | openSUSE Leap 15.2 shows more of that.

The final concept is services that bind one or more aspects (like ports or addresses) to a service name [Wayback] Documentation – Manual Pages – firewalld.service | firewalld.

Other interesting bits of information:

Below are some examples on what I learned, especially finding details about active interfaces and the zones they are bound to.

All of them are based on:

  • the xargs shell trick (I known you can do some of them without the trick, but I try to use common patterns in my solution so I do not have to remember which boundary case fails
  • the echo -n trick to skip the newline output
  • the [WayBack] firewall-cmd options (which kind of care commands)
    • --get-active-zones:

      Print currently active zones altogether with interfaces and sources used in these zones. Active zones are zones, that have a binding to an interface or source. The output format is:

      zone1
        interfaces: interface1 interface2 ..
        sources: source1 ..
      zone2
        interfaces: interface3 ..
      zone3
        sources: source2 ..

      If there are no interfaces or sources bound to the zone, the corresponding line will be omitted.

    • --list-interfaces:

      List interfaces that are bound to zone zone as a space separated list. If zone is omitted, default zone will be used.

    • --get-zone-of-interface=<zone>:

      Print the name of the zone the interface is bound to or no zone.

    • --info-zone=<zone> (which shows far more information than the manual indicates):

      Print information about the zone zone. The output format is:

      zone
        interfaces: interface1 ..
        sources: source1 ..
        services: service1 ..
        ports: port1 ..
        protocols: protocol1 ..
        forward-ports: forward-port1 ..
        source-ports: source-port1 ..
        icmp-blocks: icmp-type1 ..
        rich rules: rich-rule1 ..

Two more notes before the examples:

  1. My first hunch was to use --list-all-zones, but that shows details of all un-used zones as well.
  2. I am not fully sure about the --list-interfaces to list *all* interfaces. I might replace this later with ls /sys/class/net (see [WayBack] linux – List only the device names of all available network interfaces – Super User).

Other useful commands

Besides lising zones and interfaces, you might be interested in services and ports:

# firewall-cmd --list-services
dhcpv6-client ssh
# firewall-cmd --list-ports

List used zones

The first only shows the zone names

# firewall-cmd --list-interfaces | xargs -I {} sh -c 'firewall-cmd --get-zone-of-interface={}'
public

The second both zones and interfaces:

# firewall-cmd --get-active-zones 
public
  interfaces: ens192

When there are no bound interfaces

OpenSuSE by default does not bind interfaces to zones; it means any interface uses the default zone. That means the --list-interfaces commands in this blog post fail.

You can check this behaviour by running this command:

# ls /sys/class/net | xargs -I {} sh -c 'echo -n "interface {} has zone " ; firewall-cmd --get-zone-of-interface={} | xargs -I [] sh -c "echo [] ; firewall-cmd --info-zone=[]"'
interface eth0 has zone no zone
interface lo has zone no zone
interface wlan0 has zone no zone

Alternatives:

  1. Finding the default zone
    # firewall-cmd --get-default-zone
    public
    
  2. Details of the default zone
    # firewall-cmd --info-zone=$(firewall-cmd --get-default-zone)
    public
      target: default
      icmp-block-inversion: no
      interfaces: 
      sources: 
      services: dhcpv6-client ssh
      ports: 
      protocols: 
      masquerade: no
      forward-ports: 
      source-ports: 
      icmp-blocks: 
      rich rules: 

You can see that here the public zone is marked default which means it binds to any interface that is not bound to a specific zone.

List used zone details

# firewall-cmd --list-interfaces | xargs -I {} sh -c 'firewall-cmd --get-zone-of-interface={} | xargs -I [] sh -c "firewall-cmd --info-zone=[]"'
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources: 
  services: dhcpv6-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

List interfaces and their zones:

# firewall-cmd --list-interfaces | xargs -I {} sh -c 'echo -n "interface {} has zone " ; firewall-cmd --get-zone-of-interface={}'
interface ens192 has zone public

List interfaces and their zone details:

# firewall-cmd --list-interfaces | xargs -I {} sh -c 'echo -n "interface {} has zone " ; firewall-cmd --get-zone-of-interface={} | xargs -I [] sh -c "echo [] ; firewall-cmd --info-zone=[]"'
interface ens192 has zone public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources: 
  services: dhcpv6-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

Verifying if dns service is available, then allow it on public

Verify if a DNS is in the enabled services:

# firewall-cmd --list-services
dhcpv6-client ssh

Here no DNS service is enabled, so I need to figure out if any DNS service is available to be enabled.

This lists all the services that can be enabled in a zone:

# firewall-cmd --get-services

On my system, this returned the following list:

RH-Satellite-6 amanda-client amanda-k5-client amqp amqps apcupsd audit bacula bacula-client bb bgp bitcoin bitcoin-rpc bitcoin-testnet bitcoin-testnet-rpc bittorrent-lsd ceph ceph-mon cfengine cockpit condor-collector ctdb dhcp dhcpv6 dhcpv6-client distcc dns dns-over-tls docker-registry docker-swarm dropbox-lansync elasticsearch etcd-client etcd-server finger freeipa-4 freeipa-ldap freeipa-ldaps freeipa-replication freeipa-trust ftp ganglia-client ganglia-master git grafana gre http https imap imaps ipp ipp-client ipsec irc ircs iscsi-target isns jenkins kadmin kdeconnect kerberos kibana klogin kpasswd kprop kshell ldap ldaps libvirt libvirt-tls lightning-network llmnr managesieve matrix mdns memcache minidlna mongodb mosh mountd mqtt mqtt-tls ms-wbt mssql murmur mysql nfs nfs3 nmea-0183 nrpe ntp nut openvpn ovirt-imageio ovirt-storageconsole ovirt-vmconsole plex pmcd pmproxy pmwebapi pmwebapis pop3 pop3s postgresql privoxy prometheus proxy-dhcp ptp pulseaudio puppetmaster quassel radius rdp redis redis-sentinel rpc-bind rsh rsyncd rtsp salt-master samba samba-client samba-dc sane sip sips slp smtp smtp-submission smtps snmp snmptrap spideroak-lansync spotify-sync squid ssdp ssh steam-streaming svdrp svn syncthing syncthing-gui synergy syslog syslog-tls telnet tentacle tftp tftp-client tile38 tinc tor-socks transmission-client upnp-client vdsm vnc-server wbem-http wbem-https wsman wsmans xdmcp xmpp-bosh xmpp-client xmpp-local xmpp-server zabbix-agent zabbix-server

I was searching to see if dns was available, so I split the string with tr, then searced with grep:

# firewall-cmd --get-services | tr " " "\n" | grep dns
dns
dns-over-tls
mdns

To get details, use the firewall-cmd --info-service=servicename like this:

# firewall-cmd --get-services | tr " " "\n" | grep dns | xargs -I [] sh -c "firewall-cmd --info-service=[]"
dns
  ports: 53/tcp 53/udp
  protocols: 
  source-ports: 
  modules: 
  destination: 
  includes: 
dns-over-tls
  ports: 853/tcp
  protocols: 
  source-ports: 
  modules: 
  destination: 
  includes: 
mdns
  ports: 5353/udp
  protocols: 
  source-ports: 
  modules: 
  destination: ipv4:224.0.0.251 ipv6:ff02::fb
  includes: 

So for named (bind), I need the dns service to be enabled:

# firewall-cmd --zone=public --add-service=dns --permanent
success

Now a –list-services will not show dns as we changed the --permanent configuration, not the current configuration:

# firewall-cmd --list-services
dhcpv6-client ssh

So you need to --reload the --permanent settings:

# firewall-cmd --list-services --permanent
dhcpv6-client dns ssh
# firewall-cmd --reload
success
# firewall-cmd --list-services
dhcpv6-client dns ssh

–jeroen

Posted in *nix, *nix-tools, bash, bash, Development, iptables, Linux, openSuSE, Power User, Scripting, Software Development, SuSE Linux, Tumbleweed, xargs | Leave a Comment »

 
%d bloggers like this: