ESXi: finding unmounted VMFS volumes
Posted by jpluimers on 2021/08/26
Sometimes, especially when ESXi thinks a volume is bad, but the ESXi S.M.A.R.T. logging does not indicate so, it boots without mounting some of the VMFS volumes as datastores.
It took me a while to find the right command to list those, but I’m glad I found it:
# esxcfg-volume -l Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs). VMFS UUID/label: 552f5788-33e30274-8dba-001f29022aed/850EVO1TBR1B Can mount: Yes Can resignature: Yes Extent name: naa.600605b00aa054a0ff0000210221eaf8:1 range: 0 - 953087 (MB) Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs). VMFS UUID/label: 5ad4af1b-f3ae285c-e0f4-0cc47aaa9742/IntelNVMe1TB-BTPY750500091P0H Can mount: Yes Can resignature: Yes Extent name: eui.0000000001000000e4d25cd29be94e01:1 range: 0 - 976639 (MB) Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs). VMFS UUID/label: 5ad4aeea-6954841c-470e-0cc47aaa9742/IntelNVMe1TB-BTPY7425047S1P0H Can mount: Yes Can resignature: Yes Extent name: eui.0000000001000000e4d25c0e8dc74e01:1 range: 0 - 976639 (MB) Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs). VMFS UUID/label: 552d3e82-ccee005a-b719-001f29022aed/850EVO1TBR1A Can mount: Yes Can resignature: Yes Extent name: naa.600605b004b87fb01cc22b3487cbf9a9:1 range: 0 - 953087 (MB)
It is a similar esxcfg-* command with very long output:
# esxcfg-scsidevs -m t10.ATA_____WDC_WD3200BEKT2D22F3T0________________________WD2DWXE708P5C200:1 /vmfs/devices/disks/t10.ATA_____WDC_WD3200BEKT2D22F3T0________________________WD2DWXE708P5C200:1 502124c5-cc47df0d-3ef7-a0369f0e1091 0 simplesata t10.ATA_____SAMSUNG_MZHPV512HDGL2D00000______________S1X1NYAGB09589______:1 /vmfs/devices/disks/t10.ATA_____SAMSUNG_MZHPV512HDGL2D00000______________S1X1NYAGB09589______:1 5791a3e1-0b9368de-4965-0cc47aaa9742 0 Samsung512NVME t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0HA01043L_____:1 /vmfs/devices/disks/t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0HA01043L_____:1 59a018dd-07a9931a-7135-0cc47aaa9742 0 Samsung850-2TB-S3D4NX0HA01043L t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S2KMNCAGB04321L_____:1 /vmfs/devices/disks/t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S2KMNCAGB04321L_____:1 59a01b5c-c46ae5be-f4eb-0cc47aaa9742 0 Samsung850-2TB-S2KMNCAGB04321L naa.5000c50087762d1b:1 /vmfs/devices/disks/naa.5000c50087762d1b:1 59a33f7b-66df7c00-11b0-0cc47aaa9742 0 ST6000VX0001-1SH-Z4D3DZZV naa.600605b00aa054a0ff000021022683ae:1 /vmfs/devices/disks/naa.600605b00aa054a0ff000021022683ae:1 532cd010-6e8c01d1-45be-001f29022aed 0 Raid6SSD
or a totally different command than the other way around (listing all mounted datastores):
# esxcli storage vmfs extent list Volume Name VMFS UUID Extent Number Device Name Partition ------------------------------ ----------------------------------- ------------- -------------------------------------------------------------------------- --------- simplesata 502124c5-cc47df0d-3ef7-a0369f0e1091 0 t10.ATA_____WDC_WD3200BEKT2D22F3T0________________________WD2DWXE708P5C200 1 Samsung512NVME 5791a3e1-0b9368de-4965-0cc47aaa9742 0 t10.ATA_____SAMSUNG_MZHPV512HDGL2D00000______________S1X1NYAGB09589______ 1 Samsung850-2TB-S3D4NX0HA01043L 59a018dd-07a9931a-7135-0cc47aaa9742 0 t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0HA01043L_____ 1 Samsung850-2TB-S2KMNCAGB04321L 59a01b5c-c46ae5be-f4eb-0cc47aaa9742 0 t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S2KMNCAGB04321L_____ 1 ST6000VX0001-1SH-Z4D3DZZV 59a33f7b-66df7c00-11b0-0cc47aaa9742 0 naa.5000c50087762d1b 1 Raid6SSD 532cd010-6e8c01d1-45be-001f29022aed 0 naa.600605b00aa054a0ff000021022683ae 1
Yes, this has to do with my post earlier today: NVMe and SATA health data on ESXi: some links to investigate.
You can mount the volume persistently (with the -M or --persistent-mount option) or transiently (with the -m or --mount option), where you can either refer to the volume by name or by uuid:
esxcfg-volume -l|--list List all volumes which have been detected as snapshots/replicas. -m|--mount Mount a snapshot/replica volume, if its original copy is not online. -u|--umount Umount a snapshot/replica volume. -r|--resignature Resignature a snapshot/replica volume. -M|--persistent-mount Mount a snapshot/replica volume persistently, if its original copy is not online. -U|--upgrade Upgrade a VMFS3 volume to VMFS5. -h|--help Show this message.
So in my case, I mounted two of the volumes by hand (of which later one of the NVMe devices – IntelNVMe1TB-BTPY750500091P0H – died within warranty and the other – IntelNVMe1TB-BTPY7425047S1P0H – was causing read errors within warranty, so sent both for RMA):
-
# esxcfg-volume --persistent-mount IntelNVMe1TB-BTPY750500091P0H Persistently mounting volume IntelNVMe1TB-BTPY750500091P0H -
# esxcfg-volume --persistent-mount IntelNVMe1TB-BTPY7425047S1P0H Persistently mounting volume IntelNVMe1TB-BTPY7425047S1P0H
Alternatively, you can use esxcli to do the same: [Wayback] Mount a Datastore with ESXCLI
Mount a Datastore with ESXCLI
The
esxcli storage filesystemcommands support mounting and unmounting volumes. You can also specify whether to persist the mounted volumes across reboots by using the--no-persistoption.Use the
esxcli storage filesystemcommand to list mounted volumes, mount new volumes, and unmount a volume. Specify one of the connection options listed in Connection Options for vCLI Host Management Commands in place of<conn_options>.Procedure:
List all volumes that have been detected as snapshots.
esxcli <conn_options> storage filesystem listRun esxcli storage filesystem mount with the volume label or volume UUID.
esxcli <conn_options> storage filesystem volume mount --volume-label=<label>|--volume-uuid=<VMFS-UUID>Note
This command fails if the original copy is online.
What to do next
You can later run
esxcli storage filesystem volume unmountto unmount the snapshot volume.esxcli <conn_options> storage filesystem volume unmount --volume-label=<label>|--volume-uuid=<VMFS-UUID>
References:
- [Archive.is] ONTAP 9 Documentation Center: Remounting VMFS volumes after transition using the ESXi CLI
- [Wayback] Identifying disks when working with VMware ESXi (1014953) (which besides
esxcfg-scsidevs -malso shows some otheresxcfg-*commands likeesxcfg-mpath -bandesxcfg-scsidevs -c) - [Archive.is] How do I force a mount of a VMFS volume ? – VMware Technology Network VMTN (a long thread about fixing a corrupt VMFS volume)
- [Wayback] Mount VMFS Datastore – via GUI or via CLI [Guide] | ESX Virtualization (which besides
esxcfg-volume –lalso showingesxcfg-volume –r,esxcfg-volume –Mandesxcli storage vmfs snapshot resignature –volume-label=<label>|–volume-uuid=<id>). - [Wayback] Troubleshooting LUNs detected as snapshot LUNs in vSphere (1011387) explains trouble with snapshots showing
esxcli storage vmfs snapshot list,esxcli storage vmfs snapshot mount -l label|-u uuid(persistent mount),esxcli storage vmfs snapshot mount -n -l label|-u uuid(non-persistent mount),esxcli storage vmfs snapshot resignature -l label|-u uuid(resignature) followed by persistent partition mountesxcfg-volume -M VMFS_UUID|label - [Wayback] VMFS Resignaturing – buildVirtual explains the various pieces of the volume uuid (system time, CPU timestamp, random number, MAC address)
Knowing the command made me – through [Wayback] “esxcfg-volume -l” – Google Search – found out many ran in the same issue, including myself (:
- [Wayback] Solved: VMWare ESXI 6.5 – How to create local datastore (w… – VMware Technology Network VMTN
- [Wayback] How do I force a mount of a VMFS volume ? – VMware Technology Network VMTN (with a lot more interesting steps for fixing partition tables and volumes)
- [Wayback] Mount VMFS Datastore – via GUI or via CLI [Guide] | ESX Virtualization
- [Wayback] Powercli equivilant to esxcfg-volume -l – VMware Technology Network VMTN (use [Wayback] Get-EsxCli)
- ESXi 6.5: mount a datastore that does not automount; esxcfg-volume to the rescue
- Some wizardry: vmkfstools | virtualhobbit
–jeroen
The Wiert Corner - irregular stream of stuff 
Leave a Reply