 | Lars Fosdal on Security alarm provider Woonve… |
 | Thomas Mueller on Question got closed in May 202… |
 | Thaddy de Koning on Formulier voor bewindvoerders… |
| Thaddy de Koning on Formulier voor bewindvoerders… |
| Thaddy de Koning on Formulier voor bewindvoerders… |
Posted by jpluimers on 2022/02/23
Earlier this month, I ended ESXi: editing /etc/vmware/hostd/vmInventory.xml to fix the datastore UUID for unavailable VMs part 2 with this:
A final note: I need to check out if
.vswpfiles need to be there at all, as my ESXi servers have plenty of physical memory in order not to swap out to disk. More on that in a future blog post.
Browsing back through my blog posts, I mentioned .vswp files before, but never really dug into them:
Posted in ArchiveTeamWarrior, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Internet, InternetArchive, Power User, Virtualization, VMware, VMware ESXi, WayBack machine | Leave a Comment »
Posted by jpluimers on 2022/02/22
Note: the below holds for sure for ESXi 5.x and 6.5; as of ESXi 7.x, the VMKlinux stack is supposed to be gone for good. I still need to verify that as I’m still on ESXi 6.7.
I wrote about ESXi VMKlinux and native drivers before:
Back then, I knew that VMKlinux based drivers were being slowly phased out as of ESXi 5.5. But I didn’t really understand the actual difference.
VMKlinux is a bit like NDISwrapper – Wikipedia: it allows drivers basically written for a different platform to run on your platform. For NDISwrapper, it is about running Windows NDIS drivers on both i386 (née, x86-32, née IA-32) and x86-64 architectures. For VMKLinux, it allows drivers originally written for the Linux Kernel to be run on ESXi through an intermediate wrapper layer.
Usually having both native and wrapped code means that not all functionality is yet available natively. That was indeed the solution for my smartinfo trouble: the native drivers were having latency problems where the VMKlinux drivers were not.
Sometimes it means that hardware is only supported by the VMKlinux drivers, which (since native drives have a higher priority than VMKlinux drivers) means disabling the native driver.
This does not work in the early upgrade/install boot process: there the settings cannot be read from the configuration files yet. ESXi solves this by having a global boot option preferVmklinux=True that you can store in the boot configuration.
Most of the issues this solves are around USB boot sticks and USB disks, but not all of them, see the below links and summaries for more information.
I like the solution by temporarily using preferVmklinux=True, then just disabling the vmkusb best.
Upon rebooting the host, the host will need to use the legacy USB drivers to boot correctly. At the startup screen, you can press “e” to edit the boot parameters or later you can press “Shift + O” to edit the boot parameters.
Again add this to the boot parameters: preferVmklinux=TRUE
…
Once the system is booted and you can login to the ESXi shell you will need to enter the following commands:
esxcli system settings kernel set -s preferVmklinux -v FALSE esxcli system module set –enabled=false –m vmkusb rebootOnce the system reboots it will boot and discover the USB devices properly.
It should be noted that only a few systems have problems with the new USB driver. I had this trouble on an older system. An HP ML330 G6 gave me the trouble.
only one of the two Gigabit Ethernet interfaces will work out of the box. That’s why the interface did not get an IP address in the video above. Both interfaces were detected, but for some reason only the other one was detecting signal when I had an Ethernet cable plugged in.
…
using another computer make an SSH connection to the ESXi box, log in as root and execute the two following commands:
esxcli system settings kernel set -s preferVmklinux -v TRUE rebootAfter the reboot your both network interfaces should be working as expected.
After upgrading my ESXi host from 6.0 to 6.5, I am now unable to SSH into it or connect to it in any way. It appears to freeze half way through booting up (last message displayed : nfs41client loaded successfully) … Hardware: MacPro 6,1 Q: What are my options? TIA
If you prefer not to manually have to add the ESXi boot option by hand, you can create an ESXi bootable USB key and then simply edit both
boot.cfgandefi/boot/boot.cfgand append the option as shown below:kernelopt=runweasel preferVmklinux=True
The issue looks to be related to the new USB Native Driver (vmkusb) that was introduced in ESXi 6.5 where is it is unable to claim the specific USB device.
Although you can disable the USB Native Driver and fall back to the legacy driver as mentioned in this VMware KB 2147650, but because this is happening during the installation/upgrade process, it can get a bit tricky.
…
workaround
- Step 1 – Append preferVmklinux=TRUE to the ESXi’s boot.cfg file whether that is on a USB device which has your ESXi installer or on your PXE Server for ESXi Scripted Installations.Here is an example of what boot.cfg should look like:
kernelopt=runweasel preferVmklinux=TRUE</pre>- Step 2 – Boot up ESXi and you should now be able to see the USB device and continue with your upgrade.
- Step 3 – Once ESXi has been successfully upgraded, go ahead and run the following commands to re-enable the Native Drivers and disable the vmkusb driver (reboot required for changes to go into effect):
esxcli system settings kernel set -s preferVmklinux -v FALSEesxcli system module set –enabled false -m vmkusbreboot
Note: If you are doing an ESXi Scripted Installation, you can actually append just the first two commands (replace esxcli with localcli as hostd is not running) as part of the Kickstart
%postsection.
In ESXi 6.5, the legacy USB drivers, including
xhci,ehci-hcd,usb-uhci,usb,usb-storage, and so on, are replaced with a single USB driver namedvmkusb. The vmkusb driver is loaded by default and it is applied to all the USB Host Controllers (XHCI/EHCI/UHCI/OHCI), USB Keyboard, Mass Storage, and supported USB NIC devices connected to the host.…
You can disable the vmkusb driver by running
esxcli system module set -m=vmkusb -e=FALSE. The legacy USB drivers are loaded at the next reboot. To re-enable the vmkusb driver, runesxcli system module set -m=vmkusb -e=TRUE, and reboot the host.…
If the host will not boot completely to use the solution, please follow the below steps:
- Reboot the ESXi host.
- During the pre-boot splash screen, press SHIFT-O to modify the boot options.
- In the resulting screen, add the following to the end of the boot line:
jumpstart.disable=vmkusb
- Press the enter key to resume boot.
Note: When ESXi host comes up, vmkusb module will be disabled, and the solution can be followed to make the change permanent.
ESXi 6.5 contains many new native drivers that replace the earlier vmklinux drivers. Most of the new native drivers are enabled by default after installation or upgrade.
Some of the new native drivers are disabled by default, because they do not fully support the functions of the corresponding vmklinux drivers. For example:
- qflge is a native driver that replaces the vmklinux net-bnx2 driver, but does not support HW iSCSI.
- qfle3 is a native driver that replaces the vmklinux net-bnx2x driver, but does not support HW iSCSI and SW FCoE.
- ixgben is a native driver that replaces the vmklinux net-ixgbe driver, but does not support SW FCoE.
…
steps for enabling/disabling a native driver (which then overrules the VMKlinux driver)
Since the first days of ESX, we used the Linux derived driver modules to provide support for a large range of hardware devices. Doing so gave us a lot of hardware compatibility, but at the cost of introducing an additional layer of driver emulation. Because ESX is not Linux, we needed a translate layer that provides communication between the VMkernel and the Linux driver modules. This is the imperative layer that is VMKlinux. But since vSphere 5.5, we introduced the Native driver stack with the plan to move away from the VMKlinux driver stack. Please review the following blog posts for more in depth information about both driver stacks:
- ESXi 5.5 introduces a new Native Device Driver Architecture Part 1
- ESXi 5.5 introduces a new Native Device Driver Architecture Part 2
- Prepping an ESXi 6.7 host for Secure Boot
The vSphere 6.7 family will be the last vSphere versions that includes the VMKlinux driver stack. Future releases will only ship with the Native driver stack.
…
How to check impact on your cluster
If you run the following command, it will show you if the VMKlinux module is loaded on that specific ESXi host.
esxcli system module list | grep vmklinuxWhen it returns nothing, the system is already in full native mode and there are no VMKlinux dependencies. This however, is an ESXi host only approach. We came up with the following script to assist you in verifying an entire cluster.
With the help of William Lam, we created a script that identifies ESXi hosts with VMKlinux driver modules loaded. The usage of the script is pretty straightforward. Get, and contribute to, the script here: https://github.com/lamw/vghetto-scripts/blob/master/powershell/VMKLinuxDrivers.ps1.
–jeroen
Posted in ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/02/17
I wrote about Creating a bootable USB installer for ESXi and use it to create a bootable ESXi installation.
Just in case I ever need to do this on a non-Windows system, some links:
–jeroen
Posted in *nix, Apple, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Linux, Mac OS X / OS X / MacOS, Power User, Virtualization, VMware, VMware ESXi, Windows | Leave a Comment »
Posted by jpluimers on 2022/02/16
In ESXi: on the console/ssh, when a moved VM pauses during power-on: show which VMs have messages waiting, then answer them, I searched for [Wayback] “vim-cmd vmsvc/message” – Google Search in order to see which messages were available.
That search revealed a lot more links, so here are the ones I found most interesting:
–jeroen
Posted in ESXi4, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/02/15
This post amends the post last week on rsync backup of your ESXi box: How to make a statically linked rsync binary « The Wiert Corner – irregular stream of stuff.
Two weeks ago, I posted about Source: ESXi: searching for “vim-cmd vmsvc/message” lead me to the “Managing ESXi Without VI Client” series of blog posts.
It got me looking more deeply in the VM-Help site, and I found [Wayback] Developing for VMware ESXi – Virtual Machine and VPS Tutorials, for which I have materialised the links below and checked their WayBack machine status.
Compiling Utilities for ESXi
Given that ESXi is not based on Linux you won’t find any installer which you could use to install any Linux components that you might want to add to ESXi. However, ESXi does make use of a number of Open Source packages such as OpenSSL, Python, and Openwsman (WS-Management). The key to compiling a utility for ESXi is creating a statically linked version of the tool. With a statically linked version, there are no dependencies on other libraries that may not be present on ESXi. The downside to this method of compiling is that the utility may be larger than a dynamically linked version. With a dynamically linked version the utility assumes that other libraries are present and can rely on subroutines within those libraries.
Compiling rsync – [Wayback] How to compile a statically linked rsync binary for ESXi
Compiling Busybox – [Wayback] How to compile Busybox for ESXi … kind of Part 1
Discussion of compiling UNFS – http://www.vm-help.com/forum/viewtopic.php?f=16&t=2280&p=10185&e=10185 (not archived in the WayBack machine nor available on-line)
Notes on compling binaries – [Wayback] Stjepan Groš – HomepageCompiling Drivers for ESXi
Given the common misunderstanding that ESXi is Linux based, a new user often inquires about the process of copying a Linux driver to their ESXi install. This is not possible. ESXi includes a Linux driver compatibility module. This allows for Linux source code to be used to compile drivers for ESXi, but the drivers are still specific to ESXi. The following links provides some samples and notes for compiling drivers for ESXi.
Compiling a Silicon Image 3132 driver – [Archive.is] Wayback: Adding Driver Support to VMware ESXi 4 | Tip’s Notebook
Compiling a Marvell sky2 driver – [Wayback] Using a Marvell LAN card with ESXi 4 – KernelCrash(Note: This post was initially written when ESXi 4.0 was available. As of late 2010, ESXi 4.1 has been released, and it does actually include a sky2 driver that may or may not work with various Marvell LAN chipsets. The post is still relevant (especially the comments) if your particular Marvell chipset does not work with the sky2 driver in ESXI 4.1. Also, the post is relevant if you’re interested in porting other network drivers to ESXi)
Open Virtualization Drivers development notes
Being from the ESXi 4 and 5 era, the links seem to hold up remarkably well. Despite ESXi 3 being Linux based (see [Archive.is] VMware ESX Server – Wikipedia, the free encyclopedia), as opposed to ESXi 4 and up that run a microkernel, Linux based tools still can be used to develop tooling and drivers.
–jeroen
Posted in ESXi4, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/02/10
Not sure how I bumped into the below info, as it was in one of the open tabs of a VM that I had not accessed for a long time.
Tracing back, the links can be found via Wikipedia.
Here is the post content from the 2005 thread [Archive.is] Wayback “Abbreviation ESX”:
VMware: Origin of ESX and GSX names
in response to: kimono
ESX = Elastic or Electric Sky
GSX = Ground Sky
early days west coast cali hippy advertising thinking from the boys in the states. They dont mean’t anything and the X was added for acromyn
Related:
VMware Server (formerly VMware GSX Server) is a discontinued free-of-charge virtualization-software server suite developed and supplied by VMware, Inc.
2001
- ESX 1.0 (Elastic Sky X) & GSX 1.0 (Ground Storm X) released
- Workstation 3.0 released
The name ESX originated as an abbreviation of Elastic Sky X.
–jeroen
Posted in Power User, Virtualization, VMware, VMware ESXi, VMware Server (GSX) | Leave a Comment »
Posted by jpluimers on 2022/02/08
On my research list: [Wayback] ESXi Host with No vCenter Server · VMware vSphere Integrated Containers 1.4 Documentation: Deploy a Virtual Container Host to an ESXi Host with No vCenter Server.
It is a small guide on [Wayback] vmware/vic: vSphere Integrated Containers Engine is a container runtime for vSphere.
vSphere Integrated Containers Engine (VIC Engine) is a container runtime for vSphere, allowing developers familiar with Docker to develop in containers and deploy them alongside traditional VM-based workloads on vSphere clusters, and allowing for these workloads to be managed through the vSphere UI in a way familiar to existing vSphere admins.
Given my virtualisation infrastructure is ESXi based, I need to contemplate on this, as there are basically two choices for me:
Food for thought!
–jeroen
Posted in ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/02/03
I installed three Samsung 980 Pro NVMe SSD devices in various ESXi rigs.
This is the serial numbers that ESXi came up with:
ESXi presents the serial number (actually the EUI64) in reverse byte order of what is on the device labels.
I had to look up what EUI64 was, and it is a kind of UUID (see Universally unique identifier – Wikipedia), but for hardware devices:
- World Wide Name – WikipediaEach WWN is an 8- or 16-byte number, the length and format of which is determined by the most significant four bits, which are referred to as an NAA (Network Address Authority). The remainder of the value is derived from an IEEE OUI (or from Company Id (CID)) and vendor-supplied information. Each format defines a different way to arrange and/or interpret these components. OUIs are used with the U/L and multicast bits zeroed, or sometimes even omitted (and assumed zero). Though CID has U/L set to 1.The WWN formats include:…
- “Mapped EUI-64” formats manage to fit an EUI-64 address into an 8-byte WWN. Since the NAA is mandatory, and takes up a nibble, this represents a four-bit deficit. These four bits are recouped through the following tricks: First, two bits are stolen from the NAA by allocating NAAs 12, 13, 14, and 15 to all refer to the same format. Second, the remaining two bits are recouped by omitting the U/L and multicast bits from the EUI-64’s OUI. When reconstructing the embedded EUI-64 value, the U/L and multicast bits are assumed to have carried zero values.
…
- [Wayback] Base NVM Express – Part One – NVM Express
A namespace ID (NSID) is an identifier used by a controller to provide access to a namespace (handle to a namespace). An NVMe controller may support multiple namespaces that are referenced using NSID. EUI64 (8 bytes), NGUID (16 bytes) and UUID (128-bit) are globally unique namespace identifiers defined in the Base Specification.
- [Wayback] VMware Docs: NVMe Devices with NGUID Device Identifiers
For NVMe devices, ESXi generates device identifiers based on the information it retrieves from the devices. Generally, the NVMe devices support identifiers in EUI64 or NGUID formats, or use both formats. NGUID is a Namespace Globally Unique Identifier that uses the EUI64 16-byte designator format.
- [Wayback] ESXI6.7 nvme ssd issue – VMware Technology Network VMTN
Being a software engineer, I am disturbing by the way this software create the disk ID.
According to the following table from VMware Docs we have some case that the software will not recognize the disk or lost disks.
ID Formats Supported by Device Device Identifier Generated by Host EUI64 ID Format NGUID ID Format ESXi 6.7 and earlier ESXi 6.7 Update 2 yes yes t10.xxx_EUI64 t10.xxx_EUI64 yes no t10.xxx_EUI64 t10.xxx_EUI64 no yes t10.xxx_controller_serial_number eui.xxx (NGUID) as primary ID t10.xxx_controller_serial_number as alternative primary ID
so what if
- The NVMe SSD from the same company may use the same EUI64 for every NVMe SSD on the same interface (say using an ASUS Hyper M.2 X16 PCIe 3.0×4 Expansion card with 4 identical NVMe SSD). It is possible because of bad design from the SSD manufacturer, the EUI64 are all the same for the 4 NVMe SSD, under this case, the ESXI will only recognized one of the 4 disks (since the disk will be t10.xxx_EUI64, and EUI64 are the same for all 4 NVMe SSD, The “storage” “adapters” tab did show there are 4 interfaces (adapters), but the “stroage” “devices” will only show 1 disks.
- Why there is no “no, no” options?
I understand that the NVMe standard 1.3 (for ESXi 6.7) is the base of how the NVMe SSD should be designed, but I think the software should be smart enough to cover the mistakes the hardware company may make and so it can recognized most of the NVMe SSD that is available from the market.
…
I have the same issue in V7.0! Whatever happened to using something you know is unique like serial numbers or something based off the serial number!
My two identical NVMe drives show up as one! Very odd how this one got through testing!
…
Having same issue with the Asus hyper x16 m. 2 pcie card. I have 4 Intel 660p in it and it’s only showing 1. Even when I changed the bifurcation to x4x4x4x4 for that pcie slot. It actually show nothing. When in auto it shows 1. I’m using esxi 7 though. Trying to test out VMware horizon but esxi not detecting all the nvme in the Asus adapter.
- [Wayback/Archive.is] linux.kernel: [PATCH 0/7] Implement NVMe Namespace Descriptor Identification
This patchset implemets NVMe Namespace Descriptor Identification as of
NVMe 1.3. The Namespace Descriptor Identification allows a NVMe host
to query several Namespace Identification mechanisms, such as EUI-64,
NGUID and UUID from the target. If more than one value is set by the
target, it can transmit all set values to the host.- [Wayback] OS-6042: Need to handle NVMe devices with EUI64 values (SmartOS + ZFS)
NVMe devices with namespaces with an EUI64 value do not attach to the system. It’d be good if these did.
- [Wayback] ⚙ D19905 bhyve: Add EUI64 to NVMe device (FreeBSD)
Add the EUI64 field (part of the Identify Namespace data) to NVMe devices to support UEFI drivers.
The implementation will accept an IEEE Extended Unique Identifier (EUI-64) from the command line. If one isn’t provided, it will create one based on
- The IEEE OUI reported in the Identify Controller data
- The PCI bus, device/slot, function values
- The Namespace ID
- [Wayback] nvme-scsi: Use correct byte ordering for eui64 in Dev ID VPD – Patchwork
NVME specifies an EUI64/NGUID in little-endian format, while SCSI specifies that the Device Identification VPD use big-endian for EUI formats. The current code copies this bytestream directly from the Identification Namespace page, meaning we just need to reverse the bytestream when passing it on to the VPD.…
This seems to hold true for NGUID devices, but Keith just pointed out to me that it may not hold true for EUI64 devices. It seems like that case needs byte swiveling within each field. So I'll NAK for now until I can figure out if that's the case.…
This will break existing setups that rely on VPD 0x83 for device identification (which I think includes older SuSE distros). And once you change the setup anyway please stop using this buggy SCSI emulation.…
- [Wayback] A Quick Tour of NVM Express (NVMe)
…
- nguid, Namespace Globally Unique Identifier (NGUID) and, eui64, IEEE Extended Unique Identifier (EUI64) are assigned when the namespace is created and preserved across namespace and controller operations (e.g. reset, format).
…
These are pictures of the devices in the same order:
Posted in ESXi6.5, ESXi6.7, Hardware, NVMe, Power User, SSD, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/02/02
Last week, I wrote about ESXi: on the console/ssh, when a moved VM pauses during power-on: show which VMs have messages waiting, then answer them.
The “vim-cmd vmsvc/message” – Google Search there lead me to this very interesting series of blog posts on managing ESXi Without VI Client.
It was written in the ESXi 4 and 5 era (for a time-frame, see [Wayback] VMware ESXi Release and Build Number History | virten.net) where VIC – the [Wayback] VMware Infrastructure Client, later named VC: [Wayback] vSphere Client – ruled.
The series content holds remarkably well now that VIC/VC got first replaced by the [Wayback] vSphere Web Client (often called [Wayback] vSphere HTML 5 Client and started out of the [Wayback] ESXi Embedded Host Client fling).
Anyway, here are the posts in the series:
–jeroen
Posted in ESXi4, ESXi5, ESXi5.1, ESXi5.5, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Virtualization, VMware, VMware ESXi | Leave a Comment »
Posted by jpluimers on 2022/02/01
I started my post ESXi: editing /etc/vmware/hostd/vmInventory.xml to fix the datastore UUID for unavailable VMs with
In case I ever need this on ESXi: Insights into the VMware inventory files (
vmAutoStart.xmlandvmInventory.xmlon ESXi;inventory.vmlson VMware Workstation/Player)
Since almost all of my blog is about things I bumped into in real life, this post was a preparation because I kind of expected this to indeed happen, and it did.
Below are the screenshots and steps I took. Of course it is an N=1 experience, so your situation might differ, but I tried to be thorough and not miss any steps.
Posted in ArchiveTeamWarrior, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Internet, InternetArchive, Power User, Virtualization, VMware, VMware ESXi, WayBack machine | Leave a Comment »
The Wiert Corner - irregular stream of stuff 