ESXi: forcing VMKlinux drivers over native drivers, especially at boot time for sort-of supported hardware
Posted by jpluimers on 2022/02/22
Note: the below holds for sure for ESXi 5.x and 6.5; as of ESXi 7.x, the VMKlinux stack is supposed to be gone for good. I still need to verify that as I’m still on ESXi 6.7.
I wrote about ESXi VMKlinux and native drivers before:
Back then, I knew that VMKlinux based drivers were being slowly phased out as of ESXi 5.5. But I didn’t really understand the actual difference.
VMKlinux is a bit like NDISwrapper – Wikipedia: it allows drivers basically written for a different platform to run on your platform. For NDISwrapper, it is about running Windows NDIS drivers on both i386 (née, x86-32, née IA-32) and x86-64 architectures. For VMKLinux, it allows drivers originally written for the Linux Kernel to be run on ESXi through an intermediate wrapper layer.
Usually having both native and wrapped code means that not all functionality is yet available natively. That was indeed the solution for my smartinfo trouble: the native drivers were having latency problems where the VMKlinux drivers were not.
Sometimes it means that hardware is only supported by the VMKlinux drivers, which (since native drives have a higher priority than VMKlinux drivers) means disabling the native driver.
This does not work in the early upgrade/install boot process: there the settings cannot be read from the configuration files yet. ESXi solves this by having a global boot option preferVmklinux=True that you can store in the boot configuration.
Most of the issues this solves are around USB boot sticks and USB disks, but not all of them, see the below links and summaries for more information.
I like the solution by temporarily using preferVmklinux=True, then just disabling the vmkusb best.
- [Wayback] VMware vSphere 6.5 install can’t find USB device – Gallahad IT Inc.
Upon rebooting the host, the host will need to use the legacy USB drivers to boot correctly. At the startup screen, you can press “e” to edit the boot parameters or later you can press “Shift + O” to edit the boot parameters.
Again add this to the boot parameters: preferVmklinux=TRUE
…
Once the system is booted and you can login to the ESXi shell you will need to enter the following commands:
esxcli system settings kernel set -s preferVmklinux -v FALSE esxcli system module set –enabled=false –m vmkusb rebootOnce the system reboots it will boot and discover the USB devices properly.
It should be noted that only a few systems have problems with the new USB driver. I had this trouble on an older system. An HP ML330 G6 gave me the trouble.
- [Wayback] Installing VMware ESXi 6.7.0 on a Hades Canyon NUC – The NUC Blog
only one of the two Gigabit Ethernet interfaces will work out of the box. That’s why the interface did not get an IP address in the video above. Both interfaces were detected, but for some reason only the other one was detecting signal when I had an Ethernet cable plugged in.
…
using another computer make an SSH connection to the ESXi box, log in as root and execute the two following commands:
esxcli system settings kernel set -s preferVmklinux -v TRUE rebootAfter the reboot your both network interfaces should be working as expected.
- [Wayback] Solved: Upgrade problem !! – VMware Technology Network VMTN
After upgrading my ESXi host from 6.0 to 6.5, I am now unable to SSH into it or connect to it in any way. It appears to freeze half way through booting up (last message displayed : nfs41client loaded successfully) … Hardware: MacPro 6,1 Q: What are my options? TIA
- [Wayback] ESXi 6.5 support for Apple Mac Pro 6,1
If you prefer not to manually have to add the ESXi boot option by hand, you can create an ESXi bootable USB key and then simply edit both
boot.cfgandefi/boot/boot.cfgand append the option as shown below:kernelopt=runweasel preferVmklinux=True - I got a question from my buddy Paudie O’Riordan this morning where he was noticing a strange issue while trying to upgrade his ESXi hosts from 6.0 to 6.5 (all on the VMware HCL). Like many of…[Wayback] No suitable disk was found when upgrading to ESXi 6.5 on USB?
The issue looks to be related to the new USB Native Driver (vmkusb) that was introduced in ESXi 6.5 where is it is unable to claim the specific USB device.
Although you can disable the USB Native Driver and fall back to the legacy driver as mentioned in this VMware KB 2147650, but because this is happening during the installation/upgrade process, it can get a bit tricky.
…
workaround
- Step 1 – Append preferVmklinux=TRUE to the ESXi’s boot.cfg file whether that is on a USB device which has your ESXi installer or on your PXE Server for ESXi Scripted Installations.Here is an example of what boot.cfg should look like:
kernelopt=runweasel preferVmklinux=TRUE</pre> - Step 2 – Boot up ESXi and you should now be able to see the USB device and continue with your upgrade.
- Step 3 – Once ESXi has been successfully upgraded, go ahead and run the following commands to re-enable the Native Drivers and disable the vmkusb driver (reboot required for changes to go into effect):
esxcli system settings kernel set -s preferVmklinux -v FALSEesxcli system module set –enabled false -m vmkusbrebootNote: If you are doing an ESXi Scripted Installation, you can actually append just the first two commands (replace esxcli with localcli as hostd is not running) as part of the Kickstart
%postsection.
- Step 1 – Append preferVmklinux=TRUE to the ESXi’s boot.cfg file whether that is on a USB device which has your ESXi installer or on your PXE Server for ESXi Scripted Installations.Here is an example of what boot.cfg should look like:
- [Wayback] Important information about the new ESXi 6.5 USB driver vmkusb, and the legacy USB drivers (2147650)
In ESXi 6.5, the legacy USB drivers, including
xhci,ehci-hcd,usb-uhci,usb,usb-storage, and so on, are replaced with a single USB driver namedvmkusb. The vmkusb driver is loaded by default and it is applied to all the USB Host Controllers (XHCI/EHCI/UHCI/OHCI), USB Keyboard, Mass Storage, and supported USB NIC devices connected to the host.…
You can disable the vmkusb driver by running
esxcli system module set -m=vmkusb -e=FALSE. The legacy USB drivers are loaded at the next reboot. To re-enable the vmkusb driver, runesxcli system module set -m=vmkusb -e=TRUE, and reboot the host.…
If the host will not boot completely to use the solution, please follow the below steps:
- Reboot the ESXi host.
- During the pre-boot splash screen, press SHIFT-O to modify the boot options.
- In the resulting screen, add the following to the end of the boot line:
jumpstart.disable=vmkusb- Press the enter key to resume boot.
Note: When ESXi host comes up, vmkusb module will be disabled, and the solution can be followed to make the change permanent. - Wayback VMware Knowledge BaseL Enabling and Disabling Native Drivers in ESXi (2147565)
ESXi 6.5 contains many new native drivers that replace the earlier vmklinux drivers. Most of the new native drivers are enabled by default after installation or upgrade.
Some of the new native drivers are disabled by default, because they do not fully support the functions of the corresponding vmklinux drivers. For example:
- qflge is a native driver that replaces the vmklinux net-bnx2 driver, but does not support HW iSCSI.
- qfle3 is a native driver that replaces the vmklinux net-bnx2x driver, but does not support HW iSCSI and SW FCoE.
- ixgben is a native driver that replaces the vmklinux net-ixgbe driver, but does not support SW FCoE.
…
steps for enabling/disabling a native driver (which then overrules the VMKlinux driver)
- Current versions of ESXi 6.x are shipped with both the VMKlinux and the Native driver stack. Modern hardware is, in many cases, using the new Native drivers. However, older hardware may still depend on a VMKlinux driver module. We announced to deprecate the VMKlinux driver stack back in 2017. This blog post goes into detail … Continued[Archive.is] What is the Impact of the VMKlinux Driver Stack Deprecation? – VMware vSphere Blog
Since the first days of ESX, we used the Linux derived driver modules to provide support for a large range of hardware devices. Doing so gave us a lot of hardware compatibility, but at the cost of introducing an additional layer of driver emulation. Because ESX is not Linux, we needed a translate layer that provides communication between the VMkernel and the Linux driver modules. This is the imperative layer that is VMKlinux. But since vSphere 5.5, we introduced the Native driver stack with the plan to move away from the VMKlinux driver stack. Please review the following blog posts for more in depth information about both driver stacks:
- ESXi 5.5 introduces a new Native Device Driver Architecture Part 1
- ESXi 5.5 introduces a new Native Device Driver Architecture Part 2
- Prepping an ESXi 6.7 host for Secure Boot
The vSphere 6.7 family will be the last vSphere versions that includes the VMKlinux driver stack. Future releases will only ship with the Native driver stack.
…
How to check impact on your cluster
If you run the following command, it will show you if the VMKlinux module is loaded on that specific ESXi host.
esxcli system module list | grep vmklinuxWhen it returns nothing, the system is already in full native mode and there are no VMKlinux dependencies. This however, is an ESXi host only approach. We came up with the following script to assist you in verifying an entire cluster.
With the help of William Lam, we created a script that identifies ESXi hosts with VMKlinux driver modules loaded. The usage of the script is pretty straightforward. Get, and contribute to, the script here: https://github.com/lamw/vghetto-scripts/blob/master/powershell/VMKLinuxDrivers.ps1.
- A balanced discussion (despite the title) on two forum pages on ESXi 7 dropping VMKlinux support:
–jeroen
The Wiert Corner - irregular stream of stuff 
Leave a Reply