More ESXi5 installation steps

June 2013
M	T	W	T	F	S	S
	1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30

Posted by jpluimers on 2013/06/10

(note: part of this post is unfinished, but I wanted to make sure all the links are publicly accessible, so I posted earlier and incomplete)

I already did a few ESXi5 postings (they apply to 5.1 as well) of which the most important are:

Creating vSphere 5 ESXi embedded USB Stick (failed at first in HP XW6600, but with MBR partition table it works)
Screenshots: Creating vSphere 5 ESXi embedded USB Stick with MBR partition table

Time to finish up some additional installation steps (with a big thanks to Matthijs ter Woord):

In the HP BIOS (F10 at boot), enable WOL and WOL for any PCI NICs

Intel I350 T4 and T2 NICs (click to view a big T4 image)

Note: the XW6600 somehow does not allow WOL from an Intel I350 T4 NIC (tried both top Port 0 and bottom Port 3 on the NIC) though the Intel I350 product brief indicates WOL is available and the software release notes indicate it should be available from Port A (but does not mention which one is Port A). I’ve not tried any of the other I350 products yet.

Get your ESXi5 license key

visit http://www.vmware.com/products/vsphere-hypervisor/overview.html
at the bottom of he page at “Download VMware vSphere Hypervisor Get our free hypervisor.” then click on “Download Now” which gets you to http://www.vmware.com/go/get-free-esxi
it redirects to https://my.vmware.com/web/vmware/evalcenter?p=free-esxi5
create an account if needed, then login, then your license key will be at https://my.vmware.com/group/vmware/evalcenter?p=free-esxi5 (requires vmware account, etc, etc).

Install the vSphere client on a Windows machine

Sorry, no non-Windows versions of vSphere Client yet, I hope a future version gets you *nix versions. You can download it from these places:

Enter your license key in vSphere Client

Start vSphere Client
Login to your ESXi server
In the tree view on the far left, select your ESXi server
Go to the “Configuration” tab
Under “Software”, click on “Licensed Features”
On the far upper right, click on “Edit…”
In the dialog, select the “Assign new license key to this host”
Click on “Enter Key…”
In the new dialog, enter the key, then click on “OK”
Click on “OK” to confirm the key

Set the Hostname (and optionally IP)

This is still the same as in via Change your ESXi machine’s network hostname, DNS information (and SSL certificate):

Select the ESXi server in the tree view on the left
Select “Configuration” in the horizontal list of tabs
Select “DNS and routing” in the vertical menu (it is in the “Software” section)
Select “Properties” link on the far up right in your screen
In the dialog it is a bit counter-intuitive:
1. Make sure that “Use the following DNS server address” is selected
2. Fill in the hostname under “Name”
3. Optionally, fillin the “Domain” and “Look for hosts in the following domains”
Press the “OK” button

Enable ESX CLI and SSH

A easier way to enable SSH (now called ESXi Shell, formerly known as Tech Support Mode) than at VMware KB: Using Tech Support Mode in ESXi 4.1 and ESXi 5.x is at VMware KB: Using ESXi Shell in ESXi 5.x:

Start vSphere Client
Login to your ESXi server
In the tree view on the far left, select your ESXi server
Go to the “Configuration” tab
Under “Software”, click on “Security Profile”
There is a “Services” section on the left, with a “Properties…” hyperlink on the right: click that “Properties…” hyperlink
From the list, do the following steps for these entries: “ESXi Shell”, “SSH”, “Direct Console UI” (for the differences, read VMware ESXi Total lockdown – what’s that? – ESX Virtualization.)
1. Click the “Options” button
2. Click the “Start” button to start the service (sometimes this will automatically put the radio button at “Start and Stop manually”)
3. Select either “Start and stop manually” or “Start and stop with host” depending how secure you want to have your setup.
4. Click the “OK” button to close the “Options” dialogThere is a “Services” section on the left, with a “Properties…” hyperlink on the right: click that “Properties…” hyperlink
Check if the 3 services are now indeed running
Click the “OK” button to close the “Services Properties” dialog

Name your networks before doing anything else

Lesson learned from previous ESXi installations with many vSwitches: name your Virtual Switches, Virtual Machine Port Groups and VMKernel Ports as soon as possible. If you rename them later, then you run into trouble, because the rename operations are not propagated across other configurations. For instance if you have a NIC of a VM use a certain named VM Port Group called “VM Network”, and then you rename that “VM Network Fiber” , the VM will eventually disconnect that NIC (because the old “VM Network” does not exist any more, so it cannot find the port group, nor the switch around it).

Management Network versus VMKernel

When configuring multiple vSwitches, the first one (normally called vSwitch0) has a Virtual Machine Port Group called “VM Network” (on which the VMs communicatie) and a VMKernel Port called “Management Network”. The latter – even if you rename it – is the “Management Network” on the ESXi console (you reset it if you want to get a new DHCP address, see below). Even if you later add a new Virtual Switch with a new VMKernel, then the first one is still bound to the console “Management Network”: the one mentioned in VMware KB: Configuring the ESXi Management Network from the direct console.

Changing the DHCP IP address

This is tricky, as ESXi does not come with ifconfig. The alternative through esxcli does not allow to renew your IP address. When you do it through the official steps, the LSI service does not recognize the change. So do this upfront as described in Force a DHCP lease renewal in ESX and ESXi » boche.net – VMware vEvangelist, and reboot your ESXi server if you run against problems.

ESX: Run the following two commands locally in the service console (COS): ifdown vswif0 ifup vswif0
ESXi: Use the local console menu to “Restart Management Network”:

Replace vswif0 with the network adapter you use.

Install/Update LSI MegaRAID VIB

If you have an LSI MegaRAID like I have: Some links on using a LSI MegaRAID SAS/SATA 9260-8i controller with BBU using RAID 5/6 in VMware ESXi5. Make sure you read
http://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/MegaRAID_SAS_SW_UG_51530-00_RevJ.pdf first (their site can be really slow, so be patient: it will eventually load).

Basically follow the steps at TinkerTry IT @ home | How to make ESXi 5.1 see the health of an LSI 9265-8i RAID controller and array (seems to work with all 92xx controllers):

Go to https://www.lsi.com/support/Pages/Download-Results.aspx?keyword=9260
Open “MANAGEMENT SOFTWARE AND TOOLS”
Download “VMWare SMIS Provider VIB CIMPAT Certified – 5.5” (that’s the one I tested, direct download is at https://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/VMW-ESX-5.0.0-lsiprovider-500.04.V0.34-0012-1009007.zip (now at http://www.lsi.com/downloads/Public/RAID%20Controllers/RAID%20Controllers%20Common%20Files/VMW-ESX-5.0.0-lsiprovider-500.04.V0.34-0012-1009007.zip)
- You also might want to download this one if you want to setup a console later: “MegaCLI 5.5 P1” https://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/8.07.07_MegaCLI.zip, now http://www.lsi.com/downloads/Public/RAID%20Controllers/RAID%20Controllers%20Common%20Files/8.07.07_MegaCLI.zip
  - For installation instructions read How to install MegaCli on ESXi 5.x – Troubleshooting und Support – Wiki – Troubleshooting und Support – Dell Community.
  - The updated http://www.lsi.com/downloads/Public/RAID%20Controllers/RAID%20Controllers%20Common%20Files/8.07.14_MegaCLI.zip does NOT contain the ESXi version any more. Output will be something like this:
```
/tmp # esxcli software vib install --no-sig-check -v /tmp/vmware-esx-MegaCli-8.07.07.vib 
Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: LSI_bootbank_vmware-esx-MegaCli-8.07.07_8.07.07-01
   VIBs Removed: 
   VIBs Skipped:>/pre>
```
- and “MegaRAID Storage Manager – Windows – 5.6” https://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/13.01.04.00_Windows_MSM.zip) Do not get the file “00_37_V0_04_SMIS_VMware_Installer.zip” which is marked “VMWare SMIS Provider VIB – MR 5.6”, as it is older than the 5.5 version.
Unzip the lsiprovider zip file (it gets you the file “vmware-esx-provider-lsiprovider.vib”)
Move the file to “/tmp/vmware-esx-provider-lsiprovider.vib”
Copy the VIB somewhere on your server. There are various ways, either using an SSH client on your favourite platform like scp, or WinSCP or Putty, or just the vSphere Client, then copy it to your data store. It can be copied anywhere, I just copied it to “/tmp/LSI-MegaRAID-9260-8i/vmware-esx-provider-lsiprovider.vib”
Shutdown or Suspend all your running VMs For instance by entering this from the ESX CLI:
- “/sbin/vmware-autostart.sh stop”
- or stop/suspend them from the vSphere client.
Put your ESXi server in Maintenance Mode (see also vSphere tips and tricks for host maintenance mode | TechRepublic.) For instance entering this command from the ESX CLI: “esxcli system maintenanceMode set -e true -t 0”, or using the vSphere Client:
1. Start vSphere Client
2. Login to your ESXi server
3. In the tree view on the far left, select your ESXi server
4. Go to the “Summary” tab
5. Click on “Enter Maintenance Mode”
6. Click on “Yes” button in the dialog
Now at the ESX CLI, enter this command (change the path of the VIB file when needed) similar to How To Patch vSphere 5 ESXi Without Update Manager:
- “esxcli software vib install -v /tmp/LSI-MegaRAID-9260-8i/vmware-esx-provider-lsiprovider.vib”
- (for upgrading, use “esxcli software vib update -v /tmp/LSI-MegaRAID-9260-8i/vmware-esx-provider-lsiprovider.vib”)
Wait for a while (it can take a couple of minutes on slow systems) until you see output like this:
Installation Result Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective. Reboot Required: true VIBs Installed: LSI_bootbank_lsiprovider_500.04.V0.34-0012 VIBs Removed: VIBs Skipped:
If it fails, then retry the command like this by adding the “–no-sig-check“:
- “esxcli software vib install –no-sig-check-v /tmp/vmware-esx-provider-lsiprovider.vib”
- (for upgrading, use “esxcli software vib update –no-sig-check-v /tmp/vmware-esx-provider-lsiprovider.vib”)
Reboot the ESXi server, either from the ESX CLI (use “/sbin/reboot”) or from the vSphere Client
Disable maintenance mode. For instance entering this command from the ESX CLI: “esxcli system maintenanceMode set -e false -t 0”, or using the vSphere Client:
1. Start vSphere Client
2. Login to your ESXi server
3. In the tree view on the far left, select your ESXi server
4. Go to the “Summary” tab
5. Click on “Enter Maintenance Mode”
6. Click on “Yes” button in the dialog
Enable Promiscuous mode on the vSwitch that is going to run the Windows MSM LSI management software. I had to do this once before when installing ENDIAN Firewall – Connected client can access EFW but no other hosts: enable promiscuous mode on VMware ESXi
1. Select your ESXi server in the tree view on the left
2. Select the “Configuration” tab
3. Find the “Virtual Switch” where the GREEN NIC of your Endian connects to
4. Click on the “Properties” link for that Virtual Switch
5. Select the “Virtual Machine Port Group”
6. Click “Edit”
7. Go to the “Security” tab
8. Put a checkmark after the “Promiscuous Mode”, then set the value in the combobox to “Accept”
9. Press the “OK” button in the “Virtual Machine Port Group” dialog
10. Press the “Close” button in the “Virtual Switch” dialog

After rebooting, you can see the LSI status under “Configuration”, “Hardware”, “Health Status”. If that occasionally fails, you have to restart the “CIM Server” service under “Configuration”, “Software”, “Security Profile”, “Services”.

Add the HP specific drivers so you can see motherboard and temperature sensors

Monitoring the sensors (like temperature) of ESXi machines is tough: most standard monitoring stuff does not work as ESXi uses IPMI for monitoring.

I tried the below, but that didn’t work for an XW6600, as it does not do IPMI:

~ # esxcli hardware platform get Platform Information UUID: 0xb4 0x7b 0x3 0x7d 0x6d 0xf2 0xdd 0x11 0xbb 0xda 0x29 0x2 0x2a 0xed 0x0 0x1f Product Name: HP xw6600 Workstation Vendor Name: Hewlett-Packard Serial Number: CZC905001H IPMI Supported: false ~ # esxcli

Do not monitor your physical sensors from within a VM. Tools like SpeedFan will lie. For instance, on an XW6600 it will say the processor temperature is over 100 degrees Celsius, whereas according to the Xeon E5420 specs the maximum temperature is less than 70 degrees Celsius.

Speedfan seems to have trouble with XW6600/XW8600 anyway as per HP Communities – xw8600 cpu upgrade question – Enterprise Business Community:

SpeedFan, as an alternative, works just OK in the xw6400/xw8400 but does not read correctly in the xw6600/xw8600. Always run the latest BIOS, and there are some fan speed/temp info you can get in those later version workstation’s BIOS that is worth knowing about, including the RPMs for the front PCI fan if you have one installed and the chipset fan speed. HWMonitor will not get you those two.

Anyway, this failed:

~~Go to Driver Versions in HP supplied VMware ESX/ESXi images.~~
~~Download the HP ESXi Offline Bundle for VMware ESXi 5.x version 1.4.5 (this ZIP file: )~~
Copy the ZIB somewhere on your server. There are various ways, either using an SSH client on your favourite platform like scp, or WinSCP or Putty, or just the vSphere Client, then copy it to your data store.
~~It can be copied anywhere, I just copied it to “/tmp/HP-ESXi-downloads/vibsdepot.hp.com/hpq/apr2013/esxi-5x-bundles/”~~
~~Shutdown or Suspend all your running VMs For instance by entering this from the ESX CLI: “/sbin/vmware-autostart.sh stop”, or stop/suspend them from the vSphere client.~~
Put your ESXi server in Maintenance Mode (see also vSphere tips and tricks for host maintenance mode | TechRepublic.) For instance entering this command from the ESX CLI: “esxcli system maintenanceMode set -e true -t 0”, or using the vSphere Client:

Start vSphere Client

Login to your ESXi server

In the tree view on the far left, select your ESXi server

Go to the “Summary” tab

Click on “Enter Maintenance Mode”

Click on “Yes” button in the dialog
Now at the ESX CLI, enter this command (change the path of the VIB file when needed) similar to How To Patch vSphere 5 ESXi Without Update Manager: “esxcli software vib install -d /tmp/HP-ESXi-downloads/vibsdepot.hp.com/hpq/apr2013/esxi-5x-bundles/hp-esxi5.0uX-bundle-1.4.5-3.zip”
~~Wait for a while (it can take a couple of minutes on slow systems) until you see output like this:~~
Installation Result Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective. Reboot Required: true VIBs Installed: Hewlett-Packard_bootbank_char-hpcru_5.0.3.09-1OEM.500.0.0.434156, Hewlett-Packard_bootbank_char-hpilo_500.9.0.0.9-1OEM.500.0.0.434156, Hewlett-Packard_bootbank_hp-ams_500.9.3.5-02.434156, Hewlett-Packard_bootbank_hp-smx-provider_500.03.02.10.4-434156 VIBs Removed: VIBs Skipped:
If it fails, then retry the command like this by adding the “–no-sig-check“: “esxcli software vib install –no-sig-check-v /tmp/vmware-esx-provider-lsiprovider.vib” (for upgrading, use “esxcli software vib update –no-sig-check-v /tmp/vmware-esx-provider-lsiprovider.vib”)
~~Reboot the ESXi server, either from the ESX CLI (use “/sbin/reboot”) or from the vSphere Client~~
Disable maintenance mode. For instance entering this command from the ESX CLI: “esxcli system maintenanceMode set -e false -t 0”, or using the vSphere Client::

Start vSphere Client

Login to your ESXi server

In the tree view on the far left, select your ESXi server

Go to the “Summary” tab

Click on “Enter Maintenance Mode”

Click on “Yes” button in the dialog

After rebooting, you can see the HP sensor information under “Configuration”, “Hardware”, “Health Status” like mentioned at display – How to monitor CPU heat inside VM inside ESX? – Super User. If that occasionally fails, you have to restart the “CIM Server” service under “Configuration”, “Software”, “Security Profile”, “Services”.

Since that didn’t work, lets continue with the next step:

Add VMs from Data Stores

Adding VMs from data stores if you re-installed ESXi on a USB stick on a machine that has a RAID datastore. (funny that the machine got a new MAC and DNS IP address even though the network cable stayed in the same machine. The MAC Vendor ID was still 00:1F:29 – HP)

Start vSphere Client
Login to your ESXi server
In the tree view on the far left, select your ESXi server
Go to the “Summary” tab
Under “Storage”, click on the disk icon of the “Datastore” you want to add VMs from
Click on a folder with the VM you want to add
Right click on the .VMX file you want to add
Choose “Add to Inventory“, then follow the regular steps of adding a VM

Configure VM autostart and order

Configure the VM autostart order like mentioned in VMware vSphere 5.1:

Start vSphere Client
Login to your ESXi server
In the tree view on the far left, select your ESXi server
Go to the “Configuration” tab
Under “Software”, click on “Virtual Machine Startup/Shutdown”
There is a “Services” section on the left, with a “Properties…” hyperlink on the right: click that “Properties…” hyperlink
On the right, click the “Properties…” hyperlink
Put a checkmark next to “Allow virtual machines to start and stop automatically with the system”
Configure each VM you want to autostart: click it, then click the “Move Up” and “Move Down” buttons to move it to the right position in the startup orders.
Click the “OK” button to close the “Virtual Machine Startup/Shutdown” dialog
Check if the right VMs are in the correct startup/shutdown sequences

Install LSI MegaRAID Storage Manager

If you want to maintain your LSI RAID controller from a Windows machine (where you can do lots more than from the vSphere Client):

Go to https://www.lsi.com/support/Pages/Download-Results.aspx?keyword=9260
Open “MANAGEMENT SOFTWARE AND TOOLS”
Download “MegaRAID Storage Manager – Windows – 5.6” (that’s the one I tested, direct download is at https://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/13.01.04.00_Windows_MSM.zip)
Recursively unzip the ZIP file file
Install it on a machine that is in the same network as your ESXi server (that’s the easiest way to connect); it can even be a VM on the ESXi server itself.
On that machine, make sure your ESXi host can be resolved both by IP-address and hostname. This means that I had to change the \WINDOWS\system32\drives\etc\hosts. file to contain an entry like this: 192.168.171.153 esxi51-C.asus.rt-n66u And don’t forget (if you already installed the LSI MSM), to restart the MSMFramework service: net stop MSMFramework net start MSMFramework
Run the SETUP.EXE from the unpacked files
Set LDAP to NO if you don’t have LDAP (which is usually the default)
Run MSM
Click the “Configure Host…” button
Select the radio button “Display all the ESXi-CIMOM servers in the network of the local server” or “Display all the systems in the network of the local server”
Press the “OK” button
Choose “Yes” in the dialog “Do you want to apply the display changes now?”
Click on the IP address URL of the discovered ESXi server
Enter your credentials
https://www.tinkertry.com/esxi-5-1-can-run-lsi-megaraid-in-a-vm/

Make sure your RAID controller performs verify often

HDDs tend to fail, and rebuilding RAID arrays from big disks takes a lot of time. So better have an indication of failure soon.

For LSI MegaRAID controllers, the verify options are called “Check Consistency” (for volumes) and “Patrol Read” (for media scanning and repairing). Intel RAID has a “Parity Verify” option.

Make sure you enable these, and run them often, far more than suggested at LSI Logic MegaRAID: Running consistency checks and How often do you do a consistency check on your DSS system?.

Also Ensure that the Run Consistency Check Continuously box is not ticked as per Error “Patrol Read cannot be started, all VD have an active process” message is displayed in RAID Controller log file.

If you don’t (found out the hard way), there is no way to disable this from MSM any more (I was able to disable it from the LSI WebBIOS – which is a fancy name for the BIOS supporting mouse based operations).

Note that if you get a LSI MegaRaid warning: Patrol Read cant be started, then you most likely have overlapping schedules for the Patrol Read and Check Consistency. Which means you cannot run them both continuously at the same time (the easiest thing to setup), and have to plan for real.

VHV does not work on XW6600 and XW8600

Too bad this does not work: virtuallyGhetto: How to Enable Nested ESXi & Other Hypervisors in vSphere 5.1. The reason is that the best Xeon CPU you can get in an XW6600 or XW8600 is a Xeon 5400, which do not support Intel EPT (Xeon 5500 and up support it as per Intel VT Virtualisierungsfunktionen im Überblick – Thomas Krenn Wiki). I verified it at https://192.168.171.153/mob/?moid=ha-host&doPath=capability which indicates

nestedHVSupported

boolean

false/td>

TODO: installation steps for vMA and vCLI

TODO

Read:

Install:

vMA: vSphere Management Assistant Documentation.
vCLI: vSphere Command-Line Interface Documentation.
ovftool: OVF Tool Documentation.
powerCLI: vSphere PowerCLI Documentation.
ghettoVCB backup/restore by virtuallyGhetto at lamw/ghettoVCB · GitHub.:
ghettoVCB.sh – Free alternative for backing up … | VMware Communities.
Portal Stack: MicroServer ESXi WhiteBox Part 3 – VM Backup GUI with ghettoVCB.js.

TODO

Make a backup of your bootable USB stick, then test the restore

(And make sure you have a couple of the same sticks so if one of them fails, you have some left to restore on). I’m assuming all of your sticks are the same size, as that is the most practical way. I’m running from 1 GB sticks which is more than large enough.

Backup/restore on *nix or Mac OS X can be done through dd. It gets you a binary image of the USB stick.
Backup/restore on Windows, you can use WinImage (I’ve used .IMA extension, but .VHD extension should work just as well)

In addition to the USB stick, you have to backup/restore your configuration using vMA or vCLI (thanks RafW). I’m not sure why, but if you don’t, you loose at least

all the vSwitch / VM Kernel / VM Port Group configuration (you get a default vSwitch0 with the default VM Kernel and VM Port Group, so if you have named them differently al your VMs don’t have connected NICs).
the security settings (like SSH and ESXi shell)

And worse: not all of the partitions worked, as I was getting error messages like “no vmkcore disk partition is available”. Need to sort that out. Notes to self:

The esxcli command at VMware KB: Configuring an ESXi 5.0 host to capture a VMkernel coredump from a purple diagnostic screen to a diagnostic partition showed no active partition, though partition :7 was configured.
Need to check VMware KB: Identifying disks when working with VMware ESX/ESXi and VMware KB: Using the partedUtil command line utility on ESXi/ESX.

Backup your VMs

I was experimenting as per VMware: Ways to transfer VM’s between ESXi servers without shared storage « Bart’s Weblog.

Note: Veeam won’t work for normal backups as per Community Forums • View topic – VMware : [FAQ] Frequently Asked Questions:

VMware Licensing

Q: Is free ESXi (also known as vSphere Hypervisor) supported?

A: Free ESXi is not supported, because it has vStorage API for Data Protection and other management APIs locked down specifically to prevent ISVs from being able to backup/manage such hosts.

The alternative is to use the copy functionality that used to be in FastSCP as promised on the VMware ESX / ESXi file Management – Veeam FastSCP 3.0 page:

FastSCP is integrated with Veeam Backup

And indeed that is dead simple: Home ->

I have to test the speeds against raw SCP, as Veeam might actually be slower (the workstation that I have installed on starts very very slow due to all of the services which Veeam requires). Robert Chase: Veeam “Fast” SCP for ESXi. It is a bit more typing to do scp, but you can script that, and SCP is pretty easy: Robert Chase: Moving VM’s in VMware ESXi via SSH.

Only the Pro version of VM Explorer Editions – Free/Pro supports ESXi to ESXi backups. Need to check that out with a trial license.

	jpluimers on Ookla speedtest CLI for Window…
	Mateusz on Now that XE8 is out, some Turb…
	jpluimers on Some links that might help use…
	jpluimers on Hidden Features in Delphi rela…
	jpluimers on Watching “Why is C# Evol…

The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

Twitter Updates

My Flickr Stream

Pages

All categories

Email Subscription