The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,465 other followers

ESXi: wrong IPv4 address after moving the ESXi boot USB stick and SSD devices to an identical motherboard with different MAC addresses

Posted by jpluimers on 2021/05/17

A while ago, I wanted to move the ESXi USB stick and SSD devices to another machine with identical motherboard as it had a larger physical case more suited for expansion.

To my surprise, the management network stayed at the same IPv4 address, despite it being being from a DHCP pool, and the new MAC addresses having different IPv4 addresses assigned in the pool (I run a kind of static dynamic address system where the DHCP server has the correct mapping between MAC and IPv4 addresses).

This appears to be a known issue: by default, ESXi copies the MAC address to the vmknic of the management network instead of following hardware changes. You can see this in the screenshot showing the right physical MAC, but the wrong virtual MAC:

The fix is actually quite simple:

After this, you have to perform a reboot for the new setting to take effect.

When booting is done, the virtual MAC has been copied from the physical MAC:

Note I did not have to fiddle with /etc/vmware/esx.conf as VirtuallyVTrue had to.

For more information, see these links (I copied the content of the final link below the footer as it cannot be saved in the WayBack or Archive.is archives):

–jeroen

vmk0 management network MAC address is not updated when NIC card is replaced or vmkernel has duplicate MAC address (1031111) content:

vmk0 management network MAC address is not updated when NIC card is replaced or vmkernel has duplicate MAC address (1031111)


Last Updated: 19.12.2017Categories: Troubleshooting
Details
  • The MAC address of vmk0 management network may not get updated after you replace or assign a new MAC address to the NIC and run the esxcfg-vmknic –l command from the command line. The default setting for the ESXi host is not to select the new hardware MAC address.
  • The management network in inaccessible on an ESXi host.
  • You are unable to ping vmkernel.
  • You are unable to ping a decommissioned ESXi physical server that has been rebuilt.
  • Under certain circumstances there may be a situation where a duplicate mac address arises on the network due to ESXi vmkernel maintaining the mac address of a previous ESXi installation on a different physical server.
  • During initial install on the previous server vmkernel vmk0 interface will use the physical server’s mac address assigned to one of the vmknic interfaces instead of a random 00:50:56::XX:XX:XX.
  • On decommissioning the server if the installation USB or Local disk is moved to another server or copied if a LUN, the vmkernel vmk0 interface on the new server will still have the previous server’s physical MAC address.
Solution
Duplicate MAC addresses

Confirm by checking the physical MAC address of all physical NICs and their link status, the existing vSwitch configuration, and the current vmkernel interfaces configuration. For ESXi, use these commands:

# esxcfg-nics -l

# esxcfg-vswitch -l
# esxcfg-vmknic -l
If a duplicate MAC address is confirmed, the MAC address must be changed by deleting and recreating the vmkernel interface.
To delete a vmknic from a port group, use this command:

# esxcfg-vmknic -d -p pgName

or

# esxcfg-vmknic -d pgName

To add a vmknic to a port group, run the command:

# esxcfg-vmknic -a -i DHCP -p pgName

or

# esxcfg-vmknic -a -i x.x.x.x -n 255.255.255.0 pgName

Workaround
To work around the issue, manually configure the MAC address on the ESXi host:
  1. In the troubleshooting console, run the command:

    esxcfg-advcfg -s 1 /Net/FollowHardwareMac

  2. Restart the ESXi server.
Additional Information
Creating an Alarm to Monitor Duplicate IP address: This method can be applied to warn of duplicate MAC addresses as well.
In addition to logging to the VMkernel logs, ESXi also logs this observation in /var/log/vobd.log file which stands for the VMkernel Observation. These observations can provide critical identifying information in case of an error and is usually used during troubleshooting. In our case, if we are seeing an intermittent network connectivity to our ESXi host which is in result of a duplicate IP Address. Utilizing these VOBs is that you can create vCenter Alarms when a specific VOB has been detected.

You can do exactly the same for detecting a duplicate IP Address for an ESXi host. First, identify the VOB ID by looking in the /var/log/vobd.log file (for Duplicate IP address or Duplicate MAC address):

YYYY-01-21T15:02:07.513Z: [netCorrelator] 917174784727us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for xx.xx.xx.xx on interface vmk0, current owner being xx:xx:xx:xx:xx:xx

Note:

  • The VOB ID for this is esx.problem.net.vmknic.ip.duplicate (in the case of a duplicate IP address) or VmMacConflictEvent or VmStaticMacConflictEvent (in the case of a conflicted MAC address) and this will be used in the vCenter Alarm trigger.
  • The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
To create an alarm:
  1. Create a new Alarm and specify a name, the Monitor type will be Hosts and Monitor for a specific event.
  2. Copy the VOB ID identified above and specify that as the alarm Trigger.
  3. If you want to receive an email notification or send an SNMP trap, configure additional actions, or click Next which displays a vCenter Server alert in the UI.
  4. To confirm the alarm trigger, assign a test virtual machine with the IP Address/MAC address of an ESXi host to this virtual machine, the alarm now appears in the tasks/events.

For translated versions of this article, see:

This Article Replaces
1029279
Keywords
vmk0 management network

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

 
%d bloggers like this: