A fix for Intel i211 and i350 adapters not being detected by ESXi

Recently two readers of my blog asked me for help with a strange issue that they encountered when trying to install ESXi on their whitebox hardware. The one was a Shuttle DS57 barebone, and the other one was a Compulab Fitlet-X (an interesting tiny fanless industrial PC with 4 onboard NICs). Both have Intel i211 Gigabit Ethernet adapters that were not detected by ESXi, although they are officially supported by the ESXi 6.0 built-in igb driver (see VMware HCL entry).

Looking for a solution I found that this seems to be a common issue. Multiple reports found in the VMware Communities and other forums convinced me that resolving the issue would help a lot of people. Time for some late night troubleshooting ...


The troubleshoot and fix story
(skip that part if you have the same issue and are only interested in the result)

Strangely enough on the Shuttle system the shell command

lspci -v | grep "Class 0200" -B 1

for listing the PCI network devices detected by ESXi revealed that the i211 adapter was properly detected with the expected device ID 8086:1539 and should be available as vmnic1, but the command

esxcli network nic list

only listed vmnic0, the other NIC - an i218LM -, that was properly detected and working, but not the i211.

This means that the responsible driver (igb) was loaded for the i211, but failed to initialize the adapter. A look at the VMkernel boot messages provided more details. You can find these messages in the compressed file /var/log/boot.gz and browse them with a command like

gunzip -cd /var/log/boot.msg | less

Searching for igb brought up these interesting messages:

igb 0000:02:00.0: The NVM Checksum Is Not Valid
...
WARNING: vmklinux: pci_announce_device:1488: PCI: driver igb probe failed for device 0000:02:00.0

Apparently the driver was checking the contents of the card's NVM (Non-Volatile Memory) and was not happy with what it found there. The checksum did not match the expected value, and so the driver decided to ignore it.

A quick Google search for i211 NVM checksum revealed that this seems to be a know problem with the Linux driver for the i211 adapter, and - since the ESXi driver is derived from the Linux pendant and shares most of its code - this also affects ESXi. It was hard to find a real fix for the issue though, but Lukasz (one of the guys that asked me for help with this issue) pointed me to a blog post of someone who had the same issue with an i350 adapter and resolved it by just modifying the igb driver's source code to ignore the invalid NVM checksum (Funny side note: This post is the only one on his blog, so this is a good proof for that starting and immediately abandon a blog again can still be useful ... Thanks Gnep!).

Can it be so easy? Yes and No. Yes, because modifying the C source code was really easy even for someone like me who is not really a programmer, but No, because (re-)compiling a driver for ESXi is quite a challenge. Luckily I had a backup of an old CentOS VM that already had the ESXi 5.0 Open Source Code build system installed and prepared on it, so I "only" had to

• fetch the source code of a recent igb driver version for ESXi 5.x,
• change the driver's main C source file igb_main.c to continue with device initialization even in case of a bad NVM checksum,
• fiddle around with the build script until it would produce a valid driver binary (this was the hardest part),
• package the result in a VIB file and send it to my testers (Easy, because I did that a zillion times already with my own ESXi Community Packaging Tools) ...

... and they gave me a thumbs-up!: Their i211 adapters were properly working with the modified driver!!

How to get the modified igb driver

Well, of course it is available in the V-Front Online Depot. And like with every other package available there you can install it in an already running ESXi system with the following shell commands:

esxcli software acceptance set --level=CommunitySupported
esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib install -n net-igb -d https://vibsdepot.v-front.de

If the i211 adapter is the only network card in your system then you will even not be able to install ESXi on it without using a custom installation ISO that already includes the modified driver. In this case build that yourself with the ESXi-Customizer-PS script:

.\ESXi-Customizer-PS-v2.4.ps1 -v60 -vft -load net-igb

This will build an ESXi 6.0 installation ISO with the latest ESXi patch level and the modified igb driver pulled from the V-Front Online Depot. Please note that the driver is also compatible with ESXi 5.0, 5.1 and 5.5.

---

This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.

---

---