Network troubleshooting, Part III: A real life example (Broadcom NICs dropping packets)

Recently we had a strange problem inside a Linux VM: a rsync-job that was used to copy data from a local disk to a NFS-mounted share reproducibly failed during data copy with a "broken pipe" error message.

Using the methods I wrote about in Part I and Part II of this little troubleshooting series (and some trial and error for sure) we found out that the issue would only occur if the VM was using a certain type of physical NIC, the HP NC371i (with a Broadcom BCM5709 chipset).
Later we also discovered corresponding VMKernel.log-messages like this one:

... vmkernel: 36:02:06:55.923 cpu5:6816883)WARNING: Tso: 545: TSO packet with one segment.>
... vmkernel: 36:02:06:56.325 cpu5:7129949)WARNING: Tso: 545: TSO packet with one segment.
... vmkernel: 36:02:06:57.128 cpu4:6816885)WARNING: Tso: 545: TSO packet with one segment.
... vmkernel: 36:02:06:57.128 cpu4:6816885)WARNING: LinNet: map_pkt_to_skb: This message has repeated 640 times: vmnic1: runt TSO packet (tsoMss=1448, frameLen=1514)

Enough evidence to open a support call with VMware... The outcome was that there is a known problem with the bnx2 driver (that is used for this type of NIC). It drops TSO packets that are below a certain minimum size it expects. The issue only occurs with some of the Broadcom chipsets that this driver can handle. The BCM5709 was not on the list before we opened our case, but it looks like it is also affected.

By the way, TSO stands for TCP segmentation offload and is used to offload the necessary segmentation of large TCP packets to the NIC's hardware. A good thing, if it works flawlessly.
The obvious workaround is to disable TSO by using the appropriate driver options. You could disable it on the host's physical Broadcom-NICs, but this would mean sacrificing the performance benefits of TSO for all VMs using these NICs.
We did not do that, because all other VMs did not have any problems with TSO. Instead we decided to disable TSO only inside the Linux VM that had this problem. This solved the issue for us.

Network troubleshooting, Part II: What physical switch port does the pNIC connect to?

When you have found out what physical NIC (pNIC) a VM is actually using (see my previous post) you may want to check the external switch port that this pNIC connects to (Is it properly configured, what about the error counters?). Okay, what switch port do you need to check?
It is considered good data center practice to have every connection from every server's pNIC to the switch ports carefully documented. Do you? Do you trust the documentation? Is it up to date? If yes you are fine and can stop reading here...

If you want to be sure, and if you use Cisco switches in your data centers then there is a much more reliable way to track these connections: The Cisco Discovery Protocol (CDP). On Cisco devices this is enabled by default, and it periodically broadcasts interface information to the devices attached to its ports (like your ESX hosts).
By default ESX(i) (version 3.5 and above) will receive these broadcasts and display the information in it through the VI client. In the Hosts and Clusters-view select Networking in the Configuration tab of a host. This will display your virtual switches with their physical up-links (vmnic0, vmnic1, etc.). Now click on the little speech bubbles next to the Physical Adapters and a window like the following will pop up:

CDP information shown in the VI client

You can find a lot useful information here. The Device ID is the name of the Cisco switch. And Port ID shows the number/name of the switch module and the port number on that module. So you can tell your network admins exactly what switch port they need to check.

If CDP information is not available for a physical adapter the pop-up window will also tell you this. Possible reasons: You don't use Cisco switches or have CDP broadcasts disabled on them, or the ESX(i) host's interfaces are not in CDP listen mode.

For more detailed information on CDP and how to configure it in ESX see the VMware KB: Cisco Discovery Protocol (CDP) network information.

Network troubleshooting, Part I: What physical NIC does the VM use?

If you encounter a network issue in a VM (like bad performance or packet drops) a good first question to ask yourself is: Is this issue limited to the VM or can it be pinned to one of the host's physical NICs?
So, you need to find out what physical NIC (pNIC) the VM is actually using. In most environment this is not obvious, because the virtual switch that the VM connects to typically has multiple physical up-links (for redundancy) that are all active (to maximize bandwidth).

Unfortunately, it is not possible to find this out by using the VI client. It does not reveal this information regardless whether you use standard or distributed virtual switches.
You need to log in to the host that runs the VM (see the HowTos section for instructions) and run esxtop.
Press n to switch to the network view, and you will see a picture like this one:

Network view of esxtop (click to enlarge)

Find the VM's display name in the USED-BY column and look to the corresponding TEAM-PNIC column then. In this example the VM FRASINT215 uses vmnic1.

Updated be2net driver fixes issues with G7 blades

When we started to deploy our HP ProLiant BL620c G7 blade servers we stumbled over some issues with the driver (be2net) for the built-in FlexNIC adapters. They are documented in the VMware KB:

• ESX host stops when shutting down with the Emulex BladeEngine 10Gb Ethernet Controller be2net driver loaded
• ESX loses network connectivity with Emulex BladeEngine 10Gb Ethernet Controller be2net driver loaded

We followed the recommendation in these articles and updated the be2net driver to version 2.102.554.0. However, we still experienced hangs of the ESXi host and network outages whenever the host was rebooted or had its dvS-connections reconfigured.

These hangs were accompanied by VMKernel.log-messages like this one:

... vmkernel: 10:06:11:06.193 cpu0:4153)WARNING: CpuSched: 939: world 4153(helper11-0) did not yield PCPU 0 for 2993 msec, refCharge=5975 msec, coreCharge=6374 msec,

After opening a support call with VMware we finally found out that these problems were caused by improper handling of VLAN hardware offloading by the be2net driver, and that they only occur when you are using distributed virtual switches (dvS) like we did.
So, after configuring the blade hosts with virtual standard switches (vSS) the problem went away.

Since then we were waiting for a fixed be2net-driver (from Emulex) to be able to return to dvS. We really did not want to abandon this option because it offers some benefits (load based teaming of the physical uplinks and Network I/O Control) over the standard switch.

Today, the waiting finally ended. Emulex has finished the fixed driver, it is available here:

VMware ESX/ESXi 4.x Driver CD for Emulex OneConnect 10Gb Ethernet Controller

Update (18. Jul 2011): In the meantime VMware made two new KB articles available that reference the problems described here and the new driver:

• KB1034748: Connecting or disconnecting the only virtual machine on a ESXi vDS using Emulex OneConnect NICs causes 100% host CPU and network connectivity loss
• KB2001858: Emulex OneConnect loses network connectivity with tagged traffic

In the latter one it is also recommended to update the NIC's firmware. The current one (as of today) is available at HP as a bootable ISO file. Thanks to makö for pointing this out in this post's comments.

First post: About v-Front

v-Front (VMware Front Experience) is a blog about real-life challenges, problems, caveats, bugs, ... that you encounter when you run a reasonably large VMware environment for server virtualization.

It is not focused on the latest news from cloud vendors covering their announcements, new products, and so on, unless I find something really exciting in this area. I'm going to write about the VMware features I use and the problems I stumbled over (and how I solved them) in my own environment.

If you any questions or problems that might fit in here, but are not yet covered, then please drop me a mail at

About me:

I manage a VMware environment in the German location of a large global pharmaceutical company. We are using VMware products since about 2004, and the environment has grown from a single team's test bed to a business critical strategic hosting platform that we offer as a service to all our internal customers.

I write this blog in English to reach as many people as possible. However, my native language is German, so you can send me mails in both languages.