Why upgrading from ESXi 5.5 Update 3b to 6.0 fails

As of now users who are running ESXi 5.5 with the latest patch level Update 3b will find it difficult to upgrade to ESXi 6.0. Trying that with VMware Update Manager (VUM) or the esxcli software profile update method fails with error messages. esxcli throws this error:

[DependencyError]
VIB VMware_bootbank_xhci-xhci_1.0-3vmw.550.3.78.3223702 requires com.vmware.usb-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.

and the VUM remediation process fails with the error event

The upgrade contains conflicting VIBs. Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO image that contains the newer versions of the conflicting VIBs, and try to upgrade again.

The issue is mentioned in an updated version of the VMware ESXi 5.5 Update 3b Release Notes, with a refreshingly short list of possible workarounds: None.

What are our options in this situation, are there really no workarounds?

Sure there are workarounds, and once you understand the root cause of the issue they are easy to find. With ESXi 5.5 Update 3b the xhci-xhci VIB that is mentioned in the error message was updated to version 1.0-3vmw.550.3.78.3248547 (extending USB 3.0 support), but in ESXi 6.0 the version number of this package currently is 1.0-2vmw.600.1.17.3029758. Version strings are interpreted from left to right, so the package included in the ESXi 5.5 Update 3b ImageProfile (version 1.0-3...) is considered newer than the ones included in any of the ESXi 6.0 ImageProfiles (version 1.0-2...).

(tl;dr? If you just want to know the resolution of the issue then skip to the next paragraph So what are the workarounds?)

This is an unusual exception, but not the only one of its kind: Since Update 3 ESXi 5.5 also includes a net-e1000e driver package of version 3.2.2.1-2... (which adds support for Intel Jacksonville NICs) whereas ESXi 6.0 still includes an older version 2.5.4-6... of this package. But this does not cause the same update issue. Why?

VIB packages can have dependencies on each other which are defined in their metadata. They can require another VIB package (or even a specific version of another VIB package) to be installed, or they might rely on features (certain libraries, function sets or interfaces) that are provided by other VIB packages. To find out about these dependencies you can use the esxcli software vib get command in an ESXi shell. For the xhci-xhci package of ESXi 5.5 this reveals the following:

Here we see that the xhci-xhci VIB (version 1.0-3) is dependent on the feature com.vmware.usb-9.2.2.0 which is mentioned in the error message.

There is no easy command to find out what other package provides this feature. By experience and guessing I found that it is the package misc-drivers:

The update error message tells us that within the new ESXi 6.0 Imageprofile the requirement cannot be satisfied. If we look at the properties of the misc-drivers package that comes with ESXi 6.0 then we learn why:

The misc-drivers package of ESXi 6.0 provides the feature com.vmware.usb-9.2.3.0, but not the feature com.vmware.usb-9.2.2.0 which is required by the latest xhci-xhci v1.0-3 package. The esxupdate process wants to keep the latest xhci-xhci package that comes with ESXi 5.5, but its requirement com.vmware.usb-9.2.2.0 cannot be satisfied with the ESXi 6.0 Imageprofile. Thus the update fails.

So what are the workarounds?

We can now identify two different workarounds:

1. The easiest one is to remove the xhci-xhci package from the system before we try the update. Luckily this is easily possible, because it does not break any dependencies. We are not able to remove a package via VUM, but we can do it via esxcli in the following way:

  esxcli software vib remove -n xhci-xhci

Normally this requires a reboot, but I found that updating the system with the esxcli software profile update method is successful immediately after the package removal without a prior reboot. VUM though requires the reboot to be performed before you try the remediation again.

2. The alternative is to wait ... until VMware releases a new ESXi 6.0 patch bundle that (hopefully) includes an updated version of the xhci-xhci package that has a later version number than the one included in ESXi 5.5 Update 3b. Updating to this newer ESXi 6.0 patch will then succeed without any workaround needed.

New Year wishes ...

With this new QA glitch VMware has seriously messed up the upgrade experience for a lot of their customers :-( I hope that they add a new test case for ESXi patch bundles very soon. It could be as simple as For earlier releases of ESXi check if an upgrade to the current release can be performed without errors.

Improve communications ... it looks like there are different teams working on ESXi 5.5 and 6.0. Maybe they need to talk more to each other. And maybe they also need to re-think their package versioning standard or make it more consistent at least.

And a question at last: Why are functional enhancements like better USB 3.0 and new NIC support added into ESXi 5.5 first, and not into the current ESXi release 6.0? What does VMware consider the stable code base?

Update (2016-01-08):

VMware has released ESXi 6.0 Update 1b on Jan 7th, and that fixes the upgrade path from 5.5 U3b. So if you upgrade an ESXi 5.5 Update 3b host directly to 6.0 Update 1b then the issue mentioned here will not happen,

---

This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.

---

---