Why upgrading from ESXi 5.5 Update 3b to 6.0 fails

As of now users who are running ESXi 5.5 with the latest patch level Update 3b will find it difficult to upgrade to ESXi 6.0. Trying that with VMware Update Manager (VUM) or the esxcli software profile update method fails with error messages. esxcli throws this error:

[DependencyError]
VIB VMware_bootbank_xhci-xhci_1.0-3vmw.550.3.78.3223702 requires com.vmware.usb-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.

and the VUM remediation process fails with the error event

The upgrade contains conflicting VIBs. Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO image that contains the newer versions of the conflicting VIBs, and try to upgrade again.

The issue is mentioned in an updated version of the VMware ESXi 5.5 Update 3b Release Notes, with a refreshingly short list of possible workarounds: None.

What are our options in this situation, are there really no workarounds?

How to build a customized ESXi installation image for your whitebox server

My ESXi-Customizer tool has come a long way... Originally built in 2011 it was a great tool for building customized VMware ESXi installation images, it was downloaded over 200k times since then and helped a lot of people who had one quest: getting ESXi installed and running on their whitebox (unsupported) hardware. This year it was even mentioned in a VMware KB article ... However, please do NOT use it anymore! Because ...

Since quite a while there is a much better way to build customized ESXi images, and that is my ESXi-Customizer-PS PowerCLI script. I'm writing this post not only to explain how easy it is to create a customized ESXi installation image for your whitebox server with this script, but also to convince the last nonbelievers that it works much better than my legacy ESXi-Customizer tool.

How to embed the VMware Tools version 10.0 into ESXi

Recently I ranted about the current situation with VMware Tools, and my blog post The great VMware Tools dilemma caught some attention and got a lot of agreement. The good news is that VMware is listening to its customers and trying to improve things.

In a new blog post VMware's Brian Graf elaborates on the recommended solution to use a shared productLocker location for VMware Tools with your ESXi hosts. I complained in my post that it is not easily possible to populate such a shared location with the new "standalone" Tools version 10.0. And this has been improved now:

• The VMware Tools version 10.0 download now includes all the files that you need to upload to the shared productLocker location, and
• Brian has shared some scripts on his own blog to automate the process of creating a shared productLocker, uploading the Tools to it and configuring the hosts to use it.

Great news!

But what if you only have one or few ESXi hosts and want to avoid the overhead of creating a shared productLocker? Wouldn't it be good if we could just update the Tools that are embedded into ESXi to the new version 10.0?

To say it short: Yes, we can! The embedded Tools is just a VIB package (called tools-light) that can be updated. Unfortunately, VMware refuses (so far) to make VMware Tools 10.0 available as a VIB file, but we can build this ourselves! Here is how ...

[Update] ESXi Community Packaging Tools updated to version 2.3

After almost two years I decided to touch my ESXi Community Packaging Tools again and give them an important update. These tools consists of two scripts that enable Community Developers to create software packages for ESXi 5.x/6.x in the VMware proprietary formats VIB (VMware Installation Bundle) and ZIP (VMware Offline Bundle).

Read on to learn what's new and to not miss an exciting sneak peak of what's to come next.

VMware introduces support for Intel i219 (Jacksonville) NICs in ESXi

In this year's summer Intel introduced their new Skylake platform. These motherboards are based on the Z170 chipset and often include onboard Intel i219(-V or -LM) Gigabit Ethernet adapters. VMware ESXi did not properly support the Skylake chipset from the beginning, but it looks like this changed with the release of ESXi 6.0 Update 1 and 5.5 Update 3. I do not own such a system myself, but I have heard reports of users who were able to successfully install ESXi 6.0 U1 on such a system.

However, ESXi 6.0 Update 1 does not (yet) include a driver that supports the i219 NICs. But ESXi 5.5 Update 3 (that was published only few days later) comes with an updated e1000e driver (version 3.2.2.1) that supports these NICs. Here is how you can make best use of it, even if you are already using ESXi 6.0.

The great VMware Tools dilemma

Recently VMware made VMware Tools version 10.0.0 available as a standalone download. This version is now an official downloadable component of vSphere 6.0 with its own Release Notes document and a download page in MyVMware.

In the announcement blog post VMware's Brian Graf writes:

Good news. We have decided that there isn’t any specific reason that VMware Tools builds should be tied to vSphere releases/ESXi builds. Rather, when our engineering teams are ready with key features/updates, we should have the ability to get those benefits out to our customers as quickly as possible.

This release was announced and perceived as a great achievement, but - except for this good news that Brian shared with us - I cannot follow any of the other excited statements that were made about it. It's time to debunk some myths, and it's time to admit that the whole VMware Tools story is still a great mess!

Shut up, Windows 10! Here is a free must-have tool to protect your privacy.

If you install Windows 10 with the default "Express Settings" then it will collect a plethora of data from you and send it to Microsoft and "trusted partners": Not only whatever you ask Cortana (Microsoft's version of Apple's Siri or Google Now), but also what you type or write by hand, where you are, what networks you connect to, what you search for and more...

Sure, they are not alone with this: Google does it with Android, Apple with iOS devices, and - if you ask them - they all collect the data only to optimize the personalized services that they offer you for free. Really for free? No, you pay with your data ... However, more and more people are concerned about data privacy and want to have a choice about what data they disclose. Recently I stumbled over a great tool that helps those people when using Windows 10.

A fix for Intel i211 and i350 adapters not being detected by ESXi

Recently two readers of my blog asked me for help with a strange issue that they encountered when trying to install ESXi on their whitebox hardware. The one was a Shuttle DS57 barebone, and the other one was a Compulab Fitlet-X (an interesting tiny fanless industrial PC with 4 onboard NICs). Both have Intel i211 Gigabit Ethernet adapters that were not detected by ESXi, although they are officially supported by the ESXi 6.0 built-in igb driver (see VMware HCL entry).

Looking for a solution I found that this seems to be a common issue. Multiple reports found in the VMware Communities and other forums convinced me that resolving the issue would help a lot of people. Time for some late night troubleshooting ...

Troubleshooting VM network performance using vsish

Long time, no post ... I'm currently busy with several support requests that I opened with VMware. These keep me busy but they are also good opportunities to learn something new.

One of the cases involves troubleshooting the network performance of Lync 2013 servers running on top of vSphere. It turned out that the VMkernel system information shell (better known as vsish) is a great tool for this.

Heads up: The upcoming Leap Second adjustment may affect your IT infrastructure!

Okay, probably everyone is aware of the upcoming leap second time adjustment on June 30th, 0:00 UTC. I won't explain the story behind it although it's a very interesting topic, but others have done this already. A good starting point is the Wikipedia Leap Second article.

You might wonder: Okay, this is an anomaly that is confusing, but it's only one second, and modern IT systems should not really stumble over this, right? I was of the same opinion until recently, but the nearer the time comes the more IT vendors alert their customers of leap second related issues in their products, and you might be surprised to hear that very critical parts of your infrastructure can fail on June 30th if you do not patch them before!

Filtering logs in ESXi 6.0 - A practical example

Recently I rented a new server with an LSI MegaRAID SAS 9260-4i controller and - of course - installed VMware ESXi 6.0 on it. I will soon migrate my "production workloads" (the V-Front Online Depot, the ESXi Patch Tracker site and my Zimbra E-Mail server) onto this box to improve their availability. A disk failure is one of the most likely disaster scenarios, so a hardware RAID controller with two mirrored disks will certainly help.

After I had the host installed and configured I did a routine check of the log files and found a lot of SCSI errors like these in /var/log/vmkernel.log:

cpu1:34460)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x439d80668b00, 0) to dev "naa.600605b0058b25f01c995a4f0b03da18" on path "vmhba1:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
cpu1:32779)ScsiDeviceIO: 2646: Cmd(0x439d80716d00) 0x4d, CmdSN 0x1 from world 34425 to dev "naa.600605b0058b25f01c995a4f0b03da18" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

These errors were referring to the RAID 1 disk, and first I was worried about them - according to KB289902 the D:0x2 translates to "Device: Check condition" and the sense data codes 0x5 0x24 0x0 and 0x5 0x20 0x0 mean "Illegal request", "Invalid Field in CDB", resp. "Invalid command operation code". I double checked that the RAID controller is on the VMware HCL, I searched the Internet for these errors, found few peoples' posts describing this issue, but no solution. I even updated the firmware of the controller, all to no avail ... and then, finally, I stumbled over KB1031221 which - basically - describes that these errors are "normal" and "can be safely ignored".

Well, that calmed me down... But I'm a log purist: I wanted to get rid of these false error messages in vmkernel.log!

What's new in the V-Front Online Depot - or - how to install ESXi on an HP Pavilion 17 notebook

Recently I added some new NIC drivers to the V-Front Online Depot. Well, they are not really new, but were created back in 2011/12 by a Japanese guy who goes by the nickname ichi. He was actively participating in Dave Mishchenko's vm-help.com forums and posted the drivers there.

Although I was aware of them since then I did not add them to the Depot before - I thought that no one would really need them these days, because the NIC models that they support are pretty old and rarely used today. I was wrong, at least in one case, because I needed one of these drivers myself to install ESXi on a brand new HP Pavilion 17 notebook that I just purchased ...

CPU Microcode Updates revisited: What's new in ESXi 6.0?

Almost two years ago I wrote an FAQ post on CPU Microcode updates with ESXi 5.x. This still gets a lot of hits, but - although the general information about Microcodes still applies - the ESXi related information needs an update, because things have changed with ESXi 6.0.

So what's new and different now?

[Release] ESXi-Customizer-PS 2.4 - The ESXi image customization script

I have just released version 2.4 of my ESXi-Customizer-PS script - a PowerCLI script to create customized ESXi installation images and the successor of my legacy ESXi-Customizer tool.

This is a minor, but important update, because it fixes an annoyance that users of the latest PowerCLI version 6.0 ran into.

ESXi 6.0 hits the VMware Online Depot - What does that mean?

On April 9th VMware published the first ESXi 6.0 patch (see the ESXi Patch Tracker and KB2111976), and at the same time the Imageprofiles of both this patch and the ESXi 6.0 GA release appeared in the VMware Online Depot. Unfortunately they did not also upload all the ESXi 6.0 VIB packages at that time, but only the one that was updated with the first patch. Today, they finally made up for this, and I cannot only publish this article again, but now the instructions and examples in it will really work :-)

What does it mean to have ESXi 6.0 available in the VMware Online Depot? It makes it a lot easier to upgrade an ESXi host and to build customized ESXi 6.0 installation images, especially for free license users who do not have easy access to the GA Offline bundle. I wrote about this already in my Ultimate guide to upgrade your white box, and this is the promised update to it.

So what are the new possibilities now?

Watch out: ESXi 6.0 introduces root account lockout!

A few days ago I got an e-mail from one of my blog readers asking for help ... He had upgraded his host to ESXi 6.0 and since then he was repeatedly seeing events in the host log like the one displayed above:

Remote access for ESXi local user account 'root' has been locked for 120 seconds after xxx failed login attempts.

At the same time he could no longer log in to the vSphere client although he was using the correct password. I immediately thought: Wow, this is new in 6.0, never heard of this before! What the heck was going on here?

vCSA 6.0 tricks: shell access, password expiration and certificate warnings

It is no secret anymore that VMware is putting more and more energy in getting rid of the Windows based vCenter product and pushes us towards the Linux based vCenter Server Appliance (vCSA). Up to now the appliance was already a good choice for labs and other small deployments, but it lacked the scalability of the Windows version, and was not a good fit for large production deployments.

With the release of vSphere 6 this has changed due to the evolvements of the vCenter architecture and significant updates that were made to vCSA. The biggest remaining caveat now is that you still need a Windows machine for VMware Update Manager (VUM). I do not really use VUM in my lab, so I will definitely switch to the vCSA - whenever the time will come for me to do the 6.0 upgrade.

So I took a closer look at vCSA 6.0 ... and stumbled over some issues that are definitely worth sharing.

vSphere 6 is GA: The ultimate guide to upgrade your white box to ESXi 6.0

So finally VMware made vSphere 6 generally available (GA)! I already wrote about what we can expect from ESXi 6.0 in terms of white box support and using the free license, and I bet that a lot of people cannot wait to have their boxes upgraded ...

In this post I will explain how you can download ESXi 6.0, what you need to take care of with regards to unsupported hardware and software, how you actually do the upgrade, and how you can build your own customized ESXi 6.0 installation ISO for new installs.

Top vBlog 2015 voting is open now!

Finally ... Eric Siebert over at vSphere-Land.com has kicked off this year's Top vBlog voting - this time sponsored by Infinio!

The yearly vBlog voting is a great opportunity to show your support for your favorite virtualization blogs. Keep in mind that this is not a popularity contest - do not just vote for the blogs that everyone votes for every year, but pick those that had the best value for yourself!

You will find this blog in the overall favorites list and in the category "Best Independent Blogger".

Head over to the Top vBlog 2015 site to cast your vote - Thanks!!

New service: The VMware ESXi Patch Tracker

A while back I was looking for an easy way to stay up to date with VMware ESXi patches. VMware provides a lot of RSS feeds that keep you updated on new blog posts, new KB articles, security advisories etc., but ... I could not find anything that would just ping me whenever an ESXi patch was released.  The closest thing that I found was a link named "Get New Patch Alert" on the My VMware Patch Download Portal. That will refer you to the profile settings of your MyVMware account where you can manage various subscriptions including something that should keep you updated on new releases and patches of all sorts of VMware products. However, this just never worked for me, I never got a single e-mail out of this (anyone?), so I must assume that this service is not functional.

I finally decided to develop something on my own and provide that as a service to the community. So here is what I call the ...

VMware launches vSphere 6 - What's in ESXi 6.0 for free license and white box users?

The cat is out of the bag and it wears the number 6. VMware has announced its much anticipated new major version of their flagship product vSphere, and right now the virtualization blogosphere is humming with the news about vSphere 6.0.

Since this is an enterprise product most of the exciting new features are interesting for large installations using paid licenses and vCenter managed hosts: The long awaited ability ro use the Fault Tolerance (FT) feature with VMs that have more than one vCPU (SMP FT) is something that you can try out in your home lab if you have multiple hosts and use vCenter with an evaluation license, but the new support for virtual storage volumes (vVol) requires a modern SAN array with proper hardware support - nothing that you will find in the average home lab (at least not today). The ever increasing VM scalability now allows you to run VMs with 128 vCPUs and 4 TB RAM on physical hosts that can have up to 480 pCPUs and 12 TB RAM. Even in enterprise production environments it will be difficult to find setups that come even close to these numbers...

But what's in vSphere 6 that is useful for users of the free ESXi license and home labs?

pfSense 2.2 was released - How to install VMware Tools (sort of)

Update 2016-05-04: If you are using pfSense 2.3 (or newer) then please do not follow this guide, or it will break your system! Use the pfSense Package Manager to install the open-vm-tools package instead!

I'm a big fan of pfSense, an Open Source firewall and router appliance, that I use in my hosted lab. Recently version 2.2 of pfSense was released with a lot of bugfixes and new features. Please review the announcement blog post to find out what's new.

In my lab I upgraded one of the "not so important" pfSense VMs to version 2.2 and then tried to get VMware Tools installed and running again following my own guide that I wrote a while back for pfSense 2.0 and 2.1. But, well, things have changed a lot with pfSense 2.2, because it is now based on the latest FreeBSD version 10.1. Here is what I found out and what I ended up with.

What's new in the V-Front Online Depot

Since I launched the V-Front Online Depot about a year ago it has developed into a place where you do not only find the ESXi software packages that I created on my own, but also quite a few applications and drivers that others have contributed.

In 2014 Raphaël Schitz (he runs the excellent french hypervisor.fr blog) e.g. created and contributed the Intel Infiniband Subnet Manager (ib-opensm) package and the iperf v2 tool for ESXi, but this time I want to highlight two community developed driver packages that will help people running ESXi on white boxes with unsupported NICs.