vSphere 6 is GA: The ultimate guide to upgrade your white box to ESXi 6.0

So finally VMware made vSphere 6 generally available (GA)! I already wrote about what we can expect from ESXi 6.0 in terms of white box support and using the free license, and I bet that a lot of people cannot wait to have their boxes upgraded ...

In this post I will explain how you can download ESXi 6.0, what you need to take care of with regards to unsupported hardware and software, how you actually do the upgrade, and how you can build your own customized ESXi 6.0 installation ISO for new installs.


1. How to get the bits

I you have a paid vSphere license under subscription then you can download ESXi 6.0 (and all other vSphere 6 components that you are entitled to) from the My VMware Downloads page.

Even if you will use ESXi only with a free license key you will need a My VMware account for the following steps. If you are already running ESXi then you most probably already have one - if not then register now. It's easy and free.
The free license offering has already been updated to 6.0, so you can then go to the vSphere Hypervisor page to register and download ESXi. There you will also find the license key that you need to enable the features of the free version after installation.

The ESXi installation image can be downloaded in two different formats: On the one hand there is the ISO file (named VMware-VMvisor-Installer-6.0.0-2494585.x86_64.iso) that you just burn to a CD media and use to boot your machine and install or upgrade ESXi on it. On the other hand there is the Offline Bundle (named VMware-ESXi-6.0.0-2494585-depot.zip) that you can use to update ESXi directly from the command-line using esxcli. You also need the Offline Bundle to customize the ESXi installation and build your own installation ISO (e.g. with additional drivers).

Unfortunately the Offline Bundle is - right now - only available to you if you have a paid vSphere license under subscription registered with your MyVMware account! On the download page of the free vSphere Hypervisor you will only find the ISO file, and even if you register for a vSphere evaluation you will not be able to download the Offline Bundle, but only the ISO file.

Confused already? Here is another option: A different way to download ESXi is through the VMware Online Depot. The primary purpose of the VMware Online Depot is to serve as a synchronization source for VMware Update Manager (VUM), the product that you use to patch/update ESXi hosts that are managed centrally through vCenter. However, since it is all about patching the ESXi 6.0 GA bundle is usually not available in the Online Depot right away at the GA date, but is added there some time later (at least this is how VMware handled it for the release of ESXi 5.5).
My new service, the ESXi Patch Tracker, can notify you once the ESXi 6.0 GA bundle has been added to the VMware Online Depot, because it does nothing else than querying exactly that. I suggest that you subscribe to its RSS feeds and/or follow its Twitter Bot @ESXiPatches to get the news just in time!
Once the ESXi GA Imageprofile is available in the VMware Online Depot you can upgrade your host directly from it provided that your host has a direct Internet connection. For an offline upgrade you can then also use my ESXi-Customizer-PS script to create both the ESXi 6.0 installation ISO and Offline Bundle. But right now this is not yet an option, and I will explain that in a later post to not further confuse you :)


2. Beware of driver blacklisting: Identify your devices and drivers used

Standard disclaimer / warning The workaround described in the following sections and any software package that you install through the V-Front Online Depot are neither endorsed nor supported by VMware! Use at your own risk and never on production systems!

When VMware released ESXi 5.5 in 2013 they decided to drop some drivers for NICs from it that are commonly used in consumer grade hardware: r8168 and r8169 for Realtek chips, sky2 for Marvell chips (and less common: s2io for Neterion 10Gb cards). Luckily you could still keep them installed when upgrading from ESXi 5.1 to 5.5 or install them later on top of ESXi 5.5.

With ESXi 6.0 VMware goes a step further and actively prevents installing these drivers by using a blacklisting method that is based on VIB package dependencies! The ESXi system package misc-drivers is flagged to replace the above mentioned driver packages which means that you cannot install misc-drivers and any of these drivers - misc-drivers though is absolutely necessary for ESXi so you must install that!
As a workaround I created new packages that just includes the driver files of the net-r8169 resp. net-sky2 packages, but use a new and different name: net51-r8169 and net51-sky2. As a replacement for net-r8168 please use the updated net55-r8168 driver!

So, are you using one of the blacklisted drivers? Find out by running the command

esxcli network nic list

in an ESXi shell. This command will list all the NICs that are in use on your system together with the associated driver name (in the third column). If this command lists r8168, r8169, sky2 for you then you need to install one of the replacement packages like this - before upgrading to ESXi 6.0:

# Lower the system's acceptance level to allow Community supported packages
esxcli software acceptance set --level=CommunitySupported
# open firewall for outgoing http requests:
esxcli network firewall ruleset set -e true -r httpClient
# Install e.g. net55-r8168 from the V-Front Online Depot
esxcli software vib install -d https://vibsdepot.v-front.de -n net55-r8168
# Reboot
reboot

After a reboot your host will work exactly as before.


3. Check installed packages and take appropriate actions

Before doing the actual upgrade it's a good idea to check if you have other officially unsupported ESXi packages installed on your host, and how they will behave with ESXi 6.0. Let's go through the packages that are available through the V-Front Online Depot:

• net55-r8168: As stated above it is safe to keep this driver with ESXi 6.0.
• net-e1000e: This driver by GLRoman updates the ESXi builtin e1000e driver and supports additional Intel Gigabit NICs. In ESXi 6.0 VMware has also updated its own e1000e driver, and - as fas as I could check that - the VMware supplied version in ESXi 6.0 now supports the same set of devices as GLRoman's driver. However, please do not uninstall GLRoman's driver prior to the ESXi upgrade, because this could potentially render your host unable to boot (because of "no supported NIC found"). Instead follow the clean-up steps below (section 5) after the upgrade!
• net-tulip: This package allows to run nested ESXi hosts under Microsoft's Hyper-V. I have not tested that, but I expect that it is safe to keep that and that it will still work with ESXi 6.0.
• sata-xahci: This package enables support for additional SATA AHCI controllers. It is still required with and compatible with ESXi 6.0. Keep it if you have that installed.
• cpu-microcode: This package updates CPU microcode files. It is incompatible with ESXi 6.0, but you do not need to uninstall it prior to the upgrade, because ESXi 6.0 comes with a VMware-supplied package with the same name, and that will just replace the community supported package.
• ProFTPD: I quickly tested this FTP server with ESXi 6.0 and it worked. However, some users reported that they had issues with this package on ESXi 5.5 that affected system stability, so I no longer recommend using it. Uninstall it with
     esxcli software vib remove -n ProFTPD
  (no reboot required)
• esxcli-shell: This is an esxcli plugin that allows you to run any ESXi shell command through esxcli. It still works with ESXi 6.0, and it is safe to keep it. However, I wonder for what purposes this can still be useful (Please let me know if you use it!)
• fw-ntpd: Adds an ESXi firewall rule to allow incoming NTP traffic. The firewall subsystem has not changed in ESXi 6.0, so this package is still compatible and functional with 6.0.
• esx-tools-for-esxi: The VMware Tools for nested ESXi hosts is a VMware fling that is very useful for virtual ESXi hosts in lab environments. Actually VMware also finds this so useful that it is now officially included in ESXi 6.0. Unfortunately an upgrade to ESXi 6.0 will not remove the old fling version, so I recommend doing this manually before:
     esxcli software vib remove -n esx-tools-for-esxi
  (requires a reboot)
• vmware-esx-dvfilter-maclearn: Another cool VMware fling that is very useful in nested environments. According to William Lam it is still needed in ESXi 6.0, but none of us has already tested if it is actually still working with ESXi 6.0. At least it survives an upgrade, and it looks like it is safe to keep it.
• ib-opensm and iperf: Raphaël Schitz - the author of these packages - will test them on ESXi 6.0 and let us know if they are compatible. I cannot make a recommendation for them right now, sorry!

If you are unsure then you can check whether you have any of these packages installed by running

   esxcli software vib list | less

in an ESXi shell. This will create a list of all installed packages that you can scroll through using the cursor keys - press Q to return to the command prompt. You can also search for a specific package with

   esxcli software vib list | grep -i <package-name>

4a. Upgrade to ESXi 6.0 using the Offline Bundle

So, now we are finally ready to upgrade your white box to ESXi 6.0! If you managed to get your hands on the Offline Bundle then you can use the following procedure:

1. Upload the ESXi 6.0 Offline Bundle (VMware-ESXi-6.0.0-2494585-depot.zip) to the root directory of a datastore that is accessible for your host. Use the the legacy vSphere Client like this:

Host Configuration - Storage - Datastores - Browse Datastore...: Note the datastore name ("VMStore03" in this example)!

In the datastore browser window select the root directory (/) on the left, click on the Upload icon and select Upload File... from the dropdown list. A file selection dialog will pop up now: Select the Offline Bundle from your local hard disk and it will be uploaded to the datastore then.

2. Log in to an ESXi shell and run the upgrade like this:

esxcli software profile update -d /vmfs/volumes/VMStore03/VMware-ESXi-6.0.0-2494585-depot.zip -p ESXi-6.0.0-2494585-standard

In the command string replace VMStore03 with the name of your datastore! Wait for the command to complete successfully. Then reboot and enjoy!


4b. Upgrade to ESXi using the installation ISO

If you are unable to get your hands on the ESXi 6.0 Offline Bundle and only have the ISO file to your avail then you can still upgrade your host by booting it from the ISO and following the built-in upgrade wizard. I have quickly tested that method with a nested ESXi 5.5 host, and it yielded the same result as the Offline Bundle method - the net51-drivers and sata-xahci packages that I installed before the upgrade were preserved!

Here are the steps:

Boot the machine from the ESXi 6.0 installation ISO and advance to this screen:

Press "Cursor Down" and Space here to select "Upgrade", press Enter and confirm on the next screen by pressing F11:

Watch the magic happen ...

... and finally ...

press Enter to reboot to the upgraded system.


4c. If something goes wrong

Please note: If something goes wrong and your system turns out to be unbootable after the ESXi upgrade then it should recover itself by rebooting again from the alternate bootbank that still contains the old system state. You can also manually fail back to the old state by following the KB article Reverting to a previous version of ESXi.


5. Clean-up for net-e1000e users

If you were using GLRoman's net-e1000e package before the upgrade then you are still using it after the upgrade, although the ESXi 6.0 built-in e1000e driver supports the same set of devices. GLRoman's driver will probably also work fine with 6.0, but if you want to maximize the supportability of your hardware then you can replace it with the official VMware driver package - provided that you have the Offline Bundle available. Run the following command to do this:

esxcli software vib install -d /vmfs/volumes/VMStore03/VMware-ESXi-6.0.0-2494585-depot.zip -n net-e1000e

Use the same path to the ESXi 6.0 Offline Bundle (following -d) that you were using for the upgrade! A reboot is required.


6. Create a custom installation ISO for ESXi 6.0

Do you prefer to do a clean install of ESXi 6.0 (instead of an upgrade), or do you want to install a new white box? My ESXi-Customizer-PS script will help you to create a customized installation ISO that has the necessary packages already included! However, right now this is only possible if you have the ESXi 6.0 Offline Bundle available. Run a command like

C:\scripts\ESXi-Customizer-PS-v2.3.ps1 -izip C:\TEMP\VMware-ESXi-6.0.0-2494585-depot.zip -vft -load net51-drivers,sata-xahci

in a PowerCLI session. You need to specify the full path to the ps1-script (at the beginning) and the full path to the Offline Bundle (following the -izip switch)! In this example the script is stored in the directory C:\scripts and the Offline Bundle is in C:\TEMP.

This command will create a customized ESXi 6.0 installation ISO in the current directory that has the net51-drivers and sata-xahci packages built-in. You can as well add any of the other packages that are available in the V-Front Online Depot by specifying their names in a comma separated list after the -load switch!

7. One more thing ...: USB 3.0 support

A while ago I wrote about USB 3.0 support being introduced in ESXi 5.5. Good news: This is also available in ESXi 6.0. Even better: On my test system the xhci-xhci driver that provides this support was loaded automatically on boot, so the workaround of editing the local boot script /etc/rc.local.d/local.sh is no longer needed.


8. To be continued / updated

This is the first version of my Ultimate guide to upgrade your white box to ESXi 6.0 - I hope you like it! I will complement it with a follow-up post as soon as the ESXi 6.0 GA Imageprofile is available in the VMware Online Depot (which will make upgrading ESXi and using ESXi-Customizer-PS even easier). And I will update this post with whatever information adds value. Please comment if you miss anything or have specific questions!

Update (2015-03-31):
Good news: If you use the new free VMware Software Manager tool to download ESXi 6.0 then your downloads will include the ESXi Offline Bundle - even if you have registered for the vSphere evaluation only!

---

This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.

---

---