FAQ: CPU microcode updates and VMware ESXi

"CPU microcode update available. The guest OS tried to update the microcode ..., but VMware ESX does not allow microcode patches to be applied from within a virtual machine."

I recently installed a Windows Server 2012 on a VMware ESXi 5.0 host running on an older HP hardware (Proliant DL380 G5). Each time Windows reboots the message above is logged on the ESXi host. According to the message text it looks like I can safely ignore it, since I do not have any issues with this host or the VM. However, this made me curious - I started some research on the CPU microcode topic and decided to write up a little FAQ:

Disclaimer: I thoroughly researched the information provided below, but if you find something to be inaccurate or plain wrong, or if you can answer any of the questions/uncertainties that I raise then please comment - Thanks! 

1. What is the CPU microcode?

In modern CPUs the microcode is used to assist in translating "high level" machine code instructions into various configuration signals and more detailed circuit-level operations. It helps separate the machine instructions from the underlying electronics (arithmetic logical units, memory channels, IO buses etc.) so that instructions can be designed and altered more freely.

For a more detailed definition, examples and an explanation of why this is beneficial please refer to the Wikipedia article about Microprograms/Microcode.

The microcode of a CPU can be updated to fix known CPU bugs. Yes, CPUs have bugs (also known as errata) that can lead to system instability, strange behavior, false machine check exceptions (MCEs) and OS crashes! Only some of these bugs can be resolved through microcode updates, for others there are workarounds implemented in the mainstream Operating Systems (like the AMD erratum #383), and in very few cases a bug cannot be resolved in software, but needs a redesign of the hardwired CPU logic, so that the buggy revision of a CPU needs to be recalled (this happened with the famous Pentium FDIV bug).

2. How is CPU microcode updated?

A microcode update can be loaded into a processor by using special CPU instructions that are reserved for this purpose. Such updates are not permanent, but are stored in volatile memory - that means with each power cycle the CPU is reset to its built-in microcode, and the update needs to be applied again.

Most commonly CPU microcode updates are applied through a computer's BIOS when it is powered on or reset. Most mainstream Operating Systems (incl. Microsoft Windows, Linux and VMware ESXi) also try to update the CPU microcode at boot time. Microsoft e.g. sometimes publishes Windows updates that include such microcode fixes (see e.g. KB2493989).

3. How does VMware ESXi update CPU microcode?

VMware ESXi 5.x uses the vmkmicrocode utility to update CPU microcode. At boot time the command
   vmkmicrocode -q -d /etc/vmware/microcode
is executed. This command will check if any of the microcode data files in the directory /etc/vmware/microcode (there are three dat-files for Intel processors and one bin-file for AMD CPUs) includes an update that is applicable to the system's CPU(s), and if this is the case it will load the update into each core of each CPU.

I took a closer look at the microcode data files and found them to be to a bit old: the newest Intel data file is of 2010, and in fact these files were not updated by VMware since at least the GA release of ESXi 5.0 (I have not checked earlier ESXi versions). For me it's hard to believe that the current CPU models of Intel and AMD do not have any bugs that affect ESXi and need to be fixed through microcode updates ... so I guess that VMware relies on the hardware manufacturers to include the necessary microcode updates in the BIOS of their servers and doesn't really care that much about the update files that they supply with ESXi themselves.

4. Should I even care?

It depends. If you are running ESXi on a VMware supported modern server hardware with the latest vendor BIOS applied then you usually do not need to care about microcode updates. In such cases known problems with processors will be addressed by the hardware vendor that will include CPU microcode updates in BIOS updates.

If you are running older or whitebox/unsupported hardware, experience strange issues with it and have already ruled out other possible causes (e.g. bad RAM, unstable power supplies etc.) then you might want to look at the CPU microcode.

In the very most cases you really shouldn't bother ... But if you are paranoid or an "Update Junkie" ;-) then read on!

5. Where do you get the latest CPU microcodes?

Intel makes CPU microcode updates available on their Download Center pages. You can search these pages for "Linux Processor Microcode Data Files" to get to the relevant download links. The current version (as of today) is of Feb 22nd 2013 and consists of multiple dat-files packed into a tgz-file.

AMD used to provide the same directly on a web site names http://www.amd64.org, but this site is no longer available (for the story behind that read this post on the Gentoo Forums). They now seem to feed these files into a Linux software package called linux-firmware. You can download this package by searching for it in the Launchpad.net Ubuntu repository and will find the microcode bin-files in the sub directory amd-ucode of the downloaded tgz-archive.
The latest version is of Sep 10th 2010 though - I wonder if this is really still up-to-date or if AMD has completely dropped the ball on that now ...

6. How do I apply updated microcode in VMware ESXi?

Once you have unpacked the data files from the above mentioned sources you need to upload them to a datastore of your ESXi host and can then use the vmkmicrocode utility to check whether they are applicable to your CPU and also to apply them.
The command
   vmkmicrocode -c /vmfs/volumes/DS1/microcode.dat

will check whether the updates that are included in the referenced data file apply to your CPU. It will output a message like

Warning: CheckUpdate failed, num=140 version=0x306a9, platformID=4000000000000, status=0xbad0001
Warning: processor match failed

for each update that is not applicable to your CPU. If there is an update included for your CPU then it will log a message like this:

PCPU0 with revision (a) can use the update in file /vmfs/volumes/DS1/microcode.dat
update number 128       version(1), revision(23), date(0x1092013), size(2048)

for each core of the CPU. In this case you can finally apply the update by using the command

   vmkmicrocode -f /vmfs/volumes/DS1/microcode.dat

7. How can I install a microcode update file in ESXi so that it is automatically applied at each boot?

There are two methods to achieve that:

1. Copy the data file to a datastore that is accessible by the host and add the command to apply the file (see above) to the local boot script /etc/rc.local.

2. Install a VIB file that I created using the commands

   esxcli network firewall ruleset set -e true -r httpClient
   esxcli software vib install -v http://files.v-front.de/cpu-microcode-1.0.0.x86_64.vib --no-sig-check

in an ESXi shell.
This will install the current versions of the Intel and AMD microcode files (from the sources mentioned above) into the default directory /etc/vmware/microcode. To manually check and apply these updates before the next reboot run the command "vmkmicrocode -q -d /etc/vmware/microcode" then.

Update (2013-10-30):

In the meantime Intel made updated CPU microcodes available and also AMD has published new ones! Please find a new version of the VIB file at the URL
   http://files.v-front.de/cpu-microcode-1.1.0-1.x86_64.vib

Install it like mentioned above and you will get the Intel files of 2013-08-08 and the AMD files of 2013-07-10. By the way, the new AMD microcode includes a fix for an issue that affects ESXi resulting in a PSOD during vMotion with Opteron 62xx CPUs! For details see this forums thread: https://communities.vmware.com/thread/432033.

Update (2013-10-31):

I was just made aware that there is an even newer Intel microcode file available (of 2013-09-06), so I quickly compiled another version of the VIB. Here is the URL
   http://files.v-front.de/cpu-microcode-1.2.0-2.x86_64.vib

The new Intel file includes important fixes for Haswell CPUs. Thanks to Евгений Рощин for pointing this out (I accidentally deleted your comment to this post, sorry!).

Please note: I also had to duplicate the data files in the new VIB file, because the vmkmicrocode utility of ESXi 5.5 now looks into vendor specific sub directories (amd and intel) of /etc/vmware/microcode. So, this is the first version that will also work with ESXi 5.5!

Update (2014-02-10):

I updated my cpu-microcode package to include Intel's updates of 2014-01-22 - Thanks to S. Catnipp for pointing me to the new version.
The package is now available at the V-Front Online Depot, and can be installed or updated from there by using

esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib install -n cpu-microcode -d http://vibsdepot.v-front.de --no-sig-check

in an ESXi shell.

---

This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.

---

---