The great VMware Tools dilemma

Recently VMware made VMware Tools version 10.0.0 available as a standalone download. This version is now an official downloadable component of vSphere 6.0 with its own Release Notes document and a download page in MyVMware.

In the announcement blog post VMware's Brian Graf writes:

Good news. We have decided that there isn’t any specific reason that VMware Tools builds should be tied to vSphere releases/ESXi builds. Rather, when our engineering teams are ready with key features/updates, we should have the ability to get those benefits out to our customers as quickly as possible.

This release was announced and perceived as a great achievement, but - except for this good news that Brian shared with us - I cannot follow any of the other excited statements that were made about it. It's time to debunk some myths, and it's time to admit that the whole VMware Tools story is still a great mess!


Myth #1: Finally we have Release Notes for VMware Tools

People are complaining a lot about that they have no insight in what was changed or fixed with the various VMware Tools updates that happened in the past. Making version 10.0 available as a "stand-alone" download with its own Release Notes page is supposed to change this.

But, well, the original assumption is wrong: For every updated VIB package of ESXi VMware publishes a dedicated KB article that describes the updates. And that is also true for the tools-light VIBs that represent the embedded VMware Tools package of ESXi. You can consider these KB articles release notes, but VMware makes them really hard to find. However, you can get an easy overview by looking at my ESXiPatches site which summarizes all updated VIB packages of every ESXi (5.x and 6.0) patch and provides links to all those KB articles.

As examples you will find that the release notes of the VMware Tools embedded in ESXi 6.0 U1 are in KB2124717, and the release notes of those embedded in ESXi 5.5 U3 are in KB2110233.


Math #2: Finally we are able to update VMware Tools independently from ESXi patches

The fact is that we have always been able to do this! And this is for two reasons both of which still too many people are not aware of:

1. Every ESXi embedded VMware Tools release (starting with ESXi 3.5!) was and still is made available as a stand-alone download at http://packages.vmware.com/tools/esx/ (for Windows and various Linux Guest OSs). You can download any version there and install it in a VM, no matter what ESXi version the host has that runs the VM.
By the way the standalone version 10.0 is also available here in the directory /tools/releases.

2. VMware Tools releases are - to a great extent - compatible with earlier and newer versions of ESXi. E.g. the Product Interoperability Matrix of the VMware Tools from ESXi 6.0 U1 shows that these are interoperable with older ESXi releases down to 5.0 GA. And the VMware Tools from ESXi 5.5 U3 are even compatible with the full range from ESXi 4.0 to 6.0 U1.


Myth #3: Finally VMware will decouple VMware Tools from ESXi releases and concentrate on developing a single version

This was really my big hope when version 10.0 was published, until ... VMware released ESXi 6.0 Update 1 and 5.5 Update 3, and both updates came with yet another embedded VMware Tools package. And no, these are not identical with the standalone version 10.0! ESXi 6.0 U1 comes with VMware Tools version 9.10.5, and ESXi 5.5 U3 comes with version 9.4.15.


The 50+ Shades of ...

It is the sheer variety of VMware Tools versions that alone can drive you crazy: With the different ESXi patches and updates that were published (since ESXi 5.0 GA) VMware made available no less than 59 different VMware Tools versions, and the standalone version 10.0 is number 60! Lots of opportunities to update Tools, but also lots of opportunities to screw up!

(If you wonder how I came up with this number: I counted the different versions of the tools-light VIBs in my VIBMatrix.)

Why is that?


Why were VMware Tools embedded into ESXi in the first place? ...

So how do you install or upgrade VMware Tools on your VMs today? If you follow VMware's documentation for a manual installation or upgrade (e.g. on Windows VMs) then you use the VM's context menu dialog in the legacy vSphere Client (or the Web Client) to mount the VMware Tools ISO file to the VM and then launch the install from the virtual CDROM inside the VM (if that does not happen automatically). This can also be initiated from the VM console window of the legacy vSphere Client:

Obviously this is only possible, because the ESXi host that runs the VM has access to the appropriate VMware Tools ISO file, and that was installed on it via the tools-light VIB package.

For sure this is a very convenient way to install VMware Tools, and it also works for standalone hosts utilizing the free license. It also allows the ESXi host to upgrade Tools automatically on VM startup, and in combination with vCenter and Update Manager you can even orchestrate a VMware Tools update for multiple or all of your VMs.

So the VMware Tools were embedded into ESXi, because this enabled VMware to implement powerful functionality into ESXi and vCenter for deploying VMware Tools on your VMs and keeping them up-to-date.


... and what issues does this cause?

For a lot of people it is confusing that a VM's Tools Status (Current vs. Out-of-date) that is displayed in the vSphere Client and the Web Client depends on the patch level of the ESXi host that runs the VM. It might change when a VM is migrated between environments with different ESXi patch levels, or when you are in the progress of upgrading the hosts within a cluster.

After hosts are patched it very often happens that the VMware Tools of all VMs are shown as Out-of-date, because the hosts got a newer version of the Tools with the patch. You then feel urged to update the VMware Tools on all VMs (causing reboots and application downtimes) just to make this bad status disappear. Usually people have a hard time to figure out whether this is really necessary, because VMware does not offer a clear guidance. This is why there are different approaches today when it comes to updating VMware Tools, and all of them seem to be valid: Some only update the Tools when they know that the new version fixes an issue that they have. Some only update them with ESXi Update releases or new major versions. Few try to pro-actively update Tools on all VMs quickly after patching the ESXi hosts (which is a huge effort in large environments). Another common approach is to use regular maintenance windows for the VMs to update Tools to the latest release. Almost all these approaches have two things in common: You will never see that all your VMs have up-to-date Tools, and - after the next ESXi patch session - the whole story starts again ...

Few people know that you can break this vicious cycle by updating your VMs with Tools that are considerably newer than your ESXi version, but still backwards compatible. But this will only work if you are not already on the latest vSphere version and only temporarily - until your vSphere version and patch level gets ahead of the Tools version again.

In a blog post back in 2013 I explained how you can tackle the problem of VMware Tools version diversity in your environment, and it was Brian again who brought this idea up again in a recent VMware blogs article: You can configure an ESXi host to look for the VMware Tools ISO files in a central location (e.g. an NFS datastore) that it can share with other ESXi hosts. So you can have one central repository for all your hosts, and thus a single point to update the Tools only once for all hosts. This is great, and I'd recommend every serious VMware admin to look into this possibility! However, there are still two drawbacks:

1. It's a cumbersome manual process to copy the VMware Tools files from an ESXi host (that has them installed in a regular way) to the shared location.
2. It does not seem to be easily possible to have the new standalone version 10.0 of the Tools in such a shared location. Doh!


Ways out of the dilemma

I have three proposals for improving the current situation. I want VMware to ...

1. Stop embedding VMware Tools into ESXi!

Please deliver on the promise to decouple the Tools from vSphere/ESXi releases and patches: Make future version of VMware Tools available as standalone downloads only, like the current 10.0 release!

To be fair and clear: It is already possible to install ESXi without the tools-light VIB, so that it does not have the Tools embedded (by using the -no-tools Image profiles of the various patch releases), but I really want VMware to no longer provide the embedded Tools at all! I think this would free up a significant amount of developers' resources, so that they can fully concentrate on only one set of the Tools (starting with version 10.0).

2. Develop the idea of a central repository and make it a user-friendly feature

As I explained before the motivation for embedding the VMware Tools into ESXi is to make their life cycle (installation and upgrades) manageable within ESXi and vCenter without the need for any third party software. By further developing the idea of using a shared central repository VMware could keep this functionality even if the Tools are no longer embedded in ESXi. How about using an NFS share on the vCenter server as the repository (or on a dedicated appliance to not exclude non-vCenter customers) and providing an easy to use management interface for updating the Tools files in the central location? This would be a great way moving forward ...

On a more radical approach one could even question if we need VMware to manage the Tools' life cycle at all! Actually the VMware Tools are just a software package that you install in your VM guest, and in Enterprise environments IT people use modern software distribution systems (like e.g. Microsoft SCCM for Windows) to deploy software. If you have such a system available you would also use it to deploy and manage VMware Tools!
For Linux VMs you typically manage software by using built-in tools that access the vendors' (or your own) central software repositories, and consequently VMware and most Linux vendors have leveraged the Open Source version of the VMware Tools (the open-vm-tools package) and include that in their repositories.

3. Stop telling us that Tools are "Out-of-date". It's useless!

Why? The Tools status "Out-of-date" just tells us that the ESXi host has a newer version of the Tools embedded than what is currently installed on the VM. So what? Do I need to update the Tools now to fix a potential issue that I'm maybe not yet aware of? But is there not also a risk that only the update will introduce a new issue that did not exist before?

There is no such thing as bug-free software, so there have been quite a few known issues with VMware Tools in the past. Instead of "Out-of-date" I'd really love to see a new status named like "Update needed" that is only displayed if there is a known issue with the Tools version that is currently installed which would be fixed by a new version. This status display would be ideally implemented as a link that opens the relevant KB article in a browser window when you click on it.

I could also think of a status "Update recommended" that tells the user that a newer version of the Tools is available which has new useful features.


What do you think?

Am I exaggerating, or are my ideas just stupid? I'd really love hearing other peoples' opinions - well, always -, but particularly on this topic! It's also a great opportunity to chime in and let VMware know that they need to change something with regards to VMware Tools. So please comment, thanks!

---

This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.

---

---