A myth busted and an FAQ: ESXi is *not* based on Linux! But what is it?

I have often heard or read the statement "ESXi is based on Linux" in this or some other variation when technical people discuss about how certain things work (or are supposed to work) in ESXi. This statement is not only plain wrong, but just doesn't make any sense! In fact it is like saying that horses developed from cows, because both have a head, a tail and four legs ...

In this post I will explain why, and I also answer some other questions that are closely related to this discussion. The first one - surprisingly - is ...

What is Linux?

"Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution." (Source: Wikipedia). It is important to understand that there is not one Linux (like there is e.g. one Microsoft Windows), but a multitude of it, each assembled and distributed by different parties: There are community supported free distributions like Debian, Ubuntu and Fedora and commercially licensed and supported distributions by e.g. Redhat and Novell/SuSE.

All Linux distributions have one thing in common: The Linux kernel. The Wikipedia article quoted above calls it the defining component of Linux. Almost every operating system is built upon a kernel which is a software layer that manages all hardware of a computer - it provides e.g. CPU scheduling, memory management and device drivers on the one hand, and defined interfaces to the application processes that run on top of it on the other hand.

The first version of the Linux kernel was authored by Linus Torvalds in 1991. He is commonly referred to as the creator of Linux.

There is also a wide range of free and Open Source software that you will usually find in any Linux distribution: The X window system for graphical applications, desktop environments like Gnome and KDE, the web browser Firefox, and server services like the Apache http server, etc.

So, ESXi is just another Linux?!

The answer to this question is a clear No, because ESXi is not built upon the Linux kernel, but uses an own VMware proprietary kernel (the VMkernel) and software, and it misses most of the applications and components that are commonly found in all Linux distributions.

What do ESXi and Linux have in common!?

So ESXi is not based on Linux, but why do many people believe this? I think the main reason for this is that you can remotely open a terminal session on an ESXi host via the ssh protocol (if it has been enabled). And if you are in such a session you can use lots of commands (like cd, ls, df, grep, ps etc.) that are also available in a Linux (or other Unix-) shell.

This is possible, because a software called busybox - a lightweight shell that is able to interpret and execute these commands - runs as a userworld of the VMkernel. But busybox and all other Unix-like tool that are available in an ESXi shell are not Linux things. They are available in any Unix-like operating system and can even run on Windows (e.g. in a Cygwin runtime environment)!

If you have ever taken a closer look at the ESXi internals that the shell exposes then you might have noticed more similarities to Linux (and other Unix-like operating systems), e.g.:

• The typical directory structure with e.g. /usr, /bin, /sbin, /var, /etc, /dev, /lib and /tmp
• A SysV-style init process that uses e.g. the configuration file /etc/inittab and scripts in a directory /etc/init.d to start services at boot time
• Shared Libraries of the GNU C compiler in /lib
• In ESXi 5.1 there is even the Xorg X server installed! I believe that it is used to implement the new virtualized GPU feature in vSphere 5.1 (see number 5 in my vSphere 5.1 top new features post)

ESXi even uses the same binary format for executables (ELF) than Linux does, so it is really not a big surprise anymore that you can run some Linux binaries in an ESXi shell - provided that they are statically linked or only use libraries that are also available in ESXi! (I exploited this "feature" when describing how to run HP's hpacucli tool in ESXi and when building the ProFTPD package for ESXi).

After reading all this you might wonder "How can I tell that ESXi is not based on Linux then?". It is because all these concepts and software exists independently from Linux (and the Linux kernel!) and are used in many operating systems, resp. are implemented, maintained and distributed as OS-independent, "autonomous" packages. VMware just re-used all these mature concepts and software and this was in fact way smarter than re-inventing all these wheels! That means: ESXi and Linux are not based on each other, but share a common fundament.

But you can even use Linux device drivers in ESXi!

Not directly ... you cannot use binary Linux driver modules in ESXi. Lots of Linux device drivers can be adapted to ESXi though by modifying their source code and compiling them specifically for ESXi. That means that the VMkernel of ESXi implements a sub-set of the Linux kernel's driver interfaces, but also extends and adapts them to its own hypervisor-specific needs.

In my opinion this was another very clever move of the VMware ESXi architects and developers, because it makes it relatively easy to port an already existing Linux driver of a hardware device to ESXi. So the partners that produce such devices do not need to develop ESXi drivers from scratch. And it also enables non-commercial community developers to write device drivers for devices that are not supported by ESXi out-of-the-box!

However, you should be aware that the VMkernel only implements those driver interfaces that are necessary for acting as a hypervisor: that is mainly the network and storage stacks. You will e.g. not be able to easily port a driver for a scanner device or non-standard serial hardware to ESXi. The same applies to drivers for wireless LAN adapters, because they depend on kernel subsystems, other low-level drivers or management tools that are not available in ESXi and the VMkernel.

But ESXi is a kind of Unix, right?

Another clear No here! An operating system may only be called a "UNIX" if it fully implements the Single UNIX specification of the Open Group and has passed the associated certification process. ESXi does not (and was probably never meant to) implement all the necessary functionality and APIs that make up the UNIX standard.

You might call VMware ESXi Unix-like, but even this is disputed by official sources: Mike Foley - a Senior Technical Marketing Manager at VMware - recently explained in an interesting blog post titled "It's a Unix system, I know this" that "A shell does not an OS make". His main point is that you do (or at least should) not manage an ESXi host through the shell, but via the management tools and scriptable APIs that are available for it, e.g. through PowerCLI. Or in other words: ESXi is not designed as a "Unix", but as an embedded system that is managed through defined interfaces (of which most are not really Unix-like).

Okay, but what is ESXi then?

I used many words here to describe what ESXi is not. But how do explain what it actually is? Here is my try: ESXi is

• a bare metal (or type-1) hypervisor
• that is purpose-built,
• designed as an embedded system,
• incorporates free Open Source software components, but also uses
• a unique proprietary OS kernel - the VMkernel -
• which implements unique features that are not available in any other operating system (e.g. the VMFS and VisorFS file systems.

Wait ... classic ESX is different - That is based on Linux, right?

Those of you who have ever touched or even installed a classic ESX installation might know that it uses a Redhat Linux installation to boot itself and to provide management functionality. This Linux is the so-called Service Console OS (or short COS) of ESX. Its Linux kernel loads the VMkernel as a module and then passes complete control over the system to it. The VMkernel hypervisor in turn will then run the COS as a privileged VM.

So classic ESX really incorporates a Linux and depends on it for getting started and managed. According to this particular meaning you can really say that classic ESX is based on Linux, but that does not mean that the VMkernel and the VMware specific OS components developed from Linux. It just uses it as a piggyback.

The classic ESX architecture has probably very much boosted the perception that "ESX(i) is based on Linux". But with the rise of the ESXi architecture and the associated elimination of the COS the classic ESX architecture is obsolete now. The VMkernel of ESXi boots directly on the hardware and does not need a Linux OS anymore.

However, the classic ESX architecture specifically caused a lot of concerns and very controversial discussions about licensing topics and legal issues, and that brings us to the last question:

Does VMware ESX(i) violate the GPL?

As explained above ESX and ESXi incorporate quite a few Open Source software that is licensed under the GNU Public License (GPL). The most important terms and conditions of the GPL are the following:

1. The GPL grants the right (or "guarantees the freedom") to "use, study, share (copy) and modify the software" (source: Wikipedia).
2. Binary distributions of the software must be accompanied by its source code (to enable 1.).
3. In contrast to general believe GPL licensed software does not need to be provided for free (i.e. at no cost), but it can be sold.
4. Software that was derived from GPL licensed software must be itself licensed under the GPL.

VMware fulfills condition (2) by providing the source code of all GPL licensed software that is used in ESX(i) (and other VMware products). The Open Source code of the products are usually made available together with their binary downloads, see e.g. the download page of vSphere 5.1.

Back in 2008 there were voices claiming that the VMkernel is a derived work of the Linux kernel (and thus violates condition 4.), because of the fact that the VMkernel is loaded and initialized as a Linux kernel module in classic ESX. A post at Venturecake.com suggested this and got a lot of media coverage and lead to very lively discussions involving prominent people like the Linux kernel hacker Alan Cox and VMware's Zachary Amsden.
Venturecake.com has vanished from the Internet a long time ago, but those of you who want to travel back in time and enjoy history lessons will find this article and the attached discussions archived at the Wayback machine. Enjoy!

The accusation that VMware violates the GPL became obsolete with the architecture change towards ESXi, and - up to now - it was never decided in front of a court, because no one ever sued VMware for this. That means there is no proof as of today that VMware ever violated the GPL.

---

This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.

---

---