How to build device drivers for ESXi 5.x

Since I published my ESXi-Customizer and ESXi-Customizer-PS tools to slipstream driver packages into an ESXi installation ISO I was asked dozens of times if I can provide a driver for the unsupported device xyz or whether I can give instructions on building such a driver. So far my answer was always No, because - although I'm quite familiar with using Linux - I do not have any of the Linux kernel hacking skills that I thought would be required to do this.

However, I was always aware that

• ESXi device drivers are derived from Linux device drivers,
• VMware has published the necessary source code to build the drivers that they ship with ESXi (because they are obliged to do this by the GPL license terms)
• over at the vm-help.com forums there are some people posting who have already successfully built ESXi drivers for unsupported white box devices

Last weekend I stumbled over two posts in the vm-help.com forum by a user with the nickname trickstarter:

• ESXI 5.x Drivers Part 1: Making a Build Environment
• ESXI 5.x Drivers Part 2: Preparing to compile

This is the first time that I found fairly detailed instructions on how to build ESXi 5.x device drivers, and it encouraged me to look at this myself.

Here are the required steps to set up a build environment for ESXi 5.x drivers and compile the already included drivers:

1. Download the ESXi 5.0 U2 Open Source code archive from MyVMware (select the file VMware-esx-open-source-5.0.0u2.oss.tgz). It looks like the OSS code of ESXi 5.1 was not yet published by VMware and is only available on request.

2. The archive includes a file BUILD.txt at the top level, and this file includes a description of the recommended build environment: It's a CentOS 5.3 x64 machine. This version is a bit aged now, but you can still download the installation ISOs from http://vault.centos.org/5.3/isos/x86_64/. There are seven CD ISOs, and I cannot tell which of them are really required, because I decided to download the complete DVD ISO that I found on another mirror server.

3. Install CentOS 5.3 in a VM using the downloaded installation ISO(s). The file BUILD.txt also includes a list of the software package sets that you need to select for installation.

4. Download the OSS archive into the VM and unpack it. Change to the sub directory vmkdrivers-gpl. There you will find another BUILD.txt file, and we now need to follow the instructions included in there. There is also the recommendation to run the following steps as the root user. This is against best practice - you should normally use root only if it is really necessary, but not for compiling code -, but in this case it makes life easier, because using root you can just follow the instructions and use the build scripts as-is without the need for any modifications.

5. The drivers' source code comes with its own gcc toolchain that you need to compile first. Enter the sub directories "glibc-2.3.2-95.44", "binutils-2.17.50.0.15-modcall" and "gcc-4.1.2-9" one after the other and execute the included BUILD.txt files with
   bash ./BUILD.txt
This will build the toolchain and install it in the directory /build.

6. Now unpack the drivers' source code:
   tar xzf vmkdrivers-gpl.tgz

and run the included build script:

   ./build-vmkdrivers.sh

It should build all existing drivers. You will see some gcc warnings, but no fatal errors.

At this stage you can now try to build an own driver by adding its Linux source code to this environment and creating a build script using the existing one as a template. This still requires some Linux hacking skills, and if you have never compiled software in Linux before then you will probably get lost quickly.

For more details and instructions I strongly recommend looking at trickstarter's posts that I mentioned above. He plans to make this a series of posts with even more parts, so join and stay tuned into the vm-help.com forums if you are seriously interested in this topic! This is also the right place to look for help if you get stuck.