A Daemon's VIB - Part 3 (Building a software package for VMware ESXi)

Part 3: Packaging, SecPolicy and VisorFS

This is the third and last part of the "Daemon's VIB" series about building a software package for ESXi, using the example of the ProFTPD Offline Bundle that I recently released. In the first part I explained how to get (or create yourself) suitable binaries for ESXi, in the second part we cared about the Daemon's configuration, firewall settings and automatic startup. In this last part we will
   a) build and install the actual software package
   b) deal with the VMkernel access security policies and
   c) make the service configurable to adapt to your own needs.

For our first try to build the ProFTPD software package we now have the following directory structure and files available:

etc/
etc/proftpd.conf
etc/vmware/
etc/vmware/firewall/
etc/vmware/firewall/proftpd.xml
etc/vmware/service/
etc/vmware/service/proftpd.xml
etc/init.d/
etc/init.d/proftpd
sbin/
sbin/proftpd

During my testing I kept this structure in a (otherwise empty) staging directory on a persistent datastore and regularly backed the files up into a tgz-file from within an ESXi shell:

# change to staging directory:
cd /vmfs/volumes/datastore1/proftpd
# create a tgz-file from this tree and store it in the parent dir.:
tar cvzf ../proftpd.tgz *

This tgz-file is not only a good backup of your work, we will also need it to create the Offline Bundle. Currently there are two different tools available to accomplish this: VMware's unsupported VIB Author tool and my ESXi5 Community Packaging Tools (ESXi5-CPT). I already blogged about how these two compare when VMware made its tool available as a so-called fling.

The Acceptance Level

One thing that we need to take care of when building a VIB package is choosing a suitable Acceptance Level for it. The tools that I mentioned are primarily meant for building packages of the Acceptance Level CommunitySupported. This is the lowest possible level (in terms of trustworthiness), and it is normally assigned only to packages that have not undergone any official certification process and are not supported by VMware or any other vendor. However, you can only change a very limited set of system directories with such a package (for details see this post), so it is not suitable for our purpose.

All the higher Acceptance Levels (PartnerSupported, VMwareAccepted and VMwareCertified) require specific certification processes and support statements either by the software vendor or VMware itself. To provide a technical verification method these packages are electronically signed, and by default the signature is checked at install time.

With a package of the Acceptance Level VMwareAccepted (short form: accepted) you can add files in arbitrary system directories, so this is for us. However, we will not be able to provide a valid electronic signature ... Luckily you can skip the signature check when installing, so this is the way that we will go with our own package. Please note: VMware does - of course - not recommend doing this, and you will render your system into an unsupported state by installing an unsigned package (for details see again this post).

Now let's look at how you build the package either with VIB Author or with the ESXi5-CPT:

Method 1: How to build the Offline Bundle with VIB Author

The information provided here is taken from the VIB Author manual. You need a Linux machine and some basic Linux skills to walk through the following steps:

1. VIB Author is shipped in RPM format. Download and install it using the package manager of your favorite Linux distribution. VMware recommends using SuSE Linux Enterprise 11 SP2, but other distributions should also work well if all dependencies (see System Requirements) are fulfilled.

2. Create an otherwise empty project directory (e.g. proftpd) and populate it as follows:

proftpd/
proftpd/descriptor.xml
proftpd/payloads/
proftpd/payloads/ProFTPD/
proftpd/payloads/ProFTPD/etc/
proftpd/payloads/ProFTPD/etc/proftpd.conf
proftpd/payloads/ProFTPD/etc/vmware/
proftpd/payloads/ProFTPD/etc/vmware/service/
proftpd/payloads/ProFTPD/etc/vmware/service/proftpd.xml
proftpd/payloads/ProFTPD/etc/vmware/firewall/
proftpd/payloads/ProFTPD/etc/vmware/firewall/proftpd.xml
proftpd/payloads/ProFTPD/etc/init.d/
proftpd/payloads/ProFTPD/etc/init.d/proftpd
proftpd/payloads/ProFTPD/sbin/
proftpd/payloads/ProFTPD/sbin/proftpd

The payloads/ProFTPD directory contains our project files, and you can easily fill it by unpacking the tgz-file that we created for backup and preparation (see above).

The descriptor.xml file needs to be manually created. The required XML tags and their meanings are explained in the manual, and there is a sample file included in the VIB Author package that you can copy and edit to fits your needs. For your convenience here is the file that I created and used:

<vib version="5.0">
   <type>bootbank</type>
   <name>ProFTPD</name>
   <version>1.3.3-8</version>
   <vendor>VFrontDe</vendor>
   <summary>ProFTPD (FTP Daemon) for ESXi 5.x</summary>
   <description>ProFTPD (FTP Daemon) for ESXi 5.x</description>
   <relationships>
      <depends/><conflicts/><replaces/><provides/><compatibleWith/>
   </relationships>
   <system-requires>
      <maintenance-mode>false</maintenance-mode>
   </system-requires>
   <acceptance-level>accepted</acceptance-level>
   <live-install-allowed>true</live-install-allowed>
   <live-remove-allowed>true</live-remove-allowed>
   <cimom-restart>false</cimom-restart>
   <stateless-ready>true</stateless-ready>
   <overlay>false</overlay>
   <payloads>
      <payload name="ProFTPD" type="tgz"></payload>
   </payloads>
</vib>

3. Now create a VIB file and the Offline Bundle by running the command

   vibauthor -C -t proftpd -v ProFTPD.vib -O ProFTPD.zip

from within the parent directory of the project directory.

The command will issue a warning that a VIB file with "accepted" acceptance level need to be signed, but will happily produce a VIB file and an Offline Bundle ZIP file for us.

Now let's look at how you do the packaging with the ESXi5 Community Packaging Tools:

Method 2: How to build the Offline Bundle with tgz2vib5 and vib2zip

The information provided here is also available on the ESXi5 Community Packaging Tools project page. You will need access to a Windows system (Windows XP or newer, Windows 8 is untested).

1. Download the latest versions of my tools from the project page. They are available as a self-extracting archive: Just run it to install it. All needed tools are included, there are no special requirements or dependencies.

2. Run tgz2vib5.cmd from the installation directory. This will launch a GUI in that you select the tgz-file that we prepared for backup and preparation (see above):

Building the ProFTPD VIB file with TGZ2VIB5 v2.1

In this GUI you also enter the package's metadata (section VIB Description Data) that will be part of the descriptor.xml file and select the Package properties that will also go in there. You can load presets for these properties, and in our case it is safe to load the presets for a firewall rule package. You only need to change the Acceptance Level from the default community to accepted.

After pressing the Run! button a VIB file named ProFTPD-1.3.3-8.x86_64.vib will be created in the working directory that you selected.

3. Now run the vib2zip.cmd script from the installation directory. This will launch another GUI:

Building the ProFTPD Offline Bundle with VIB2ZIP v1.1

For the source VIB directory select the working directory that you chose with TGZ2VIB5 before and that contains the VIB file that we created (Please note: All VIB files that are stored in this directory will be added to our Offline Bundle, so make sure that this is the only one in there).
Enter the same metadata that you used with TGZ2VIB5, and optionally add a KB-URL and a contact e-mail address.
Then press Run! to create the Offline Bundle. It will be stored as ProFTPD-1.3.3-8-offline_bundle.zip in the working directory that you selected.

Now that we have created an Offline Bundle with either the one or the other method let's install it and see how it works.

Installing the Offline Bundle

To install the bundle into an already existing ESXi host we need to upload it to a datastore of this host and run a command like

esxcli software vib install --no-sig-check -d /vmfs/volumes/datastore1/ProFTPD-1.3.3-8-offline_bundle.zip

in an ESXi shell (replace datastore1 with the name of your datastore). The --no-sig-check parameter is needed to skip the signature verification for our unsigned package.

The installation will run fine, but if you then test the FTP access to the host you will notice that the FTP daemon is not listening and the connection times out. Why?
When I stumbled over this error I checked the system's log file for any hints and suspicious error messages and found this in /var/log/vmkernel.log:

2012-11-12T14:14:20.305Z cpu0:5783)VmkAccess: 637: proftpd: access denied:: dom:appDom(2), obj:appDom(2), mode:inet_socket_bind(17) port no: (21)
2012-11-12T14:14:20.305Z cpu0:5783)VmkAccess: 207: ipAddr = 0.0.0.0, port = 21

access denied for inet_socket_bind?? It looks like the VMkernel does not allow the FTP daemon to bind to the FTP port 21! But this worked before when we executed the daemon from a datastore directory. So what is different now?

The VMkernel Access Control System (SecPolicy)

The difference is that the FTP daemon is now (after having installed it) a part of the system image. And as such it is under control of the VMkernel access security system. This system is managed by the secpolicytools command and configured by the files that you find in the directory /etc/vmware/secpolicy.

There is no publicly available documentation about this access control system, so everything that I was able to find out about it is based on reverse engineering (and might also be slightly or completely wrong). I believe that - by the configuration files that are stored in the above mentioned directory - security policies are defined that control what software packages (also referred to as tardisks) are granted what system permissions (like reading a file, writing a file, binding to a TCP socket, etc.).

Maybe I'm going to dig deeper into this one day, but for now I was glad to find out that you can grant the required permissions to our ProFTPD daemon by adding a new text file in the directory /etc/vmware/secpolicy/tardisks that includes the following line:

   -E unlabeled superDom

I named this file ProFTPD, but in fact its name has no relevance at all. It is only important that we make it a part of our software package! It is not enough to manually add the file after our package was installed, it needs to be made a part of and installed with our package.

I think that the above mentioned line will grant every possible system permission to our package, much more than it really needs. So it is kind of a quick-and-dirty solution, but limiting the permissions to exactly what is needed would cause a great testing effort and would also require a much better understanding of the security policy definitions than I have right now.

Armed with this knowledge we need to go back to the beginning of this post and add the following file and directories to our package:

etc/vmware/secpolicy/
etc/vmware/secpolicy/tardisks/
etc/vmware/secpolicy/tardisks/ProFTPD

Create a tgz-file with these updated contents and a new Offline Bundle, and after installing this new Offline Bundle the FTP Daemon will work just fine!

How to make the config file editable

When we prepared and tested our package by manually creating the associated files in the ESXi system directories we did not have any problems with changing, overwriting or editing these files. However, once we install the files as part of our Offline Bundle we are no longer able to change them. By default all files that are part of the system image are read-only, even if we assigned w(rite) permissions to them using chmod!

This is okay for the binary, init-scripts and firewall rules, because it prevents them from being accidentally overwritten or changed, but wouldn't it be fine if we could change the daemon's config file /etc/proftpd.conf after it has been installed? Obviously this must be possible in a way, because there are other ESXi configuration files stored in /etc that you can change (like /etc/hosts).

I found out how to do this after reading the excellent white paper VisorFS: A Special-purpose File System for Efficient Handling of System Images that was published by VMware Labs. This is a must-read for anybody who is seriously interested in learning how the ESXi file system works (I'm not referring to VMFS here, but to VisorFS which is the file system of the ESXi system itself!).
To say it short: If you set the so-called Sticky Bit for a file before adding it to the tgz-file (that we use to create the Offline Bundle) then this file will be editable after the Offline Bundle has been installed. The sticky bit is set with a chmod command like this:
   chmod +t etc/proftpd.conf

And what's even better: If such a file is located under /etc then it will be automatically saved to /bootbank/state.tgz by the script /sbin/auto-backup.sh that runs through cron every minute. And that means: The changes will persist upon reboots!

So after running through another iteration of creating a tgz-file (after setting the sticky bit on etc/proftpd.conf), building the Offline Bundle and installing it we finally reached the status of the ProFTPD Offline Bundle that I recently published ... and this also ends the "Daemon's VIB" series. I hope that my findings will help you with creating your own software packages for ESXi, and I welcome any feedback!