A Daemon's VIB - Part 2 (Building a software package for VMware ESXi)

Part 2: Service configuration and startup

This is the second part of the "Daemon's VIB" series about building a software package for ESXi, using the example of the ProFTPD Offline Bundle that I recently released. In the first part I explained how to get (or create yourself) suitable binaries for ESXi. In this part I will focus on what is necessary
   a) to make the service start up automatically on system boot and
   b) to make it fully manageable via the vSphere Client.

The Daemon's config file (/etc/proftpd.conf) ...

... is not something to look at in this context, but it is definitely needed, because the built-in configuration defaults of proftpd do not really fit to the specific runtime environment of ESXi. After some testing I ended up with the following config file and it is included in the package as /etc/proftpd.conf:

/etc/proftpd.conf for VMware ESXi
(Click on the picture to open it as a text file)

The comments should make everything pretty clear. For a complete overview of the available options please refer to the Configuration Directives document at proftpd.org.

Dealing with the firewall

ESXi has a built-in firewall that by default blocks all incoming and outgoing IP traffic except for the communication that was specifically white-listed. For testing the basic functionality of the proftpd binary I quickly disabled the firewall completely by running the command

esxcli network firewall unload

but this is - of course - not to be considered as a permanent solution. The right way to do this is to create a proper firewall rule file in XML format and store it in the directory /etc/vmware/firewall:

<!-- Firewall configuration information for ProFTPD -->
<ConfigRoot>
  <service id='0000'>
      <id>ProFTPD</id>
      <rule id='0000'>
          <direction>inbound</direction>
          <protocol>tcp</protocol>
          <porttype>dst</porttype>
          <port>21</port>
      </rule>
      <rule id='0001'>
          <direction>inbound</direction>
          <protocol>tcp</protocol>
          <porttype>dst</porttype>
          <port>
              <begin>49152</begin>
              <end>49999</end>
          </port>
      </rule>
      <rule id='0002'>
         <direction>outbound</direction>
         <protocol>tcp</protocol>
         <porttype>src</porttype>
         <port>20</port>
      </rule>
      <enabled>true</enabled>
      <required>false</required>
  </service>
</ConfigRoot>

So I created this file as /etc/vmware/firewall/proftpd.xml, and - instead of disabling the firewall completely - we need to make it reload its rule files including this new one by using

esxcli network firewall refresh

and automagically the network ports required by ProFTPD will be opened on the firewall. In addition you will see the new firewall rule in the vSphere Client (see Host Configuration / Security Profile / Firewall / Properties): There you can enable/disable the rule or limit the allowed IP addresses to specific subnet ranges.

If you are somewhat familiar with basic IP communication then the XML tags used here will be pretty self-explanatory. The main target is to define a service with the id/name ProFTPD and associate three rules (id 0000 to 0002) with it:

1. Rule 0000 allows inbound communication on tcp port 21, the standard control port that an FTP server listens on. It is defined as a destination (dst) port which enables FTP Clients to connect to it from the outside through the firewall.
2. Rule 0001 is needed for passive FTP connections. In these cases the FTP server will (passively) wait for the client to make a second connection for data transfers on an arbitrary tcp port (>1024). With ProFTPD you can limit the port range (see /etc/proftpd.conf above) used for the data connections, and this is what I have done here, randomly defining a range of 49152 to 49999.
3. Rule 0002 is needed for active FTP connections. That means that the server will actively connect the client for opening the data channel. It uses tcp port 20 for that, thus we enable it as a source (src) port for outbound connections.

(Side note: You can find more detailed explanations on the active vs. passive FTP topic on this page at slacksite.com. For my case I just decided to support both methods.)

All firewall rules defined here will be enabled by default. If you set the required-tag to true then you won't be able to disable this rule in the vSphere client (or through any other means).

When creating your own XML files for firewall rules there is one important thing to keep in mind: The XML tags and values are case sensitive! Typing <configRoot> instead of <ConfigRoot> will lead to the file not being accepted and used by the firewall rule engine, and typing proFTPD instead of ProFTPD might also lead to confusion and frustration later on ...

Automatic startup

ESXi comes with a number of built-in services that start up automatically when the system boots and are cleanly stopped when it shuts down. To enable this for our own service we need to create an init-script for it in the directory /etc/init.d. To create the init-script for proftpd I simply copied the script for the built-in ntpd service, stored it as /etc/init.d/proftpd and replaced every occurrence of ntpd with proftpd:

#!/bin/sh
#
# proftpd:
#   Start and stop the PROFTPD daemon
#
# chkconfig: on 10 90
# description: ProFTPD daemon.
#

PROFTPD="/sbin/proftpd"
PROFTPD_CONFIG="/etc/proftpd.conf"
PROFTPD_OPTS="-c ${PROFTPD_CONFIG}"

export PATH=/bin:/sbin

#
# Sanity check
#
[ -x "${PROFTPD}" -a -f "${PROFTPD_CONFIG}" ] || exit 0

#
# Log action
#
proftpd_log() {
   echo "${1}"
   logger proftpd "${1}"
}

#
# Start proftpd
#
proftpd_start() {
   proftpd_log "Starting proftpd"

   if [ -n "$(pidof -xs "${PROFTPD}")" ] ; then
      return 1
   fi

   "${PROFTPD}" ${PROFTPD_OPTS}
}

#
# Stop proftpd
#
proftpd_stop() {
   proftpd_log "Stopping proftpd"

   if [ -z "$(pidof -xs "${PROFTPD}")" ] ; then
      return 1
   fi

   pkill "${PROFTPD}"
 
   # Wait for processes to terminate properly before checking status:
   sleep 1 
   
   [ -z "$(pidof -xs "${PROFTPD}")" ]
}

#
# Process comand line options
#
case "${1}" in
   start)
      proftpd_start
   ;;

   stop)
      proftpd_stop
   ;;

   status)
      if [ -n "$(pidof -xs "${PROFTPD}")" ] ; then
         echo "proftpd is running"
         exit 0
      else
         echo "proftpd is not running"
         exit 3
      fi
   ;;

   restart)
      proftpd_stop

      # wait till all instances have stopped
      while [ -n "$(pidof -xs "${PROFTPD}")" ] ; do
         sleep 1
      done

      proftpd_start
   ;;

   *)
      echo "Usage: $(basename "$0") {start|stop|status|restart}"
      exit 1
   ;;
esac

When copying the script or creating it on your own make sure that the file has eXecutable permissions by using a shell command like chmod +x filename on it.

The script assumes that the daemon's executable is stored in /sbin (line 10) and its config file in /etc (line 11). Like every init-script it needs to be called with one of four possible parameters: start, stop, status and restart. It's obvious what these parameters will do ...

The results of the different actions are returned by the script as exit codes: The stop function e.g. will return (true=) 0 if the daemon was successfully stopped (line 57). By the way I had to insert a short sleep command before this check, because the proftpd process(es) take(s) some milliseconds to properly terminate after being sent a stop signal (line 52), and the test for still existing processes would be made too early otherwise.
It is important to have the init-script return the proper exit codes, because these will be checked by ESXi when you later start or stop the daemon through the vSphere Client interface! For

• start: return 1 (or false) if the service is already running or can not be started, 0 (or true) if the startup is successful (instead of checking this explicitly our script just returns the status code of the daemon itself).
• stop: return 1 if the service is already stopped or cannot be stopped, 0 if it is successfully stopped.
• status: return 0 if the service is running, 3 if not
• restart: same as start, because this is just executing stop and start consecutively (our script is not perfect here, because it will hang forever if the daemon can not be successfully stopped for whatever reason).

In the comment section at the beginning of the file you need to pay special attention to the line 6 starting with # chkconfig:. Here you define the default startup configuration of the service. With the first keyword following this tag you specify whether the service is enabled (on) by default or not (off). With the numbers in the second and third column you influence the startup order of all defined services. The lower the number in the second column is the earlier the service starts at system boot, and the lower the number in the third column is the earlier it is stopped at system shutdown.
If you have multiple services that depend on each other then you can make sure that they are started and stopped in the required order by setting these values accordingly. For the proftpd service we do not have any special dependencies and requirements, so I just stuck with the values of ntpd.

You can later manually disable and re-enable a service by using the chkconfig command line tool in an ESXi shell. The tool will save the current configuration to the text file /etc/chkconfig.db. Of course it is more comfortable to make these changes in the vSphere Client's GUI, but before we can do this we need to create another proftpd.xml file ...

Service definition

... and store it in the directory /etc/vmware/service:

<ConfigRoot>
   <service id="0000">
      <id>proftpd</id>
      <required>false</required>
      <ruleset>ProFTPD</ruleset>
      <packagename>ProFTPD</packagename>
      <packagedescription>This VIB includes the ProFTPD FTP Daemon.</packagedescription>
   </service>
</ConfigRoot>

Let's explain the meaning of the tags:

• id is the name of the service as it will be shown in the vSphere Client GUI.
• id is also the name of the service's init-script, unless you explicitly override it by specifying a script  name via the initfilename tag (not used here).
• If you set required to true then you won't be able to disable this service via the vSphere Client or any other means (Actually this is not used by any of the built-in services, and I have not yet tested if this really works).
• ruleset refers to the firewall rule set of the service and needs to be set to the exact same string that you used for the id tag in the firewall XML file (see above).
• packagename and packagedescription are the name and description of the installed VIB package that includes the service (and that we will build later). I don't think that these tags have a technical relevance, but they are also shown in the vSphere Client GUI.

Again it is important to note that not only the XML tags, but also the enclosed values are all case sensitive. You need to take careful note of that e.g. the ruleset name used here is exactly the same string as the id in the firewall rule file!

Next step: Make it all persistent

We now have most of the parts together for our package. So far we have created and tested all files in the ESXi shell of a running system, and you might have noticed: If you reboot the system (e.g. to test if the init-script is working properly) all the files that you stored in /sbin and under /etc will be gone. The reason for this is that the system directories of ESXi all reside only on a volatile RAM disk.

So while you are still creating and testing the files you should always create copies of them on a persistent datastore to not lose your work after a reboot! (Is it too mean that I haven't told you that at the beginning of this post? ;-)

To make your system changes persistent you need to build and install a software package for it, and this (plus some related caveats) will be the subject for the third and last part of the Daemon's VIB series.