The SBHVL project - Part 1: Basic setup, Networking and Security

This week I introduced the Small Budget Hosted Virtual Lab (SBHVL) project, and in this post I will describe the basic setup of the physical lab host, the network architecture and some related security aspects.


Installing ESXi on the physical box

As I already mentioned I have rented a server from a hosting company to run the lab on, and - compared to running your own box at home - this does of course require different steps for installing the server and involves different challenges for configuring networking. If you run the box at home then you have physical access to the server, its console and CD-ROM drive, so you just go and install the latest version of ESXi on it using an installation CD that you burned yourself. I recommend that you create the ISO file for that yourself using my ESXi-Customizer-PS script, because that allows you to create an installation ISO that includes the latest patch level of ESXi. This way you just install the server with this ISO and don't need to apply any updates immediately after the installation.

If your server is located at a hosting company then you either need some sort of remote access to the console and a virtual CD-ROM drive that you can connect the installation ISO file to (like an HP iLO board). The hosting company that I chose does offer this, but only temporarily and as a paid service. So, when ordering the server I also booked this service and added the remark that I want to install ESXi ... only to find out that these guys would hand over the server to me after installing ESXi themselves. That was okay for my part, but had one caveat: They installed ESXi 5.0, but not the latest patch level. So, my first action was to update the host to the current patch level, and I followed these steps to do this:

1. Temporarily enable the Remote shell on the host: In the vSphere client go to "Host Configuration / Security Profile / Services Properties" and start the "SSH" daemon there. In the "Firewall properties" at the same place enable the service "httpClient" to allow outgoing http(s) connections.
2. Log on as root to the Remote shell using a Windows SSH client like putty
3. Query the VMware online depot for the latest image profile using this command:
      esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
4. Apply the latest standard image profile from the depot (as of now this is "ESXi-5.0.0-20120704001-standard") by running
      esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.0.0-20120704001-standard
5. This will update ESXi to the latest patch level. Reboot the host to make the changes effective.

Of course you can update any ESXi 5.0 host using this procedure unless it does not have a direct (or transparent) Internet connection, so that a direct https-download would not work. In this case you need

a) to download the patch bundle from the VMware Patch download portal using a web browser on computer that has a working Internet connection,
b) upload the bundle to a datastore of the host using the vSphere Client,
c) then follow the steps above, but replace the https-URL in steps 3 and 4 with the full file path to the zip file.
(Yet another option is to use the --proxy switch with the esxcli commands, if you have a proxy server available)

Configure Networking

After you have installed the host you need to do some basic configuration tasks. Configuring networking including virtual switches is one of them and I will elaborate on that, because you have a plentitude of options here depending on the already existing network infrastructure.

If you want to run the box at home then you probably have some sort of Internet connection through a DSL or cable router. Typically this router is the only device on your home network that has direct Internet connection using a public IP address. All other devices at home use private IP addresses (e.g. from a 192.168.x.x range) that they can get automatically assigned from the router via DHCP, and they also use the router as a gateway for transparently accessing the Internet. Normally the router uses a technique called NAT (Network Address Translation) to provide this access: It replaces the private IP adresses of your home clients with its own public IP address and then forwards the modified packages to the recepient machine on the Internet. The answers that come back are treated conversely and are forwarded to the client that sent the original request (If you want to dive into the NAT topic then I recommend reading Cisco's How NAT Works document).

If this is your current setup at home then you don't need to worry a lot about the network setup and security of your virtual lab. Just use the one vSwitch that was already created during the installation of the ESXi host and keep the VMKernel port and the VMNetwork portgroup on it. If you have more than one physical uplink NIC then connect all of them to this vSwitch, and you are basically done. Your physical host and all VMs running on it will then be able to use the private IP addresses of your home network that can even be assigned via DHCP if your router provides this.

The reason why you do not need to worry about network security in the above scenario is the fact that neither the host nor any of the VMs that it is running is directly exposed to the Internet by default. If your box is running in the datacenter of a hosting provider then you have a different situation: The physical host is directly attached to the Internet, it has its own public IP address, and it is not protected by any external device providing firewall capabilities like a NAT router. In this situation you basically have two choices:

1. Allocate additional public IP addresses (one for each VM that you plan to run) from your provider: This has two important drawbacks: Your provider will charge for additional IP(v4) addresses, and their prices will raise over time, because official IPv4 addresses are getting rarer and rarer (Please note: An alternative is to use IPv6 addresses, but IPv6 is a topic on its own that I still need to dig into. Maybe I'll come back to that later). The second drawback is that each VM that uses its own public IP address is also directly exposed to the Internet and needs to have its own firewall (and other security precautions) to protect it from the inevitable attacks from the outside.

2. The second choice is to set up an additional private network for your VMs to isolate them from the outside world - just like the setup that you have at home. This is the configuration that I chose for my hosted virtual lab, and it looks like this:

SBHVL Networking and Security

We have two virtual switches here: The first virtual switch (vSwitch0) is connected to the Internet through the host's physical uplink and includes a VMKernel port (VMKernel-Public) with the first public IP address and a Virtual Machine port group labeled VMNetwork-Public. The second virtual switch (vSwitch1) does not have any physical uplink and includes a Virtual Machine port group labeled VMNetwork-Private and a second VMKernel port (VMKernel-Private).

A Linux powered all-in-one device

The central component in this picture is a Linux VM which is the only VM that is connected to the VMNetwork-Public port group on vSwitch0 and has its own public IP address. Like any of the other VMs it is also connected to the VMNetwork-Private port group on vSwitch1 and - basically - takes the role of the NAT router in your home office. In the SBHVL setup it has (at least) three different functions:

a) It is used by all other VMs as the gateway to the outside world (the Internet) and uses NAT to masquerade their private IP addresses. By default it does not let pass any unsolicited packages from the outside to the private network on the inside (Later I will explain how to modify this behavior). This function is also called a Masquerading Firewall.
b) It runs a DHCP server for the private network's IP subnet (192.168.33.0/24 in the example). A DHCP server does not only provide IP addresses to its clients (the other VMs), but also other configuration settings like the gateway and DNS server address. For both these settings the Linux box uses its own private IP address (192.168.33.254), because ...
c) ... it is not only the gateway, but also the name server for the other VMs. It is authoritative for the DNS zones inside.sbhvl.org and outside.sbhvl.org (that means it manages these zones itself) and forwards all other requests to the DNS server of my hosting provider. Please note: sbhvl.org is a fictitious domain - in reality I use another one for my setup that I won't disclose here. I will shortly explain the DNS setup in more detail.

There are many different Linux flavors available to set up this machine. If you are already familiar with a specific Linux distribution then choose just that. This is also what I have done: I chose OpenSUSE, because I used it in the past and I like its fool proof administration GUI called YaST (Yet another Setup Tool). YaST allows you to set up all the services that I mentioned here without the need to manually edit any configuration files:

SuSE YaST configuration GUI

OpenSUSE Linux is the free Open Source variant of the commercial SuSE Linux Enterprise Server (SLES). And there is an interesting, but not well known relationship between VMware and SuSE: If you have licensed VMware vSphere then you are entitled to get the VMware branded version of SLES for free. This offer includes access to the installation sources and a free subscription to patches and updates. For details please look at the SLES for VMware product page and/or this SuSE page. If you do not want or can not use an Open Source Linux distribution for your environment then SLES for VMware is a good alternative.

The DNS setup

Running an own DNS server is optional. You could as well use only the DNS servers provided by your hosting company (or your telecom provider if you run the lab at home), but this may limit the DNS functionalities that you are able to use.

For my lab I chose a very specific DNS setup that offers a maximum of freedom and flexibility: A while ago I have registered a DNS domain through a provider (not my hosting provider, but this doesn't really matter) - I will call it sbhvl.org just for explaining my example. My domain provider allows to delegate the authority for sub domains of this domain to your own DNS server. And this is what I have done: I configured the two sub domains inside.sbhvl.org and outside.sbhvl.org and declared my own virtual Linux box to be the authoritative name server for them. I decided to have two sub domains, because I wanted to be able to clearly distinguish between the two Internet facing public IP addresses (of the physical host and the Linux VM) on the one hand, and the internal private IP addresses. That means: The host, the Linux VM and all the other VMs use inside.sbhvl.org as their primary DNS suffix for their internal IP addresses and have their own names registered in there.

So what is the advantage of running a DNS zone like this one on your own DNS server? One advantage is that you can use Dynamic DNS (DDNS) to create new and update existing DNS entries. This is a nice feature e.g. when used in conjunction with a DHCP service like in my setup. When a Windows VM requests an IP address from the DHCP service it provides its computer name in the request. The DHCP service will then not only provide a valid IP address from the internal network, it will also register the computer name with the DNS service so that it is automatically resolved to the correct IP address (and even the associated reverse lookup entry if a reverse lookup zone exists). So there is no need to create these entries manually, and this comes in very handy when running a virtual test lab.

Wrap-up

This concludes my first post on the SBHVL project. In this part I introduced a secure network design for the hosted virtual lab, explained how the firewall, DHCP and DNS service were implemented and how they play together.

In the next part I will explain how I have set up backup and what options for disaster recovery (DR) this implementation offers.