How to smoothly migrate a VM from one vDS to another

If you are using virtual Distributed Switches (vDS) in your environment you may have stumbled in a situation where you wanted to migrate a VM from one cluster to another using different vDS and couldn't do it: VMotion is not possible in this scenario. Even if the source and target host are CPU-compatible and both have access to the VM's storage, vCenter will deny a live migration whenever the target host is not part of the vDS that the VM is connected to. This is somewhat understandable, because the VM would lose its network port while running, and this is probably not what you intended.

So, let's power off the VM and try a cold migration ... but - surprise(?) - even this will be rejected by vCenter, with the same error message:

vCenter rejecting an inter-vDS migration

One way to cope with that is to remove the vNIC from the VM, do the migration and then add a new vNIC to the VM. However, this has consequences that might be undesirable: The new NIC will have a new MAC address and will be detected as a new device by the guest OS which might led to even more problems, another reboot being necessary etc.

There is another, smoother way to do it: In our environment we have created an extra vDS that serves the only purpose of enabling inter-vDS migrations. We called it TransferSwitch, another good name would be MigrationSwitch, and we have added all hosts of all clusters that are managed by vCenter to this vDS, but without any physical uplinks. A migration of a VM from one vDS to another then becomes a three steps process:

1. Connect the VM's NIC to the TransferSwitch
2. Migrate the VM to the target host
3. Connect the VM's NIC to the destination vDS

This way you can do an inter-vDS migration without the necessity of removing and re-adding the VM's NIC. It would be enough to only temporarily create the TransferSwitch for each migration and add only the source and target host to it. However, we decided to keep it permanently and add all hosts to it. Just to "be always prepared" ... ;-)

About the VMware Tools of ESXi 5.0 and why you should install them on vSphere 4.x

There is a rather new VMware KB article available that describes an interesting problem with the VMware Tools version of ESX(i) 4.1 Update 2: If the clock resolution of a Windows VM has been changed from the default then the VMware Tools service will continually consume 15% CPU performance (in a 1 vCPU VM, for 2 vCPU VMs it will be 7%, etc.).
We have seen this problem on few of our VMs, it looks like there are certain Windows applications around that change the clock resolution thus causing the problem. Detailed background information about the Windows clock resolution (and why it is not a good idea to change it) is available from Microsoft.

The resolution documented in the KB article is to downgrade the VMware Tools to an earlier version or - and this is probably surprising for most of us - to install the VMware Tools version of ESXi 5.0 instead.
This reminds us of the fact that VMware has changed their VMware Tools support policy with the introduction of vSphere 5.0: The VMware Product Interoperability Matrixes now include a selection for the VMware Tools, and it shows that the Tools of ESXi 5.0 are "interoperable" not only with ESXi 5.0 but also with ESX(i) 4.1 and even ESX(i) 4.0:

VMware Tools Interoperability Matrix

 ... whereas earlier versions were only interoperable with the corresponding ESX(i) version.

So, if you are still on vSphere version 4.1 or 4.0 and are planning to upgrade to vSphere 5 sooner or later then you can start deploying the VMware Tools of ESXi 5.0 now, and avoid the effort of future tools upgrades.
You can download the latest version of the ESXi 5.0 tools here at packages.vmware.com.

If you run a manual custom installation of the ESXi 5.0 tools in a Windows VM you will notice that there are some new components included:

VMware Tools 5.0 components default selection

The default selection of components (this is what you get when doing an automatic install or upgrade) is now more suitable for VMware ESXi than it was with earlier versions of the tools, but it still includes two components that are useful for VMware Workstation and completely useless when running ESXi: the Record/Replay Driver and the Audio Driver. Earlier versions of the Tools would also install the Shared Folders component by default, although it is also only useful with VMware Workstation.

A last hint: There is still another "feature" in the VMware Tools package for Windows that I personally find very annoying: Once you have installed the Tools you are by default not able to modify or repair the installation through the "Add or Remove Programs" control panel applet. To fix this find the GUID key for the VMware Tools package in the registry under
   HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall
and change the NoModify and NoRepair values there to 0.

vSphere Hypervisor ESXi 5.0 Driver Rollup 2

Since the release of ESXi 5.0 on August 24th 2011 VMware and 3rd party hardware vendors have updated and added lots of device drivers that can be downloaded in the Drivers & Tools section of VMware's vSphere 5 download center. These drivers are not included in the original ESXi 5.0 installation ISO.

However, at the same place you can download something that is called the vSphere Hypervisor ESXi 5.0 Driver Rollup 2. In fact this is the original installation ISO plus all drivers that were updated or added up to the release date of the rollup (for the Rollup 2 this is Feb 8 2012). So, if you are going to install a new server you should choose this ISO (instead of the GA release ISO) to make use of the latest drivers.
There have been reports that you cannot download the Rollup ISO if you have the free Hypervisor license only, but this is not true (at least not anymore): If you have registered for the Free VMware vSphere Hypervisor you will also be able to download the Rollup ISO.

Please be aware that the Rollup ISO does only contain updated drivers, but no Hypervisor patches, so after deploying a machine with it you still need to apply the latest VMware patches in order to have a system that is fully up to date. There are instructions available on how to do this if you do not have vCenter Update Manager available: See e.g. a post by Chris Colotti.

Update (2012-03-15): There is an issue with the current Rollup 2 ISO that you need to be aware of, especially if you are considering the installation of VMware vShield 5.0. Read more in this blog post.

[Release] ESXi5 Community Packaging Tools v1.0

I just released the first version of my ESXi5 Community Packaging Tools. These are two scripts:

• tgz2vib5.cmd (to convert a TGZ file to a VIB file, formerly included with ESXi-Customizer)
• vib2zip.cmd (to combine one or more VIB files into an Offline Bundle ZIP)

These scripts can be used by Community developers (who provide e.g. drivers for unsupported Whitebox hardware) to package their software into VMware proprietary formats.

Go, read anything you want to know about these tools (and the different formats that they handle), and download them on the project page.

QUADStor delivers Storage Virtualization and VAAI - for free!

Are you a VMware ESXi small business (or even home) user and do you enviously look at Enterprise and high budget customers who can afford fancy high performance SAN boxes with storage virtualization features and VAAI support?
Stop being envious and have a look at QUADstor's storage virtualization software. It is currently in beta and offered for free. And it has all the features that are normally only available with high priced enterprise solutions:

• Thin provisioning
• Transparent compression
• Block level deduplication (inline and post-process)
• Support for NAS, iSCSI and even Fibre Channel (with target capable QLogic adapters)
• VAAI support for VMware ESXi hosts with iSCSI and FC

QUADStor architecture

QUADStor adds a layer on top of the traditional Linux or FreeBSD block device layer and transforms a pool of block devices (which could also be software RAID protected volumes) into thin provisioned so called VDisks that are carved out of this pool. Each VDisk can be deduplicated and/or compressed independently from other VDisks.

VDisks can be used in (at least) three different ways. They can be

• locally formatted with a file system (like Linux ext3) and exported as NAS shares via Samba, NFS etc.
• exported as an iSCSI block device via a modified version of IET (iSCSI Enterprise Target)
• exported as a Fibre Channel block device via a modifed version of the target capable QLogic qla2xxx device driver (so this only works with QLogic FC adapters)

If a VDisk is exported as an iSCSI or FC target that is used by VMware ESX(i) hosts then QUADStor also implements VAAI support. VAAI stands for vSphere APIs for Array Integration. It was introduced with vSphere 4.1 and allows to offload certain disk operations to the storage array. It is based on SCSI open standards and basically has these functionalities (also called primitives): ATS (Atomic Test and Set for hardware assisted locking), XCOPY (Full copy to offload file copy tasks) and Block ZEROing (for initializing disk regions with zeros). The most impressive one is XCOPY, because it allows VM clone or Storage VMotion tasks to be completely offloaded to the storage array. Please note that with QUADStor the XCOPY primitive is only supported for ESXi 5.0.
For more information on VAAI please read the vStorage APIs for Array Integration FAQ article in the VMware KB.

Following is a quick guide for setting up the QUADStor software and configuring it as an iSCSI target for VMware ESX(i) hosts:

1. Prerequisites

The QUADStor software is to be installed on an existing FreeBSD or Linux (RHEL/CentOS 5.0 and 6.0 or SLES11 SP1) box that serves as the storage server. I chose an ESXi 4.1 VM with 2 vCPUs and 4 GB RAM that I set up with SLES for VMware. The following software packages need to be installed with the OS: apache2 (used for the administration web GUI), gcc and kernel-default-devel (the compiler and Linux kernel sources are needed for the compilation of various QUADstor kernel modules). In SLES you can install them with the yast administration and setup tool.

2. Installation

For downloading the QUADStor software you need to register at their web site. The download consists of two rpm packages: A Core package and an ITF package. They need to be installed by running the following commands in a shell:

rpm -ivh quadstor-core-xxx.rpm
rpm -ivh quadstor-itf-xxx.rpm

(xxx is the current build number)

The core package contains the software's back-end including a postgresql database and the web administration GUI. The ITF package contains the iSCSI stack (a modified version of IET) and a modified target-capable version of the qla2xxx driver for QLogic Fibre Channel adapters. They need to be compiled to fit to the current kernel of the server by running the command

/quadstor/bin/builditf

Now you are almost ready to go ... Launch a browser to connect to the web administration GUI (it listens on the standard http port):

QUADStor management web GUI

and add the license key that you received when registering in the menu Management / Licensing:

Adding the QUADStor license key

3. Adding and exporting storage

Attach a new blank disk to your QUADStor VM and add it to the QUADStor pool in the Physical Storage / View Storage menu:

Physical disk view

Click on the "Add" link to add the disk, and in the next menu choose whether you want to use compression with this disk:

Enable Compression for physical disk

After pressing "Submit" you are ready to create you first VDisk from the pool in the Virtual Disks / Add VDisk menu:

Add a QUADStor VDisk

Important note: If you want to access the VDisk with VMware ESX(i) hosts you need to check the "512 byte emulation" option, because ESX(i) is not able to handle other block sizes!
In the Virtual Disks / View/Modify menu you can then list, modify and delete existing VDisks:

Configured VDisks menu

Click on the "Modify" link to configure the VDisk settings:

Modify VDisk properties

Each VDisk is automatically exported via iSCSI. If you click on the "iSCSI" link in the Virtual Disks menu you can configure CHAP authentication for the selected VDisk:

Configure iSCSI CHAP authentication

Finally you can view several statistics counters for a VDisk if you click on the "Statistics" link in the Virtual Disks menu:

VDisk statistics

4. Attaching an iSCSI exported VDisk to an ESX(i) host

I used the ESXi software iSCSI initiator to access the QUADStor target. Its setup and configuration is beyond the scope of this article, but there are sources with detailed explanations available, e.g. the VMware KB article Configuring and troubleshooting basic software iSCSI setup.

After you have attached the iSCSI target to the host you can format it as a VMFS datastore, and it will be listed in the Datastores view of the vSphere client:

View of a VAAI enabled QUADStor datastore

The "Supported" in the Hardware Acceleration column denotes that ESXi has detected and is using the VAAI extensions with the QUADStor datastore.

Now you are ready to create VMs on the QUADStor datastore and experiment with the features it offers. If you run into trouble do not hesitate to contact the QUADStor support. It is very responsive and quick in bug fixing!

Conclusion: The QUADStor storage virtualization software is a work in progress, but definitively a great and affordable opportunity to try out advanced storage virtualization features, even in your home lab. Don't hesitate and get started today!