Do you know where your virtual vCenter server is running?

Although there have been and will always be discussions on this issue ... it is common practice and supported by VMware to run your vCenter server in a Virtual Machine. I would even consider this best practice, because then you will have the high availability and all other advantages that a VM has over a physical server.

However, there are some things to keep in mind with such a setup. You might for example run into a situation where you lose access to vCenter, e.g. because of a network or storage issue, a guest OS crash in the vCenter VM or a host failure that is - for some reason - not accommodated for by HA (although HA does work independently from vCenter!). In such a situation you better know on what host your virtual vCenter runs (or was running), because you need to connect to this host to troubleshoot and resolve the issue and get things up and running again. The same applies to the VM that runs the vCenter database (if this is a separate one).

This is not really a big problem if you have only a handful of hosts or if you run the vCenter VM in a dedicated management cluster that consists of maybe two hosts only. Then it is feasible to connect to these few hosts one after the other until you have found the one that the vCenter VM is registered to. But what if you run vCenter in a cluster of 32 hosts with DRS enabled in fully automated mode?

Here are four different precautionary methods to make the VMware admin's life easier in such a case:

1. Disable automated DRS for the vCenter VM

This is most commonly used: You move the VM to a "known" host in the cluster and then disable DRS for it so that it will never be automatically migrated to another host. You can change the DRS automation level for individual VMs in the cluster's settings (see vSphere DRS / Virtual Machine Options) without changing the cluster's default setting. This fixes the issue in a very pragmatic way, but has some drawbacks:

• You will no longer be able to put the host that runs the vCenter VM into maintenance mode automatically - you will need to manually migrate the vCenter VM away from it (This will break the nice feature of fully automated cluster patching with Update Manager!). And you will have to move it back afterwards to restore the original state.
• If the chosen host crashes for some reason the vCenter VM will get restarted by VMware HA (hopefully) on a randomly chosen host in the cluster. Then you will also have to remember moving it back to the original host (once it's back up and running).

So these scenarios need to be documented, and the necessary additional steps need to be well understood and taken into account by all VMware admins working with the vSphere environment. If you have ever worked in a large global enterprise that has outsourced IT infrastructure activities to offshore companies then you will probably agree that it might be next to impossible to really ensure that.

2. Use a "VM to hosts" DRS rule

Since vSphere 5.0 you can also create and use so-called "VM to hosts" DRS rules. With these you can bind a group of VMs to a group of hosts. So you can define a VM group (see Cluster settings / vSphere DRS / DRS Groups Manager) that includes only the vCenter VM (plus maybe the database VM) and assign it to a host group that only includes a single host. The assignment can be made in two different ways: By using a preferential "should" rule or a mandatory "must" rule.

Avoid the "must" rule here, because this will again prevent the host from entering maintenance mode fully automatically and in addition it might cause issues with HA. But if you choose a "should" rule here then you will be able to put the one chosen host into maintenance mode automatically, and - what's even better - the vCenter VM will be automatically migrated back to the one host once it becomes available again!

So this method is definitely a better choice than option 1. Frank Denneman wrote a detailed blog post about the differences and caveats of "should" vs. "must" rules. Be sure to read it if you are going to choose this setup!

3. Track the vCenter VM through a script

What if you do not want to change any DRS settings (because this limits flexibility and the choices for effective load balancing)? Can you have your vCenter VM freely floating around the cluster and just track where it is? Sure, you can do this by using e.g. a PowerCLI script which runs at a regular interval and records the host's name into a text file on some network share or another place outside of the vCenter server itself. The script can be as simple as this:

connect-viserver vcenter.example.org | out-null
(get-vm VCENTER).Host.Name
disconnect-viserver -Confirm:$false

However, - no matter how short you choose the interval for running the script - you might still miss a vMotion of the vCenter VM occurring right before it gets into trouble ...

4. Track the vCenter VM through a vCenter alarm triggered script

There is at least one more method that you can use to track the location of your vCenter VM: Lately I posted about a way to monitor vSphere replication RPO with vCenter alarms, and this is similar, because we will use a vCenter alarm to monitor vMotion activity.

The first step is to put the vCenter VM (and any other VMs whose migrations you want to track) into a separate VM folder (or Resource Pool) in the vCenter inventory. Then create a new vCenter alarm on this VM folder (You could also define the alarm at the root of the vCenter inventory, but then it would apply to all VMs and introduce too much of an overhead):

• Choose to monitor "Virtual Machines" and select "Monitor for specific events ...":

Track vMotion alarm - General settings

• Define the event triggers shown below to have the alarm status set to "Warning" (= yellow) whenever the VM is "hot migrating" (that means "being vMotioned"), and back to "Normal" (= green) once a migration finished successfully. There will be different events at finish depending on whether the VM was manually migrated ("VM migrated") or automatically through DRS ("DRS VM migrated"):

Track vMotion alarm - Triggers

• At last define an alarm action that is triggered whenever the status is reset to "Normal" (= green) that means whenever a vMotion completes. We choose to run a Windows shell script on the vCenter server here:

Track vMotion alarm - Actions

The script referenced in the alarm definition might look e.g. like this:

echo [%DATE%, %TIME%] Migrated to %VMWARE_ALARM_EVENT_HOST%>>"\\nas01\TrackVM$\%VMWARE_ALARM_EVENT_VM%.log"

It will write out a time-stamped line indicating that the VM was migrated to the host %VMWARE_ALARM_EVENT_HOST% into a log-file that is named after the VM (%VMWARE_ALARM_EVENT_VM%) and stored on a file share named \\nas01\TrackVM$ in this example.

The two VMWARE_ALARM_EVENT* environment variables (and a lot more) are automatically set by the vCenter service when calling the script. If you are using a file share to store the log file (like in this example) then you need to make sure that the user account that runs the vCenter service has proper permissions on this share. Of course you can also think of other methods to process the data: Instead of writing to a remote log file you could e.g. trigger an action on a web server by using wget.exe. Your imagination is the only boundary here ...

As always I hope that you found this post useful. Please comment if you have any questions or remarks to add!

---

---