Explaining the CX + vSphere5 + VAAI support riddle

I recently blogged about the oddity of EMC CX arrays not being supported with VAAI on vSphere 5.0 anymore and - like many others - I wondered what are the real reasons behind this. Today Chad Sakac (EMC) explained the riddle in a web cast covering a VNX engineering update and - well - exactly this issue.

If you missed it: he just published the presentation that he showed on his blog. Here is a summary:

[Update] ESXi-Customizer Powershell script version 1.2

Following up on some user feedback I updated my ESXi-Customizer-PS (Powershell) script to version 1.2. I introduced this script in the last part of my ImageBuilder Deep Dive series (see also part 1 and 2). This version adds advanced features by providing the following new optional command line parameters:

• -ipname  : provide a name for the customized ImageProfile (the default is derived from the cloned VMware ImageProfile)
• -sip : manually select an ImageProfile from the current list (default = auto-select latest available ImageProfile)
• -hprel <mon>  : select HP packages from release dated <mon> (e.g. jun2012) (default = select latest available HP packages)

Please download the latest version of the script from my Google Code page.

Use the command line parameter -help for complete usage instructions:

ESXi-Customizer-PS v1.2 help screen

How to list vSphere 5.0 HA restarts with PowerCLI

VMware HA usually does a good job restarting VMs in case of an ESX(i) host failure. Imagine this happened at night - you come into office the next morning and want to know what VMs were restarted (or eventually failed to restart) because of the HA event.

You can find that out by looking at the vCenter event log, but if this happened several hours (or even longer) ago the vSphere client will no longer display the associated events. Even if the events are still displayed you will have a hard time to find them and compile a list of restarted VMs.

To cope with this situation I wrote a small PowerCLI script that searches the vCenter event log, finds the relevant entries and prints the list of VMs that were restarted during the latest HA event. It works with vSphere 5.0 only. Here it is:

param(
    [string]$vcenter = "localhost",
    [int]$last = 24,
    [switch]$help = $false
)

$maxevents = 250000

"`nScript to generate list of successful and failed VM restart attempts after an HA host failure."
if ($help) {
"Optional parameters:
-help:               display this help
-vcenter servername: connect to vCenter server servername (default is localhost)
-last n:             analyze events from the last n hours (default is 24)
"
exit
} else {
"(Use -help for list of parameters)"
}

$stop = get-date
$start = $stop - (New-TimeSpan -Hours $last)

if (!(get-pssnapin -name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue )) { add-pssnapin "VMware.VimAutomation.Core" }

write-host "`nConnecting to vCenter server $vcenter ..."
Connect-VIServer $vcenter | out-null

write-host "`nGetting all events from $start to $stop (max. $maxevents) ..."
$events = Get-VIEvent -Start $start -Finish $stop -MaxSamples $maxevents

write-host Got $events.Length events ...

write-host -nonewline "`nSearching for host failure events ..."
$ha = @()
$events | where-object { $_.EventTypeID -eq "com.vmware.vc.HA.DasHostFailedEvent" } | foreach { $ha += $_ }
write-host (" found " + $ha.Length + " event(s).")
if ($ha.Length -eq 0) {
    write-host "`nNo host failure events found in the last $last hours."
    write-host "Use parameter -last to specify number of hours to look back.`n"
    exit
} else {
    write-host ("`nLatest host failure event was " + $ha[0].ObjectName + " at " + $ha[0].CreatedTime + ".")
}

$events = $events | where-object { $_.CreatedTime -ge $ha[0].CreatedTime }

write-host "`nList of successful VM restarts:"
$events | where-object { $_.EventTypeID -eq "com.vmware.vc.ha.VmRestartedByHAEvent" } | foreach {
    write-host $_.CreatedTime: $_.ObjectName
}

write-host "`nList of failed VM restarts:"
$failures = @{}
$events | where-object { $_.FullFormattedMessage -like "vSphere HA stopped trying*" } | foreach {
    $vmname = $_.FullFormattedMessage.Split(" ")[6]
    if (!($failures.ContainsKey($vmname))) {
        $failures.Add($vmname,$_.CreatedTime)
        write-host $_.CreatedTime: $vmname
    }
}

Disconnect-VIServer -Force -Confirm:$false

The script takes three optional parameters:

• -vcenter servername: Connect to the vCenter server named servername (default is localhost)
• -last n: Analyze events of the last n hours (default is 24). If e.g. the HA event was during a weekend, and you run this script on Monday you may need to raise this to as much as 72. Please note: The higher n is the longer the script will run and the more memory it will consume!
• -help: display help on parameters

Usually you may want to specify at least the vCenter server name. If you have only one you can of course hard code it into the script as the default value (instead of localhost) in line 2.

How does it work?

As a first step the script reads all events of the last n (default: 24) hours and searches them for host failure events (event id "com.vmware.vc.HA.DasHostFailedEvent", see line 36). If there were multiple host failures in this time frame it will only look at the latest one and discard all earlier events (line 46).

To find out what VMs were restarted the script looks for events of id "com.vmware.vc.ha.VmRestartedByHAEvent". It will print the time stamps of these events and the names of the VMs that were restarted (line 49 to 51).

At last the script looks for events messages that start with the text "vSphere HA stopped trying". These events are thrown when HA fails to restart a VM multiple times in a row. With vSphere 5 HA will try very hard and repeatedly to restart a VM, so you might see this event multiple times for each failing VM. The script records the failing VMs in a hashed array in order to print their names only once, together with the time stamp of the latest failing restart attempt.

[Update] ESXi-Customizer Powershell script version 1.1

With the last part of my ImageBuilder Deep Dive series (also don't miss part 1 and 2) I published a Powershell script to build a customized ESXi 5.0 installation ISO, optionally adding the contents of the HP Online VIBs depot.

HP has recently updated the contents of their Online depots adding the latest releases of their bundles (see this post). I tested the first version of my script and found that it would incorrectly add older versions of the HP packages. So, I have fixed it now to add only the newest version of each available package. Other changes:

• Skip the hp-smx-limited package. It is not included in HP's customized images and conflicts with the full CIM provider package hp-smx-provider. In fact it is only useful for the new ProLiant Gen8 servers if you want to monitor these through the Agentless Management Service (AMS) only.
• Added a new command line switch -test. It will skip the actual package download and image building - useful for testing the script and seeing what it would add without wasting any download bandwidth.

The new version 1.1 is available for download now.

ESXi-Customizer-PS v1.1 help screen

HP ProLiant Firmware and Driver updates

Today HP released a new Service Pack for ProLiant (SPP): Version 2012.06.0(B). The SPP is a DVD image that includes the latest firmwares and drivers for all ProLiant servers (and associated components like Blade enclosures, iLO boards, NICs, CNAs and Smart Array controllers). Also included is the HP Smart Update Manager (updated to version 5.1.0) that you can use either on a Windows OS or by booting a ProLiant server off the DVD into Linux.

At the same time HP also published updates for their customized ESXi (4.1 U2 and 5.0 U1) installation CDs that include updated Offline bundles and updated drivers (both are also separately available).

Last but not least a new Virtual Connect firmware (version 3.60) hits the street today.

You can find download links to all the mentioned components on my HP & VMware Links page.