[Release] vSphere 5.0 Update 2 and ESXi 5.1 Build 914609

VMware has released Update 2 for vCenter 5.0 (both the Windows version and the Linux appliance) and ESXi 5.0:

2012: A year of blogging and tooling

It's this time of the year again ... time to look back on what you have achieved this year ... the time of annual reviews, and here is mine.

Finally...: HP Online Firmware Updates for ESXi

If you have VMware ESXi running on an HP ProLiant server then you had to take it down for updating the firmware of its hardware components and boot it with the Service Pack for ProLiant Boot CD ... until now!

How to clone ESXi 5

I recently received an e-mail with the following question: How do I copy an ESXi installation from a USB key drive on to the internal hard disk of my computer? Normally you would answer like this: You don't do it. You backup the ESXi configuration (e.g. with the vicfg-backup tool), install a fresh copy of ESXi on your hard disk and then restore the saved ESXi configuration.

A Daemon's VIB - Part 3 (Building a software package for VMware ESXi)

Part 3: Packaging, SecPolicy and VisorFS

This is the third and last part of the "Daemon's VIB" series about building a software package for ESXi, using the example of the ProFTPD Offline Bundle that I recently released. In the first part I explained how to get (or create yourself) suitable binaries for ESXi, in the second part we cared about the Daemon's configuration, firewall settings and automatic startup. In this last part we will
   a) build and install the actual software package
   b) deal with the VMkernel access security policies and
   c) make the service configurable to adapt to your own needs.

A Daemon's VIB - Part 2 (Building a software package for VMware ESXi)

Part 2: Service configuration and startup

This is the second part of the "Daemon's VIB" series about building a software package for ESXi, using the example of the ProFTPD Offline Bundle that I recently released. In the first part I explained how to get (or create yourself) suitable binaries for ESXi. In this part I will focus on what is necessary
   a) to make the service start up automatically on system boot and
   b) to make it fully manageable via the vSphere Client.

A Daemon's VIB: Building a software package for VMware ESXi (Part 1/3)

Part 1: Binaries

I recently released an Offline Bundle for installing the ProFTP Daemon in ESXi 5.x. This was the result of a self-educational project: The subject of this project was not really how to develop software for ESXi, but how to nicely package software for ESXi in a way that makes it easy to distribute and easy to deploy and use for the average VMware administrator. I chose the example of an FTP daemon, because I already stumbled several times over requests for having this available in ESXi, and I know that there were some hacks for it around since the days of ESXi 3.5.

I promised to share what I have learnt in this project, and I will deliver on this promise in three parts. This first part is about binaries and binary compatibility.

[Release] ProFTPD (FTP server) for VMware ESXi 5.x

ProFTPD is a GPL-licensed, feature rich and highly configurable FTP server. Today I have made available an Offline Bundle to install ProFTPD in VMware ESXi 5.x. Installing it will enable you to transfer files from and to an ESXi host using an FTP client program.

[Update] ESXi5 Community Packaging Tools 2.1

I just published updated versions of my ESXi5 Community Packaging Tools. What's new:

• The TGZ2VIB5 tool now allows to build the VIB payload file from a directory structure (rather than using a pre-packaged tgz file as input)
• You can now change the Acceptance Level of the VIB package

Are ESXi 5.x patches cumulative?

I find this question being asked again and again in the VMware forums by people who cannot just use Update Manager for patching, and most times the answers that are given there are a bit vague and unclear or even plain wrong. This is because ESXi patching is a really complex and not well documented matter. So I need to provide some background information before I'm going to answer this question:

HP releases Service Pack 2012.10.0 for ProLiant and updated drivers for ESXi 5.0 und 5.1

HP has released version 2012.10.0 of their Service Pack for ProLiant (SPP). This is a (bootable) ISO file that contains the latest firmware and driver packages for ProLiant servers. What's new:

VMware Converter 5.0.1 released, now supports vSphere 5.1

As announced in my last post VMware released the Converter 5.0.1 today which now supports ESXi 5.1 and vCenter 5.1.

• Download link
• Release Notes

Be aware: Converter 5.0 does not (yet) work with vSphere 5.1

Those of you who have just completed a brand new vSphere 5.1 deployment or upgraded your production environment to 5.1 may run into a real show stopper ... once you try to virtualize a physical server into this environment using VMware Converter. Why? Because the current release of VMware Converter (5.0) does not work together with ESXi 5.1 and vCenter Server 5.1! The 5.0 version of the Converter is a bit old now and has not yet been adapted to work with vSphere 5.1. Maybe VMware just forgot that, or was in such a hurry to release vSphere 5.1 that they intentionally neglected Converter compatibility.

If you desperately need to use Converter with vSphere 5.1 *now* then you have the following work arounds available:

The status of SMP Fault Tolerance

VMware FT (Fault Tolerance) has been around for a while an enables you to protect a VM in a way that allows for a seamless non-disruptive fail over to a copy of it running on another host. The execution of the primary and the secondary copy of an FT-protected VM is synchronized through the vLockstep protocol by sending all input data over a dedicated FT logging link to the host that runs the secondary copy and replaying it there. If the primary host fails for whatever reason the secondary host will detect this and instantly activate its own copy of the VM to become the primary one - with no loss of data or network connections.

New esxcli namespaces (incl. SMART disk monitoring)

With ESXi 5.1 there were several enhancements to the esxcli command that you can use in an ESXi shell, but also remotely, to get and set system configuration parameters.

My VMworld 2012 Europe schedule

I'm looking forward to be at VMworld 2012 (Europe) in Barcelona next week! Below is my planned session schedule. There are some promising topics on it ... I will also try to visit some Lab sessions and meet my fellow vExperts, vendors and other people.

VMware Labs' latest fling: VIB Author and how it compares to the ESXi5 Community Packaging Tools

The VMware Labs have published a new tool called VIB Author that "allows ESXi administrator to create custom VIBs at the CommunitySupported level". When I read this news my first thought was: Hey, that's exactly what my ESXi5 Community Packaging Tools (ESXi5-CPT, consisting of tgz2vib and vib2zip) are supposed to do! So, how do these two compare?

ESXi-Customizer-PS script updated to be compatible with PowerCLI 5.1

I was made aware that my ESXi-Customizer-PS script fails when running with the new PowerCLI version 5.1. With this version VMware made changes to the ImageBuilder object model (dropping the SoftwareChannel object) that broke my script.

The SBHVL project - Part 3: Management, Remote Access and more Security

This is the third and last part of the SBHVL (Small Budget Hosted Virtual Lab) project post series (I will assume that you have already read at least Part 1, see also Part 2). This time I will explain how to remotely access, use and manage the environment.

Updated ESXi-Customizer-PS script allows to create ESXi 5.1 Offline bundle

Following the availability of the ESXi 5.1 packages in the VMware Online depot I have updated my ESXi-Customizer-PS script and made two minor, but important changes:

ESXi 5.1 GA is now available in the VMware Online Depot

The brand new ESXi 5.1 GA packages have finally been uploaded to the VMware Online Depot. That means you can use them with the PowerCLI ImageBuilder snapin and my ESXi-Customizer-PS script to build your own customized installation ISOs or Offline bundles.

To improve the usage of the new ESXi 5.1 ImageProfiles I have updated my ESXi-Customizer-PS script (see my next post).

The ESXi 5.1 GA ImageProfiles

[Updated] How to update your (free / whitebox) ESXi server to ESXi 5.1

Now that vSphere 5.1 (incl. ESXi 5.1) has been released one of the most frequently asked questions of home users is: How can I update my existing ESXi 5.0 installation without too much effort and without taking any risk? Here is a step-by-step procedure:

vSphere 5.1 is generally available (incl. the free and the HP customized version of ESXi)

Late yesterday VMware made vSphere 5.1 generally available. Go to
   http://www.vmware.com/go/download-vsphere
to download. The free version of ESXi 5.1 is also already available:
   http://www.vmware.com/go/get-free-esxi

And HP hardware owners will be happy to find that there is also an HP customized version of the ESXi 5.1 ISO available for download. The shortcut http://www.vmware.com/go/downloads-image-hp-esxi still points to the 5.0 version, but HP's ESXi 5.1 GA is available here:
   https://my.vmware.com/web/vmware/details?downloadGroup=HP-ESXI-5.1.0-GA-10SEP2012&productId=285

Right now it is not yet possible to update an existing ESXi 5.0 installation to 5.1 using the well known methods (via Update Manager and the VMware Online Depot). You will probably be able to update by booting the host with the ESXi 5.1 ISO, but the other methods should also be available shortly.

The SBHVL project - Part 2: Backup and Disaster Recovery

This is part 2 of the SBHVL (Small Budget Hosted Virtual Lab) project post series (see also the Introduction and Part 1). This time it's about backing things up and recovering from a disaster.

Installing and maintaining a test lab can cause significant efforts and will take time. This is true for the initial installation and configuration of the lab infrastructure (our physical host), but also for every single test scenario that usually consists of multiple VMs, software installed on them, configuration work, test data etc. All this is definitely worth being backed up, although it's just a test lab...

HP releases new Service Pack for ProLiant and new drivers for ESXi 5.0 und 5.1

HP has released version 2012.08.0 of their Service Pack for ProLiant (SPP). This is a (bootable) ISO file that contains the latest firmware and driver packages for ProLiant servers. What's new:

Sep2012 release of the HP Customized ESXi 5.0 ISO

HP has just published updated software packages and device drivers for VMware ESXi 5.0 (at http://vibsdepot.hp.com). With this Sep2012 release HP changes the way how they provide this software: Up to now they have made a complete customized installation ISO (including all the packages) available for download on their own web site. Now they have replaced this download link with a shortcut URL that redirects to a VMware download page: http://www.vmware.com/go/downloads-image-hp-esxi.

The SBHVL project - Part 1: Basic setup, Networking and Security

This week I introduced the Small Budget Hosted Virtual Lab (SBHVL) project, and in this post I will describe the basic setup of the physical lab host, the network architecture and some related security aspects.

vSphere 5.1 new features: My top 5 and some less known

This week VMware announced vSphere 5.1 at VMworld in San Francisco. The VMware Blogosphere is already flooded with lists of what's new in this release, so I decided to elaborate on my personal top 5 new features and some less known new features instead of repeating the news over and over. Here we go:

Introducing the Small Budget Hosted Virtual Lab (SBHVL) project

This week there is the VMworld 2012 in San Francisco, and lots of the VMware bloggers have been very busy in the last days preparing for this event. I'm sure there will again be lots of exiting news, product updates and announcements from VMware and other vendors. I won't be there, but I will closely follow these news.

How do you stay up-to-date on VMware product patches and updates?

A while ago I posted the news about vCenter Server Update 1a and the latest ESXi patch being released, and in the comments section a reader asked some questions about updating VMware ESXi specifically and other VMware products in general.

His first question was: How do you know that a vCenter/ESXi/any VMware product patch/update is released?As a VMware admin, how do you stay up-to-date on this matter? My first thought was: Now, this is easy to answer, but in the end I was not able to find a single definitive answer to this question ...

How to simplify and automate VMware ImageBuilder with the ESXi-Customizer-PS script

I published a new and improved version of my ESXi-Customizer-PS script today. This PowerCLI script greatly simplifies and automates the process of creating fully patched and customized ESXi 5.x installation ISOs using the VMware PowerCLI ImageBuilder snapin.

Over time I added more and more options to the script, and I think it is time now to provide a detailed tutorial to show how to make efficient use of it. Grab the latest version (1.3) from my Google Code page and give it a try. The only prerequisite you need is an installation of the current VMware vSphere PowerCLI.

VMware vCenter Update 1a and ESXi 5.0 Build 768111

VMware has released Update 1a for vCenter and important patches for ESXi 5.0 on July 12th 2012.

What's new and fixed in vCenter Update 1a:

• This is the first update that is also available for the vCenter (Linux based) appliance
• For the appliance it switches the embedded DB2 database to VMware's own vPostgres database.
• The problem with excessive memory consumption of the tomcat6 service (KB2013890) has been resolved.
• The problem with HA restarts failing for VMs that were migrated with Storage VMotion  (KB2013639) is fixed.
• For a full list of fixes see Release notes.

What's new in the ESXi 5.0 Patch Release ESXi500-201207001:

• Fixes the problem with syslogging stopping when the syslog server becomes unreachable  (KB2003127).
• Fixes the VM autostart feature that was broken in the free version of 5.0 (see my earlier post about this, and the fix announcement of Kyle Gleed)
• For a full list of fixes see KB2019107 and further KB articles that are referenced there.

Here is a link to the full ESXi 5.0 update bundle: ESXi500-201207001.

[Release] ESXi5 Community Packaging Tools 2.0

Yesterday I have released version 2.0 of my ESXi5 Community Packaging Tools. To re-iterate on that: These tools enable Community developers to create software packages for ESXi 5 in the VMware proprietary formats VIB (VMware Installation Bundle) and ZIP (VMware Offline Bundle).

Changes in this version:

Explaining the CX + vSphere5 + VAAI support riddle

I recently blogged about the oddity of EMC CX arrays not being supported with VAAI on vSphere 5.0 anymore and - like many others - I wondered what are the real reasons behind this. Today Chad Sakac (EMC) explained the riddle in a web cast covering a VNX engineering update and - well - exactly this issue.

If you missed it: he just published the presentation that he showed on his blog. Here is a summary:

[Update] ESXi-Customizer Powershell script version 1.2

Following up on some user feedback I updated my ESXi-Customizer-PS (Powershell) script to version 1.2. I introduced this script in the last part of my ImageBuilder Deep Dive series (see also part 1 and 2). This version adds advanced features by providing the following new optional command line parameters:

• -ipname  : provide a name for the customized ImageProfile (the default is derived from the cloned VMware ImageProfile)
• -sip : manually select an ImageProfile from the current list (default = auto-select latest available ImageProfile)
• -hprel <mon>  : select HP packages from release dated <mon> (e.g. jun2012) (default = select latest available HP packages)

Please download the latest version of the script from my Google Code page.

Use the command line parameter -help for complete usage instructions:

ESXi-Customizer-PS v1.2 help screen

How to list vSphere 5.0 HA restarts with PowerCLI

VMware HA usually does a good job restarting VMs in case of an ESX(i) host failure. Imagine this happened at night - you come into office the next morning and want to know what VMs were restarted (or eventually failed to restart) because of the HA event.

You can find that out by looking at the vCenter event log, but if this happened several hours (or even longer) ago the vSphere client will no longer display the associated events. Even if the events are still displayed you will have a hard time to find them and compile a list of restarted VMs.

To cope with this situation I wrote a small PowerCLI script that searches the vCenter event log, finds the relevant entries and prints the list of VMs that were restarted during the latest HA event. It works with vSphere 5.0 only. Here it is:

param(
    [string]$vcenter = "localhost",
    [int]$last = 24,
    [switch]$help = $false
)

$maxevents = 250000

"`nScript to generate list of successful and failed VM restart attempts after an HA host failure."
if ($help) {
"Optional parameters:
-help:               display this help
-vcenter servername: connect to vCenter server servername (default is localhost)
-last n:             analyze events from the last n hours (default is 24)
"
exit
} else {
"(Use -help for list of parameters)"
}

$stop = get-date
$start = $stop - (New-TimeSpan -Hours $last)

if (!(get-pssnapin -name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue )) { add-pssnapin "VMware.VimAutomation.Core" }

write-host "`nConnecting to vCenter server $vcenter ..."
Connect-VIServer $vcenter | out-null

write-host "`nGetting all events from $start to $stop (max. $maxevents) ..."
$events = Get-VIEvent -Start $start -Finish $stop -MaxSamples $maxevents

write-host Got $events.Length events ...

write-host -nonewline "`nSearching for host failure events ..."
$ha = @()
$events | where-object { $_.EventTypeID -eq "com.vmware.vc.HA.DasHostFailedEvent" } | foreach { $ha += $_ }
write-host (" found " + $ha.Length + " event(s).")
if ($ha.Length -eq 0) {
    write-host "`nNo host failure events found in the last $last hours."
    write-host "Use parameter -last to specify number of hours to look back.`n"
    exit
} else {
    write-host ("`nLatest host failure event was " + $ha[0].ObjectName + " at " + $ha[0].CreatedTime + ".")
}

$events = $events | where-object { $_.CreatedTime -ge $ha[0].CreatedTime }

write-host "`nList of successful VM restarts:"
$events | where-object { $_.EventTypeID -eq "com.vmware.vc.ha.VmRestartedByHAEvent" } | foreach {
    write-host $_.CreatedTime: $_.ObjectName
}

write-host "`nList of failed VM restarts:"
$failures = @{}
$events | where-object { $_.FullFormattedMessage -like "vSphere HA stopped trying*" } | foreach {
    $vmname = $_.FullFormattedMessage.Split(" ")[6]
    if (!($failures.ContainsKey($vmname))) {
        $failures.Add($vmname,$_.CreatedTime)
        write-host $_.CreatedTime: $vmname
    }
}

Disconnect-VIServer -Force -Confirm:$false

The script takes three optional parameters:

• -vcenter servername: Connect to the vCenter server named servername (default is localhost)
• -last n: Analyze events of the last n hours (default is 24). If e.g. the HA event was during a weekend, and you run this script on Monday you may need to raise this to as much as 72. Please note: The higher n is the longer the script will run and the more memory it will consume!
• -help: display help on parameters

Usually you may want to specify at least the vCenter server name. If you have only one you can of course hard code it into the script as the default value (instead of localhost) in line 2.

How does it work?

As a first step the script reads all events of the last n (default: 24) hours and searches them for host failure events (event id "com.vmware.vc.HA.DasHostFailedEvent", see line 36). If there were multiple host failures in this time frame it will only look at the latest one and discard all earlier events (line 46).

To find out what VMs were restarted the script looks for events of id "com.vmware.vc.ha.VmRestartedByHAEvent". It will print the time stamps of these events and the names of the VMs that were restarted (line 49 to 51).

At last the script looks for events messages that start with the text "vSphere HA stopped trying". These events are thrown when HA fails to restart a VM multiple times in a row. With vSphere 5 HA will try very hard and repeatedly to restart a VM, so you might see this event multiple times for each failing VM. The script records the failing VMs in a hashed array in order to print their names only once, together with the time stamp of the latest failing restart attempt.

[Update] ESXi-Customizer Powershell script version 1.1

With the last part of my ImageBuilder Deep Dive series (also don't miss part 1 and 2) I published a Powershell script to build a customized ESXi 5.0 installation ISO, optionally adding the contents of the HP Online VIBs depot.

HP has recently updated the contents of their Online depots adding the latest releases of their bundles (see this post). I tested the first version of my script and found that it would incorrectly add older versions of the HP packages. So, I have fixed it now to add only the newest version of each available package. Other changes:

• Skip the hp-smx-limited package. It is not included in HP's customized images and conflicts with the full CIM provider package hp-smx-provider. In fact it is only useful for the new ProLiant Gen8 servers if you want to monitor these through the Agentless Management Service (AMS) only.
• Added a new command line switch -test. It will skip the actual package download and image building - useful for testing the script and seeing what it would add without wasting any download bandwidth.

The new version 1.1 is available for download now.

ESXi-Customizer-PS v1.1 help screen

HP ProLiant Firmware and Driver updates

Today HP released a new Service Pack for ProLiant (SPP): Version 2012.06.0(B). The SPP is a DVD image that includes the latest firmwares and drivers for all ProLiant servers (and associated components like Blade enclosures, iLO boards, NICs, CNAs and Smart Array controllers). Also included is the HP Smart Update Manager (updated to version 5.1.0) that you can use either on a Windows OS or by booting a ProLiant server off the DVD into Linux.

At the same time HP also published updates for their customized ESXi (4.1 U2 and 5.0 U1) installation CDs that include updated Offline bundles and updated drivers (both are also separately available).

Last but not least a new Virtual Connect firmware (version 3.60) hits the street today.

You can find download links to all the mentioned components on my HP & VMware Links page.

ImageBuilder Deep Dive, Part 3: The Power of the Shell

Let's start this third and last part of my ImageBuilder series with a short retrospective summary: In part one we introduced an easily adaptable script for building a customized ESXi 5.0 installation ISO. In the second part we looked closer at the basic cmdlets and learnt some new advanced ones.

In this final post we will walk through an improved and more universal version of the first script. It is more about Powershell in general than about ImageBuilder, so it will definitely help if you are already somewhat familiar with Powershell. But even if this is not the case you can still benefit by just using the improved script as it is. It took me some time to write it, because - according to Powershell - I was an absolute beginner when starting. I learnt a lot while developing it, but it is probably still not perfect ...

You can just download a copy of the script and use it. For the curious here is the listing:

#
# ESXi-Customizer-PS.ps1 - a script to build a customized ESXi installation ISO using ImageBuilder
# Version: 1.0
# Author: Andreas Peetz ([email protected])
#

param(
    [string]$obDir = $(Split-Path $MyInvocation.MyCommand.Path),
    [string]$isoDir = $(Split-Path $MyInvocation.MyCommand.Path),
    [switch]$hp = $false,
    [switch]$help = $false
)

$AccLevel = @{"VMwareCertified" = 1; "VMwareAccepted" = 2; "PartnerSupported" = 3; "CommunitySupported" = 4}

function AddVIB2Profile($vib) {
    if ($AccLevel[$vib.AcceptanceLevel.ToString()] -gt $AccLevel[$MyProfile.AcceptanceLevel.ToString()]) {
        write-host -nonewline (" [New AcceptanceLevel: " + $vib.AcceptanceLevel + "]")
        $MyProfile.AcceptanceLevel = $vib.AcceptanceLevel
    }
    Add-EsxSoftwarePackage -SoftwarePackage $vib -ImageProfile $MyProfile | Out-Null 
    if ($?) { " [OK]" } else { " [FAILED]" }
}

# Write info and help if requested
write-host "`nScript to build a customized ESXi installation ISO using ImageBuilder"
if ($help) {
    write-host "Optional parameters:"
    write-host "   -help         : display this help"
    write-host "   -obDir <dir>  : directory of Offline bundles to add (default = script directory)"
    write-host "   -isoDir <dir> : directory to store the customized ISO (default = script directory)"
    write-host "   -hp           : add packages from the HP VIBs Depot (default = no)`n"
    exit
} else {
    write-host "(Call with -help for instructions)"
}

# Load the ImageBuilder Snapin (if not already loaded)
if (!(Get-PSSnapin -name VMware.ImageBuilder -ErrorAction:SilentlyContinue)) {
    if (!(Add-PSSnapin -PassThru VMware.ImageBuilder)) {
        # Error out if loading fails
        write-host "FATAL ERROR: Cannot load the ImageBuilder Snapin. Is the PowerCLI installed?"
        exit
    }
}

# Add the VMware ESXi base depot
write-host -nonewline "`nConnecting the VMware ESXi base depot ..."
if ($baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml) {
    write-host " [OK]"
} else {
    write-host "`n   FATAL ERROR: Cannot add VMware base Online depot. Please check your internet connectivity and/or proxy settings!"
    exit
} 

# Get the newest ImageProfile from the base depot
$LatestIP = (Get-EsxImageProfile "ESXi-5.0.0-20*-standard" | Sort-Object -Descending Name)[0]
write-host ("Using latest ImageProfile " + $LatestIP.Name)
write-host ("(dated " + $LatestIP.CreationTime + ", AcceptanceLevel: " + $LatestIP.AcceptanceLevel + ",")
write-host ($LatestIP.Description + ")")

# Create your own Imageprofile
$MyProfile = New-EsxImageProfile -CloneProfile $LatestIP -Name ($LatestIP.Name + "-customized") -Description ($LatestIP.Description + " + customizations")

if ($hp) {
    # Add the HP VIBs depot
    write-host -nonewline "`nConnecting the HP Online Depot ..."
    if ($hpDepot = Add-EsxSoftwareDepot http://vibsdepot.hp.com) {
        write-host " [OK]"
    } else {
        write-host "`n   FATAL ERROR: Cannot add HP Online depot. Please check your internet connectivity and/or proxy settings!"
        exit
    }
    $hpDepot.Channels[0] | Get-EsxSoftwarePackage | foreach {
        write-host -nonewline ("   Add VIB " + $_.Name + $_.Version )
        AddVIB2Profile $_
    }
}

# Loop over Offline bundles
write-host "`nAdding Offline bundles from" $obDir ...
foreach ($oBundle in Get-Item $obDir\*.zip) {
    write-host -nonewline "   Adding" $oBundle ...
    if ($ob = Add-EsxSoftwareDepot $oBundle) {
        write-host " [OK]"
        $ob.Channels[0] | Get-EsxSoftwarePackage | foreach {
            write-host -nonewline "      Add VIB" $_.Name $_.Version
            AddVIB2Profile $_
        }
    } else {
        write-host " [FAILED]`n      Probably not a valid Offline bundle, ignoring."
    }
}

# Export the Imageprofile into an installation ISO file
$isoFile = $isoDir + "\" + $MyProfile.Name + ".iso"
write-host -nonewline ("`nExporting to ISO file " + $isoFile + ". Please be patient ...")
Export-EsxImageProfile -ImageProfile $MyProfile -ExportToIso -FilePath $isoFile -Force
if ($?) { " [OK]" } else { " [FAILED]" }

write-host "`nAll done."

Let's look at the script line by line:

Line 7 - 12: The script accepts command line parameters that are defined inside a param() statement. One of the possible parameters is the switch -help. If specified the script will print a help text (explaining the rest of the parameters) and exit (see line 25ff.). Another optional switch is -hp which will add the HP Online VIBs depot (s. line 65ff.). Using -obDir you can specify a directory where the script will look for Offline bundle zip files to be added. And with -isoDir you can specify the directory to store the created ISO file. The default for both directories is the path where the script itself is stored (which is determined by $(Split-Path $MyInvocation.MyCommand.Path)).

Line 14: One of the advanced Powershell features is using hash tables like the one that is created here. It contains the string representations of the four possible acceptance levels that a package or an image profile can have. They are assigned a ranking value (1 to 4) that is used to determine what the most restrictive acceptance level is ("VMwareCertified" = 1) as opposed to the least restrictive ("CommunitySupported" = 4).

Line 16 - 23: In these lines we define a Powershell function named AddVIB2Profile that adds a VIB package ($vib) that is passed as an argument to the custom image profile. As a first step it compares the acceptance level of the software package with that of the image profile (line 17) to determine if the latter needs to be lowered in order to accept the package (line 18 - 19). This is where the above mentioned hash table is used. Then it adds the package using the Add-EsxSoftwarePackage cmdlet. As I want the output of the script to be pretty and not garbled by useless information I pipe the output of the cmdlet into out-null. That means that it is just discarded. In line 22 I use the special Powershell variable $? to test whether the previous command was successful or not. This is a basic and the easiest way to add error handling to a Powershell script, and I will use that a lot here.

Line 25 - 36: This is actually the start of the main script. It prints a title line (line 26), then it will either display a help text and exit the script if -help was specified (line 27 - 33), or it will display only a short line explaining that -help can be used for instructions (line 35) and continue. BTW the special characters `n represent a new line when used in Powershell strings.

Line 39 to 45: Here we will check whether the required snapin VMware.ImageBuilder was already loaded (e.g. because you used the PowerCLI desktop shortcut to start your Powershell session). If not then we try to load it with Add-PSSnapin (line 40). If this fails (e.g. because PowerCLI is not installed on the computer) then the script errors out, because it depends on the ImageBuilder snapin being available and loaded.

Line 47 to 54: Now we add the VMware ESXi Online depot to the ImageBuilder session (also with some error handling). Nothing spectacular, we've seen this before.

Line 57 to 60: In line 57 we grab all standard image profiles from the VMware depot, create a sorted array of them and just pick the first one (with index [0]). This is the newest one, because we sorted the array in descending order. In the following lines we output some information about this image profile that is stored in its attributes.

Line 63: Here we create our custom image profile by cloning the latest one from the VMware depot. Its name and description are derived from the original VMware profile's properties.

Line 65 to 78: If the command line switch -hp was specified then we connect the HP Online VIBs depot to the ImageBuilder session (line 68 to 73) and add all included software packages using the AddVIB2Profile function that we defined at the beginning of the script.

Line 81 to 93: The foreach loop starting at line 82 is another essential part of the script code: Here we loop over all zip files that are located in the obDir directory and try to add them as Offline bundles to the ImageBuilder session (line 84). If it succeeds because a zip file is a valid Offline bundle (and not some other sort of file archive) then all software packages that are included in the bundle will be added to the custom image profile, again using our self-defined AddVIB2Profile function (line 88).

Line 96 to 99: Finally we export the custom image profile into an installation ISO. Its file name is derived from the image profile's name.

How do you run this Powershell script (and others)?

Open a plain Powershell window from the Start menu, or use the PowerCLI desktop shortcut to start an already customized Powershell session.

By default Powershell will only execute scripts that were electronically signed for security reasons. This script (and most other useful scripts that you will find in other places) is not signed, so you will need to relax the security check by using the Set-ExecutionPolicy cmdlet like shown in the following screenshot. You can do this for the current user only by specifying -Scope CurrentUser. This will also work if you do not have administrative permissions on the computer. If you have administrative permissions then you can leave the -Scope parameter and make the setting system global. Setting the execution policy needs to be done only once - it will be saved to the registry and reused for future Powershell sessions.

You can just call the script by its name (adding parameters as required). Since the script is from an "untrusted" source you still need to confirm its execution every time you start it:

ESXi-Customizer-PS help screen

Finally this is an example output from a complete script run:

ESXi-Customizer-PS example run

The script that I provide here is fully functional and flexible enough to meet most requirements. However, it is certainly not perfect. E.g. it could have better error handling. If you have any comments or ideas on how to improve it (or if you run into errors when trying it) then please add a comment to this post!

Update (2012-06-07): In the meantime I updated this script to fix errors. Please watch out for the latest version using this download link!

How to disable Storage I/O Control for an unavailable datastore

If you run your VMs on FC based storage then you occasionally have the necessity to unmap a storage LUN from one or more ESX(i) hosts (e.g. when retiring a storage array or re-organizing your storage layout). It is important to do this in the right way using the procedures that are documented in

• KB1029786 for ESX(i) 4.1
• KB2004605 for ESXi 5.0

The procedure is very complex for vSphere 4.1, but fortunately is has become much easier with vSphere 5.0.

What happens if you fail to follow these procedures and just unpresent the LUN on the storage array, so that the hosts can not access it anymore? The ESX(i) hosts will detect an APD (all paths down) condition in this case, and - particularly ESX(i) 4.1 hosts - can become very unhappy about this (see KB1016626 and KB1030980). Again, ESXi 5.0 hosts are much more resilient to APD conditions: they will eventually turn them into PDL (= permanent device loss, s. KB2004684) conditions and will completely recover from the LUN loss after rescanning the HBAs ... unless you have Storage I/O control (SIOC) enabled on the lost datastores ... and this is what happened to us today :-( The vmkernel.log files were flooded with the following messages, because SIOC was trying to access the lost datastore:

Permanent Device Loss (PDL) with SIOC enabled

Now the problem is: You cannot just disable SIOC on a lost datastore using the vSphere client - you should have done this before unmapping the storage LUN!

One way to recover from this situation is to reboot any of the affected hosts. However, I really wanted to save the time to put all the hosts in maintenance mode and reboot them one after the other. So I looked for a way to forcibly unmount the datastore directly on the host via esxcli or similar. All the ways that are documented in the VMware KB articles and docs did not work in my case, but I finally stumbled over this wonderful blog post by William Lam:

Does SIOC actually require Enterprise Plus and vCenter?

It is a bit old and refers to ESX(i) 4.1, but it is still valid for ESXi 5.0! Here William describes a way to enable (and disable!) SIOC for a datastore directly on a host without using (even without having available!) a vCenter server.

And it is really easy: All you need to know is the device ID of the datastore/LUN. In ESXi 5.0 you can find it out by using the command

   # esxcli storage vmfs extent list

It will list all datastores with their labels and device IDs (starting with naa.). And then you can use the following vsish command to disable SIOC for the device

   # vsish -e set /storage/scsifw/devices/<naa-id>/iormState 1496

That's it! After waiting a few seconds the affected datastore suddenly disappeared from the list of mounted datastores in the vSphere client, and the VMkernel.log error messages also stopped.

Please note: vsish is a powerful but largely undocumented utility to query and set VMkernel parameters. It is only available in an ESXi local or remote shell. William has quite a few posts about it on his virtuallyGhetto blog. This time it saved us the trouble and time of rebooting a whole cluster of hosts ...

What's new with HP ProLiant Gen8 servers

I recently visited an HP roadshow presentation of their new ProLiant server generation Gen8. This has been around for a while, and you might already have seen some announcements, slides etc. Sometimes it can be hard to really learn about what is actually innovative, new and better (compared to older generations) with Gen8, because of all the buzz words and marketing going on about initiatives like "Voyager", "ProActive Insight Architecture" and so on. So I took some notes about "the hard facts" and decided to turn them into this blog post.

According to HP Gen8 is basically about reducing the following three cost factors:

• Manual actions
• Energy costs
• Unplanned downtimes

Before I focus on these three items, let's quickly summarize what hardware changes come with Gen8:

• Support for Intel Sandy Bridge EP (Xeon E5-2600) processors. These new CPUs have
• the I/O hub integrated on the CPU die
• four (formerly three) memory channels per CPU socket
• two (formerly one) Quick Path Interlinks (QPIs) per socket
• up to 8 CPU cores
• Support for PCIe 3.0 (formerly 2.0)
• The new iLO generation 4
• incorporating a new 4GB flash RAM (for storing drivers, firmwares etc.)
• removing the necessity to deploy hardware management agents into the OS (everything is monitored by and through iLO)
• introducing a mobile app (for Android and iOS) for remote management
• New SmartArray controllers (22x and 42x models) with FBWC (= Flash Backed Write Cache) only (no more BBWC = Battery Backed Write Cache)

Reducing manual actions

Together with Gen8 HP introduced a new version (5.0) of their Smart Update Manager (SUM). It was completely rewritten to be more intelligent and shall be able to completely automate the update process of complex hardware setups with multiple dependencies. If you have ever planned and executed firmware updates for HP Blade enclosures then you know that this was a very time consuming and error prone process so far with lots of manual tasks. HP SUM 5.0 now promises to make it a no-brainer and a streamlined, completely automated experience without the requirement of planned downtimes. I am very curious to try this out ...

At the same time HP announced a new support policy for their Service Packs for ProLiant. These are bootable ISOs containing certified sets of drivers and firmwares for ProLiant hardware and the new HP SUM tool. They are the successors of the Firmware maintenance CDs/DVDs that you may already know. HP plans to release a new Service Pack every four months and will support at least three consecutive releases - that means you may update all your firmwares only every 12 months while keeping your environment fully supported by HP. Of course there will be hotfixes between releases to address urgent issues. You can find the relevant links on my HP & VMware links page.

Reducing Energy costs and unplanned downtimes

HP has greatly increased the number of hardware sensors in the new ProLiants, they are distributed all over the chassis on multiple levels forming a "3D sea of sensors". With "HP ActiveHealth" (another great marketing term) there is also a new system for detecting and pro-actively mitigating hardware failures. This intelligence is part of the new iLO4 board that will also store a complete history of more than 1.600 system parameters and measured values. This data can be exported and uploaded to HP SnS for hardware analysis and troubleshooting.

They also claim to have made their power supplies even more efficient: The best efficiency value is still 94%, but it is now kept over a broader range of output powers.

Availability and next steps

At the time there are Gen8 version of the two socket models ML350p, DL160, DL360p and DL380p, BL460c, SL230s and SL250s. Although one of the main drivers for the new platform is the new Intel CPU architecture there will also be Gen8 models of the AMD based ProLiants available later this year.

One last note: If you already have gotten your hands on a new Gen8 ProLiant and want to install VMware vSphere on it then be sure that you use the HP Customized ESXi ISOs. The VMware vanilla ISOs do not yet include the required hardware drivers.

Updated: What's the deal with EMC CX arrays not supporting VAAI with vSphere 5.0?

(If you have read this post before then skip to the bottom for recent updates)

We are in the process of upgrading our production environment to vSphere 5.0 U1. A new vCenter 5.0 server is already in place, and we attached the production ESXi hosts (that are still on 4.1 U2) to it.
The next step would be to upgrade the hosts. In the meantime, since this is a qualified production environment running more than 1.500 virtual servers I became a bit paranoid about hardware and software/firmware compatibility, and decided to double check if the environment is fully supported (or if we should upgrade any firmware first). I was pretty confident that we are on the safe side because we never had any issues with vSphere 4.1 and always kept the environment up to date.

But then I stumbled over this VMware KB article: EMC CX and VNX Firmware and ESX requirements for vStorage APIs for Array Integration (VAAI) support. It states that ESXi 5.0 does not support VAAI on our Clariion CX4 arrays even although they have the latest FLARE code:

	vSphere 4.1	vSphere 5.0
CX4 Series Flare 30/29/28+	VAAI Supported	Not Supported
VNX Series OE 31 or Later *	VAAI Supported	VAAI Supported

I checked the test hosts that we already updated and they showed VAAI being supported on the CX4 LUNs in the vSphere client. However, the KB article recommends to disable VAAI on these hosts ...

I quickly searched the Internet for relevant posts and statements and browsed through the VMware Community forums. There I found this post where people complain about the VAAI status shown as unsupported. It looks like there are dependencies to the LUN size (smaller or larger than 2TB) and to the array's failover mode (sounds like you need to use ALUA / failover mode = 4 which we are already using). But from this post I get the impression that everything is fixed with the latest 30.x FLARE code releases).

We contacted both VMware and EMC to get some clarification now, and I will keep this post updated with any new information. In the meantime I ask everyone using ESXi 5.0 with EMC CX storage to comment on this post whether you are using VAAI, if you have any issues with it, and what FLARE code you have on the arrays. BTW, you can check what VAAI primitives are in use with your storage LUNs by running the following esxcli command:
   esxcli storage core device vaai status get

Thanks in advance for any helpful comments!

---

Update (2012-05-20): I got some feedback from other users, VMware and EMC on this issue, and used my best Google-Fu to get related information about vSphere 5, VAAI, VMFS-5 and/or EMC CX support. Here are the results (somewhat unsorted):

• Several users use this unsupported setup, but do not have any issues with it (so far).
• pape pointed to some useful VMware KB articles in the comments of this post. Here is an overview of these and some more related articles:
• KB2006858: ESXi 5.0 hosts fail to mount VMFS5 volumes that are formatted with ATS-only capabilities (describes an issue with VMFS-5 volumes and ATS locking seen on EMC arrays, but this is solved with ESXi 5.0 Update 1)
• KB1033665: Disabling the VAAI functionality in ESX/ESXi
• KB2012967: Disabling VAAI functionality for a specific array (Now that would be interesting, but the link doesn't work as of now)
• KB1021976: vStorage APIs for Array Integration FAQ (this FAQ also covers differences in ESXi 4.1 and 5.0 resp. VMFS-3 vs. VMFS-5)
• KB2003813: vSphere 5 FAQ: VMFS-5
• The following VMware blog posts of Cormac Hogan are interesting in the context of VAAI usage with VMFS-3 vs. VMFS-5:
• vSphere 5.0 Storage Features Part 1 - VMFS-5
• vSphere 5.0 Storage Features Part 3 - VAAI
• VMFS Locking Uncovered (About the improvements that were made with file locking during VMFS evolution)

Regarding my specific issue (lack of official support for VAAI with EMC CX arrays and ESXi 5.0) I finally got one really detailed and helpful comment from an EMC representative: EMC tested the VAAI functionality of ESXi 5.0 with their EMC CX4 arrays (I specifically asked for the CX4). It has been "tested as functional", but it only passed the certification tests of the ATS and Zero primitives, but failed the certification test of the XCopy primitive.
I wonder why this happened, because VAAI was fully supported with ESXi 4.1 (i.e. it must have also passed the XCopy test), and with ESXi 5.0 there were no changes to the XCopy primitive (at least I could not find any information about such a change).

Anyway, this is probably the reason why EMC and VMware do not officially support VAAI with ESXi 5.0 on CX arrays. From a customer's point of view this is disappointing, but from EMC's point of view this decision is understandable, because they want to focus on their current products and not waste support resources on somewhat outdated arrays like the CX ones. However, according to this EMC representative it may be possible to come to an individiual support agreement with EMC and VMware (EMC calls this RPQ = Request for Product Qualification) if you nevertheless must or want to have an official support statement for whatever reason.

I also had the idea of disabling only the XCopy primitive to be on the safe side, but I don't want to do this globally, because we also use a fully VAAI supported VNX array with our production hosts. So I was glad to find a link to the promising KB article KB2012967 (see list above), but this link currently doesn't work. I will ask VMware about this ...

---

Update (2012-06-20): In the meantime an EMC representative confirmed to me that all three VAAI primitives work fine with vSphere 5.0 and CX arrays, and that they even support it, but you need to file an RPQ with them in order to get formal support.

We have this configuration (VAAI turned on for both our CX4 and VNX arrays) running with vSphere 5.0 in production for about 6 weeks now without any problems.

The link to VMware KB article KB2012967 still doesn't work, but it is now listed to as being "[Archived]" in the reference section of KB1033665 ...

---

Update (2012-07-11): It turned out that you no longer need an RPQ with EMC. They now officially support the vSphere 5's VAAI implementation on the CX4. The combination is listed in their latest Simple Support Matrix for VMware vSphere 5 (Powerlink account needed for download):

Snippet from the EMC Simple Support Matrix for vSphere 5 (of July 2012)

VMware though does still not officially support it (KB2008822 is unchanged as of today). In another blog post I already explained the reason for this.

ImageBuilder Deep Dive, Part 2: A closer look and advanced functions

In the first part of this Image Builder Deep Dive I introduced a script that you can use to build your own customized installation ISO and can easily be adapted to your own needs. In this second part we will take a closer look at the basic cmdlets we used and get to know some more cmdlets with advanced functionality.

To make best use of the following explanations I suggest that you load the complete script that we go through in this part into the PowerShell ISE and run it line by line while reading the post. You can also experiment with the various code samples by pasting them into the command prompt of the ISE.

Using Online depots with proxy servers

At the beginning of the first part's script we added two Online depots to the ImageBuilder session using the Add-EsxSoftwareDepot cmdlet. I stated that this is not possible when using a proxy server, and  - after some testing - I need to correct this statement: I got it working with a proxy server that does not require authentication. With a proxy server requiring authentication it worked on the first try and failed when adding the second online depot (?!). Originally I had a proxy auto-configuration URL entered in the Internet Explorer settings, and this made adding an Online depot consequently fail. Your mileage may vary ...

By default PowerCLI will use the Windows proxy settings (that means what you have configured in Internet Explorer). You can change that by using the Set-PowerCLIConfiguration cmdlet. However, the possibilities are somewhat limited. You can either use the system proxy settings or no proxy at all:

# Load the VimAutomation Snapin
Add-PSSnapin VMware.VimAutomation.Core
#
# Set and get the proxy configuration
Set-PowerCLIConfiguration -ProxyPolicy NoProxy
Set-PowerCLIConfiguration -ProxyPolicy UseSystemProxy
Get-PowerCLIConfiguration
#

Please note: You need to add the VMware.VimAutomation.Core snapin first (if not already added) to use these cmdlets. To display the current setting use the Get-PowerCLIConfiguration cmdlet.

How to "download" the VMware Online base depot

Now you may wonder: What if your computer has no Internet connection (or only via a proxy configuration that causes the above mentioned problems)? Can I somewhat "download" an Online depot and re-use it later on the same or another computer? Yes, you can!

To be exact: You cannot create an offline copy of a complete Online depot, but you can export a specific ImageProfile not only to an installation ISO, but also into a re-usable Offline bundle:

# Load the ImageBuilder Snapin
Add-PSSnapin VMware.ImageBuilder
#
# Reference the VMware ESXi base depot
$baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
#
# List available ImageProfiles sorted by name (filtered on updated standard profiles)
Get-EsxImageProfile "ESXi-5.0.0-2012*-standard" | Sort-Object -Descending Name
#
# The previous command outputs "ESXi-5.0.0-20120504001-standard" as the newest profile (as of May 5th 2012)
# Export this Imageprofile into an Offline bundle zip file
Export-EsxImageProfile -ImageProfile ESXi-5.0.0-20120504001-standard -ExportToBundle -FilePath 'U:\ESXi-5.0.0-20120504001-standard.zip'
#

Now you can re-use this exported zip file any time and on any other computer by adding it with the Add-EsxSoftwareDepot cmdlet. Obviously this Offline bundle zip file only contains the ImageProfile that you exported and only the packages that are part of this ImageProfile.

Please note: The HP Online depot cannot be "downloaded" in this way, because it does not expose any ImageProfiles that you can export. However, in this special case you can also open the http://vibsdepot.hp.com URL in a browser and download the Offline bundles that it includes from the linked web pages.

Looking closer at the ImageProfile object

Apropos ImageProfiles: So far we only looked at the names of these objects. Obviously VMware codes the release date of an ImageProfile (= patch level) into its name using the format YYYYMMDD. Hopefully they will stick with this naming standard - it makes it easy to identify the newest ImageProfile at first sight and programmatically.

The ImageProfile object has more interesting attributes though:

# Examining the ImageProfile's attributes
$ip = Get-EsxImageProfile ESXi-5.0.0-20120504001-standard
$ip | format-list
$ip.VibList
#

By piping the object through the format-list cmdlet you can display all its attributes in a nicely formatted list view:

Name            : ESXi-5.0.0-20120504001-standard
Vendor          : VMware, Inc.
Author          : 
Description     : For more information, see http://kb.vmware.com/kb/2019863.
CreationTime    : 30.04.2012 21:47:06
ModifiedTime    : 30.04.2012 21:47:06
ReadOnly        : False
VibList         : {ata-pata-atiixp 0.4.6-3vmw.500.0.0.469512, net-nx-nic 4.0.557-3vmw.500.1.11.623860, scsi-rste 2.0.2.0088-1vmw.500.1.11.623860, net-e1000 8.0.3.1-2vmw.500.0.7.515841...}
AcceptanceLevel : PartnerSupported
Guid            : 076da48a930e11e1b58d0017a477682c
Rules           : 
StatelessReady  : True

The Description attribute is interesting, because its value includes a URL to a VMware KB article with further information about this ESXi 5.0 patch release.

The VibList attribute is an array of all included software packages. The list is shortened in the formatted object view to retain readability of the output. The third command above ($ip.VibList) will output the complete list of packages with names and versions.

The AcceptanceLevel attribute is something that you normally don't need to take care of, but we will come back to it at the end of this post.

How to compare two ImageProfiles

If you want to quickly check the differences of two ImageProfiles you could list their attributes one after the other like shown before and compare the outputs, especially the VibLists. However, there is a more elegant way to do this - you can use the Compare-EsxImageProfile cmdlet:

# Comparing two ImageProfiles
Compare-EsxImageProfile ESXi-5.0.0-20120504001-standard -ReferenceProfile ESXi-5.0.0-20120404001-standard
#

This command will create and output an ImageProfile comparison object. In this example it looks like this:

Equal               : False
PackagesEqual       : False
RefAcceptanceLevel  : PartnerSupported
CompAcceptanceLevel : PartnerSupported
OnlyInRef           : {}
OnlyInComp          : {}
UpgradeFromRef      : {VMware_bootbank_esx-base_5.0.0-1.13.702118}
DowngradeFromRef    : {}

What do we see here? The two ImageProfiles are not equal (Equal = False), they do not contain the exact same list of packages (PackagesEqual = False), but they have the same AcceptanceLevel (PartnerSupported). The remaining four attributes summarize the differences in the software package list of the two ImageProfiles. Their names are pretty self-explanatory, and in this example the only difference is that the comparison profile (ESXi-5.0.0-20120504001-standard) contains a newer version of the esx-base package than the reference profile (ESXi-5.0.0-20120404001-standard).

Examining, adding and removing single software packages from a depot

In the first part of this series of posts we used a pipeline of commands to add all packages of a depot to our custom profile. Now we are going to take a closer look at a single package using the Get-EsxSoftwarePackage cmdlet:

# Reference downloaded HP offline bundle for be2net driver
$be2net = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2net-4.0.355.1-offline_bundle-487292.zip"
#
# Look at all versions of a single software package (net-be2net)
Get-EsxSoftwarePackage net-be2net
# Look at all properties of a package's specific version
Get-EsxSoftwarePackage net-be2net -Version 4.0.88.0-1vmw.500.0.7.515841 | fl
#

The output of the command in line 34 looks like this:

Name                     Version                        Vendor     Release Date    
----                     -------                        ------     ------------    
net-be2net               4.0.88.0-1vmw.500.0.7.515841   VMware     15.12.2011 00...
net-be2net               4.0.88.0-1vmw.500.0.0.469512   VMware     19.08.2011 01...
net-be2net               4.0.355.1-1OEM.500.0.0.406165  Emulex     16.09.2011 00...

That means that there are three different versions of the net-ne2net package, the first two are in the VMware base depot, and the third one is in the Offline bundle we downloaded from HP.

In line 36 we examine the properties of a specific version of a package. We will see that the SoftwarePackage object has some interesting attributes:

Name            : net-be2net
Version         : 4.0.88.0-1vmw.500.0.7.515841
Vendor          : VMware
Summary         : Updates the ESX 5.0.0 net-be2net
Description     : For build informatin, see KB  http://kb.vmware.com/kb/2007675
ReleaseDate     : 15.12.2011 00:00:00
Depends         : {vmkapi_2_0_0_0, com.vmware.driverAPI-9.2.0.0}
Conflicts       : {}
Replaces        : {}
AcceptanceLevel : VMwareCertified
MaintenanceMode : False
LiveInstallOk   : False
LiveRemoveOk    : False
SourceUrls      : {https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/net-be2net/VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.7.515841.vib}
Guid            : VMware_bootbank_net-be2net_4.0.88.0-1vmw.500.0.7.515841
StatelessReady  : True
Tags            : {driver, module, category:bugfix, severity:general}

Like with the ImageProfile object the Description property contains a link to a VMware KB article with further information on this package. The SourceUrl indicates from where the package will be downloaded. And the package's Tags provides additional classification categories.

The three properties Depends, Conflicts and Replaces describe relationships to other packages (or features that are provided by other packages). Their names are self-explanatory, and ImageBuilder will take care of them for us. You will e.g. not be allowed to add a package that conflicts with another package that is already part of the profile.

Now let's add this single package to a custom profile using the Add-EsxSoftwarePackage cmdlet:

# Create a new ImageProfile
$MyProfile = New-EsxImageProfile -CloneProfile ESXi-5.0.0-20120504001-standard -Name MyProfile
#
# Add an older version of the net-be2net package to the custom profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile "net-be2net 4.0.88.0-1vmw.500.0.7.515841"
# Add the neweset net-be2net package to the custom profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile net-be2net
#

If you just specify the name of the software package (net-be2net in this example), and there are multiple versions of this package available then Add-EsxSoftwarePackage will automatically add the newest version. If you want to add an older version then you can specify it like shown in line 42. Whenever the ImageProfile already includes a package of the same name, but a different version, it will be replaced by the added package, even if it is older! If the ImageProfile already includes the exact same version then the command will fail, or in other words: you cannot add the same version twice.

Finally, you can also remove a package from an ImageProfile using the Remove-EsxSoftwarePackage cmdlet:

# Remove a package from the custom profile
Remove-EsxSoftwarePackage -ImageProfile $MyProfile net-bnx2
#

You won't be able to remove packages that are required by other packages of the ImageProfile. In some case you can override this dependency check by using the parameter -Force, but you really shouldn't do that, because it will most likely result in an invalid ImageProfile.

Adding community developed packages

If you have ever used my ESXi-Customizer script to add a Community developed software package to an ESXi 5.0 installation ISO then you may wonder: Can you also do that with ImageBuilder? Yes, you can, and here is how:

# Add an Offline Bundle with a community developed software package
Add-EsxSoftwareDepot 'U:\$Download\fwenable-ntpd-1.2.0-0-offline_bundle.zip'
# Change the AcceptanceLevel of the custom profile to "CommunitySupported"
$MyProfile.AcceptanceLevel = "CommunitySupported"
# Add the package to the profile
Add-EsxSoftwarePackage -ImageProfile $MyProfile fwenable-ntpd

In this example we add a software package that includes a custom firewall rule to allow incoming NTP queries (this way you can use the ESXi host as NTP server. Read this post for background information). The Offline bundle was created using my ESXi5 Community Packaging Tools by converting a standard tar.gz file to a VIB file first, and then packaging the VIB file into an Offline bundle zip. The project page of the tools contains detailed instructions on how to do this.

There is only one particularity that you need to be aware off when adding Community developed packages this way: Software packages that were created (and certified) by VMware or trusted partners contain an electronic signature that is checked when adding (or installing) the package. This qualifies them for one of the following three so-called AcceptanceLevels: VMwareCertified, VMwareAccepted or PartnerSupported. Of course, Community developed packages do not have trusted electronic signatures, which means that you can only install them if they have the least restrictive (and least trustworthy) AcceptanceLevel called CommunitySupported. The ImageProfile object also has an AcceptanceLevel, and that must be lowered to the same level before you are able to add a Community developed package. This happens in line 52.

This ends the second part of the ImageBuilder Deep Dive series. The next and last part will not deal with the ImageBuilder snapin specifically, but with Powershell coding in general. We will then develop the script from the first part to a more general and versatile solution.

ImageBuilder Deep Dive, Part 1: Building your own customized ESXi ISO

I know that my ESXi-Customizer script has gotten some popularity and a lot of people use it for merging community developed or commercial third-party hardware drivers into the ESXi installation ISO. It has some limitations though - some of you might have stumbled over them already, at least I have described them in the "Known issues" section. Probably the worst is the fact that it cannot handle complex software bundles like ESXi patches (e.g. the ESXi 5.0 Update 1 package).

My recommendation for such cases is to use the VMware supplied and supported method to create your own customized installation ISOs: The ImageBuilder PowerCLI snapin. With this post I'm going to start a series of blog posts that will cover ImageBuilder in detail and will help you to make effective use of it.

This first post will cover the prerequisites and installation of the PowerCLI and will introduce a script that will help you to get the task done. You won't need in-depth PowerShell knowledge for this, but it will definitely help if you are also interested in the remaining parts of the series that will go into the details of the ImageBuilder cmdlets and finally refine the first script to a more universal solution.

Prerequisites and installation

Obviously you need a Windows computer with the current version (2.0) of Powershell installed. Powershell is Microsoft's state-of-the-art scripting language - it is already included with Windows 7 and Windows 2008 R2 server, for earlier versions of Windows it is available as part of the Windows Management Framework. If you are a serious Windows admin you probably have looked at it before. If not then this is a good opportunity to do it. You can start learning e.g. at Microsoft's Script Center, but this is not a prerequisite for now!

The functionality of Powershell can be extended through so-called snapins. VMware makes available such snapins to add functions (so-called cmdlets) that you can use to manage vCenter servers and ESX(i) hosts. The thing is called PowerCLI, and once you have downloaded and installed the current version you are ready to go!

Following is a Powershell script that will create an ESXi 5.0 installation ISO with the current patch level, the HP Offline bundles and some HP drivers. I suggest that you do the following to walk through the explanations below:

• Download a copy of the script to your computer
• In Explorer right-click on the file and choose "Edit" from the context menu. This will open it in the Powershell ISE (Integrated Scripting Environment) editor.
• Within the ISE you can select single (or multiple) lines of the script and execute them separately from the rest:

Run selected lines of code in the Powershell ISE

The first script

# Load the ImageBuilder Snapin
Add-PSSnapin VMware.ImageBuilder

# Reference the VMware ESXi base depot
$baseDepot = Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

# Reference the HP VIBs depot 
$hpDepot = Add-EsxSoftwareDepot http://vibsdepot.hp.com

# List the VIB packages of HP depot
$hpDepot.Channels[0] | Get-EsxSoftwarePackage

# Reference downloaded HP driver offline bundles
$be2iscsi = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2iscsi-4.0.317.0-offline_bundle-469760.zip"
$be2net = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\be2net-4.0.355.1-offline_bundle-487292.zip"
$lpfc820 = Add-EsxSoftwareDepot "U:\HP-ESXi5-Drivers\lpfc820-8.2.2.105.36-offline_bundle-489567.zip"

# List available Imageprofiles sorted by creation date (newest first)
Get-EsxImageProfile | Sort-Object -Descending CreationTime | Format-List Name,CreationTime

# Create your own Imageprofile
$MyProfile = New-EsxImageProfile -CloneProfile ESXi-5.0.0-20120404001-standard -Name MyProfile -Description "ESXi-5.0.0-20120404001-standard + HP components"

# Add all the HP VIB packages to MyProfile
$hpDepot.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
$be2iscsi.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
$be2net.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile
$lpfc820.Channels[0] | Get-EsxSoftwarePackage | Add-EsxSoftwarePackage -ImageProfile $MyProfile

# Export the Imageprofile into an installation ISO file
Export-EsxImageProfile -ImageProfile $MyProfile -ExportToIso -FilePath "U:\MyProfile.iso"

Line 2: This command will add the ImageBuilder snapin to the current Powershell session. Please note: If you start your session with the PowerCLI shortcut on your desktop then the snapin will automatically be loaded and you can skip this line.

Line 5: The Add-ESXSoftwareDepot cmdlet adds an Online depot or a downloaded Offline bundle as a package source to the current ImageBuilder session. The VMware depot referenced here includes the base ESXi 5.0 sources and is needed in any case. Obviously adding an Online depot requires a working connection to the Internet. I could get this to work with a direct connection only, but not through a proxy server. If someone knows how to make this work with a proxy then please comment here!

Line 8: This adds an additional Online depot that was made available by HP and includes their Offline bundles for ESXi 5.0.

Line 11: A software depot object like $hpDepot that was created through the Add-EsxSoftwareDepot cmdlet can be organized in multiple channels (but mostly it's only one channel). By piping the first channel into the Get-EsxSoftwarePackage cmdlet we can list the software packages that are included in this depot. The output looks like this:

Name                     Version                        Vendor     Release Date    
----                     -------                        ------     ------------    
hpnmi                    2.0.11-434156                  hp         29.07.2011 20...
char-hpilo               500.9.0.0.9-1OEM.500.0.0.43... Hewlett... 07.10.2011 14...
hp-ams                   500.9.0.0-55.434156            Hewlett... 09.03.2012 23...
char-hpcru               5.0.0.8-1OEM.500.0.0.434156    Hewlett... 15.07.2011 17...
hpbootcfg                01-00.10                       Hewlett... 15.07.2011 16...
hpacucli                 9.0-24.0                       Hewlett... 02.02.2012 06...
hponcfg                  04-00.10                       Hewlett... 13.11.2011 02...
hp-smx-provider          500.02.10.61.43-434156         Hewlett... 18.11.2011 02...

This step is not really needed to build the new ISO. So you can safely skip it if you already know or are not interested in the contents of a depot.

Lines 14-16: In these lines we add three additional depots, but this time these are not Online depots, but Offline bundles that we downloaded before. You can find the download links on my HP & VMware links page (see section ESXi 5.0 drivers for the HP Emulex 10GbE Converged Network Adapters). But please note: The files that you download from VMware's driver pages are in zip format, but they are not Offline bundles. They include Offline bundles though, so you need to extract the *offline_bundle*.zip files from the downloaded zip-files first. And of course you need to change the file paths used in the script to your own download directory.

Line 19: The base depot that we added in line 5 includes not only software packages, but also so-called image profiles. An image profile is a grouping of software packages out of the depot. In this case each image profile makes up a specific ESXi patch level. The Get-EsxImageProfile cmdlet will list all available image profiles. We pipe it through the Sort-Object cmdlet here to sort the output by creation date. The output will show the newest image profile first and looks like this:

Name         : ESXi-5.0.0-20120404001-no-tools
CreationTime : 16.03.2012 22:59:09

Name         : ESXi-5.0.0-20120404001-standard
CreationTime : 16.03.2012 22:59:09

Name         : ESXi-5.0.0-20120302001-no-tools
CreationTime : 16.03.2012 22:59:08

Name         : ESXi-5.0.0-20120302001-standard
CreationTime : 16.03.2012 22:59:08

Name         : ESXi-5.0.0-20120301001s-standard
CreationTime : 16.03.2012 22:59:08

Name         : ESXi-5.0.0-20120301001s-no-tools
CreationTime : 16.03.2012 22:59:08

Name         : ESXi-5.0.0-20111204001-standard
CreationTime : 31.10.2011 11:24:00

(... output shortened for readability ...)

Each image profile comes in two different flavors: The -standard one includes all packages that make up an ESXi 5 installation, the -no-tools version is the same without the VMware Tools package. We will choose the standard version in our example.

Line 22: The image profiles of the Online depot are read-only. Since we want to modify one of these and add more packages, we need to create a copy first. With the New-EsxImageProfile cmdlet we create a new image profile ($MyProfile) by cloning the newest standard image profile (ESXi-5.0.0-20120404001-standard in this case). We choose the newest one, because it represents the most recent patch level of ESXi! We also assign a new name to the cloned profile (this is mandatory) and a new description (optional).

Lines 25-28: In line 11 we already listed the software packages that are included in the $hpDepot. By piping the output of this command to the Add-EsxSoftwarePackage cmdlet we add all the included packages to our newly created image profile $MyProfile. And we do the same for the three Offline bundles that we added in the lines 14 to 16.

Line 31: In the last line we use the Export-EsxImageProfile cmdlet to "export" our customized image profile into an ISO file. Now guess what: This ISO file is a complete ESXi 5.0 installation media! You can use it to install a machine with ESXi 5.0 with the current patch level and all the HP packages that you added before.

Booting an ImageBuilder customized ESXi 5.0 installation ISO

Wrap-up

In this first part of my ImageBuilder Deep Dive series I introduced a script that you can use to build an up-to-date ESXi 5.0 installation ISO with additional drivers. You learnt some basic terms, the most important cmdlets that are necessary to get the job done, and you will (hopefully) be able to modify the script and adapt it to your own needs.

In the next part I will introduce some more useful ImageBuilder cmdlets, and I will explain how you can integrate community developed software packages to the installation media (a job that you can also do with my ESXi-Customizer script). Stay tuned!