[Updated] esxcli plugin to run shell commands (new version)

[Please note: I have updated this blog post to reflect a new version of my esxcli-shell plugin and the changes in there. What's new in version 1.1 (of 2012-01-12):
- supports for advanced shell function like pipelines, I/O redirection
- execute multiple shell commands at once (or a single command spanning multiple lines)
- logging of shell commands to /var/log/shell.log ]

The esxcli command is a very powerful tool to query and configure various aspects of an ESXi host's configuration, and you can not only use it in an ESXi shell, but also remotely through the (perl based) vSphere CLI and the (PowerShell based) vSphere PowerCLI.

The esxcli commands are organized in so-called namespaces (e.g. hardware, software, network etc.) for managing the sub-components of the server, and - out-of-the-box - you are limited to the commands that are exposed through these namespaces. But you cannot run arbitrary ESXi shell commands through it ... until now!

I have released an esxcli plugin (for ESXi 5.x) that allows to run any ESXi shell command through esxcli. It implements the esxcli namespace shell:

esxcli shell help

What is this good for? It allows you to run any ESXi shell command remotely without enabling and using ssh! This means e.g. that you can start e.g. shell scripts (like William Lam's ghettoVCB.sh script) through a remote esxcli or a PowerCLI script.

Installation

The esxcli-shell package is of the AcceptanceLevel CommunitySupported, so you need to change your host's configuration to this level before you can install the package. You can do this via the vSphere Client (see Host Configuration / Security Profile / Host Image Profile Acceptance Level: Edit...) or by running the esxcli command
   esxcli software acceptance set --level CommunitySupported

Download the latest version of the esxcli-shell Offline Bundle from the V-Front Online Depot and upload it to a datastore of your ESXi host. Then install it with the following esxcli command:
   esxcli software vib install -d /vmfs/volumes/datastore1/esxcli-shell-X.X.X-Y-offline_bundle.zip

(X.X.X-Y is the current version number of the plugin. Replace datastore1 with the name of the datastore that you have uploaded the bundle to.)

Alternatively you can install the package directly from the Online Depot (provided that your host has direct Internet access) with the commands
   esxcli network firewall ruleset set -e true -r httpClient
   esxcli software vib install -d http://vibsdepot.v-front.de -n esxcli-shell

In order to make the installation effective you need to either reboot the host or restart the hostd daemon by running
   /etc/init.d/hostd restart

in an ESXi shell. Please note: This will not interrupt any VMs, but will drop any management connections like vSphere client and remote esxcli sessions!

You can also deploy esxcli-shell through Update Manager by importing the Offline Bundle and adding it to a baseline. The hosts' acceptance level still needs to be changed (like described above) though, and you need to manually reboot the hosts or restart hostd on them, because the package does not force a reboot.

Usage with vSphere CLI

Obviously it doesn't make any sense to use esxcli shell commands in a local or remote (ssh) ESXi shell, because there you can run any shell commands directly. However, you can also run esxcli commands remotely from a Windows or Linux machine by installing the vSphere CLI on it. To run an ESXi shell command you then need to use the following syntax
   esxcli -s hostname -u username -p password shell cmd -c "command1" [-c "command2" ...]

hostname is the name of the ESXi host that you want to run the command on. username is the name of a privileged user on this host (normally root), and password is the password of this user. command1, command2 etc. are the shell commands that you want to execute (at least one is required). If a command contains spaces then enclose it in quotes (like "ls -la").

Since version 1.1 you can use any character in a command line that is also available at a regular shell prompt. This includes the pipeline (|) and input (<) and output (>) redirection symbols. For easy output redirection of all specified commands you can also use the options -l (or --logfile=) and -e (or --errlog=) to specify log files for redirecting the standard and error output channels to. If you do not want to overwrite existing log files, but append to them, you need to use the flag -a (or --lappend). Finally you can make esxcli change the working directory before executing the command by specifying it with  -d (or --workdir=). The default working directory is /var/log/vmware. So the complete syntax like shown in the screenshot above is:

   esxcli -s hostname -u username -p password shell cmd -c "command1" [-c "command2" ...] [-l logfile] [-e errorlogfile] [-a] [-d workdir]

Here are some more examples using the vSphere CLI in Windows, from simple to complex:

REM #--- Display the plugin's about info and disclaimer:
esxcli -s vESXi01 -u root -p *** shell about

REM #--- Run a command pipeline 
esxcli -s vESXi01 -u root -p *** shell cmd -c "vmkload_mod -l | grep e1000"

REM #--- Run "lsof" and append output to a log file:
esxcli -s vESXi01 -u root -p *** shell cmd -c "lsof" -l "/vmfs/volumes/LStore1/lsof.log" -a
REM #- or -
esxcli -s vESXi01 -u root -p *** shell cmd -c "lsof >>/vmfs/volumes/LStore1/lsof.log"

REM #--- Run "vmkfstools -y" for space reclamation via UNMAP on datastore DS01:
esxcli -s vESXi01 -u root -p *** shell cmd -c "vmkfstools -y 90" -d "/vmfs/volumes/DS01"
REM #- or -
esxcli -s vESXi01 -u root -p *** shell cmd -c "cd /vmfs/volumes/DS01" -c "vmkfstools -y 90"

REM #--- Run a vendor program that prompts for input twice, answer with "answer1" and "answer2":
esxcli -s vESXi01 -u root -p *** shell cmd -c "echo -e \"answer1\\nanswer2\" | /opt/vendor/bin/program"
REM #- or -
esxcli -s vESXi01 -u root -p *** shell cmd -c "(cat <<END" -c "answer1" -c "answer2" -c "END" -c ") | /opt/vendor/bin/program"

Usage with PowerCLI

Here are the same examples in PowerCLI syntax:

# Connect to ESXi host (will prompt for credentials)
Connect-VIServer vESXi01

# Create the Esxcli object
$esxcli = Get-EsxCli -VMHost vESXi01

#--- Display the plugin's about info and disclaimer:
$esxcli.shell.about()

#--- Run a command pipeline 
$escli.shell.cmd("vmkload_mod -l | grep e1000",$null,$null,$null,$null)

#--- Run "lsof" and append output to a log file:
$esxcli.shell.cmd("lsof",$null,$true,"/vmfs/volumes/LStore1/lsof.log",$null)
#- or -
$esxcli.shell.cmd("lsof >>/vmfs/volumes/LStore1/lsof.log",$null,$null,$null,$null)

#--- Run "vmkfstools -y" for space reclamation via UNMAP on datastore DS01:
$esxcli.shell.cmd("vmkfstools -y 90",$null,$null,$null,"/vmfs/volumes/DS01")
#- or -
$esxcli.shell.cmd(("cd /vmfs/volumes/DS01","vmkfstools -y 90"),$null,$null,$null,$null)

#--- Run a vendor program that prompts for input twice, answer with "answer1" and "answer2":
$esxcli.shell.cmd("echo -e `"answer1\\nanswer2`" | /opt/vendor/bin/program",$null,$null,$null,$null)
#- or -
$esxcli.shell.cmd(("(cat <<END","answer1","answer2","END",") | /opt/vendor/bin/program"),$null,$null,$null,$null)

Passing options to an esxcli command is a bit tricky in PowerCLI: You always need to list all the possible arguments in a specific order - and this is the alphabetical order of the long option names. The --help screen (as shown in the screenshot above) will show the options in exactly the right order:

1. command(s) to run (--command), specify as array if more than one: ("cmd1","cmd2", ...)
2. name of error log file (--errlog)
3. flag for append mode (--lappend): $true, $false or $null for the default (=$false)
4. name of log file (--logfile)
5. name of working directory (--workdir)

If you do not want to use an optional parameter then just specify the value $null for it.

Syslogging / Auditing

For auditing reasons any command that you type in a regular shell is logged to /var/log/shell.log. In version 1.1 of the esxcli shell plugin I added code that will also log all executed commands to this file. Unfortunately all the commands that are run through esxcli will appear as if they have been run by the user root - even if you used another account for the remote esxcli connection -, because they are run through the hostd daemon that runs in the security context of root. I haven't found a way to safely determine the remote user's account.

Defeating command timeouts

If the shell command(s) that you execute through the esxcli shell plugin in PowerCLI take a very long time to complete then the command might fail with an error message "The operation timed out". You then need to raise the PowerCLI timeout for web operations with the command

  Set-PowerCLIConfiguration -WebOperationTimeoutSeconds n

with n being the new timeout value (in seconds). The default is 300 sec. (= 5 min.). You can also disable the timeout by using a negative value for n, but I would not recommend this.

Please note that this feature requires PowerCLI 5.1.

The Making of ...

I created this esxcli plugin after reverse-engineering already existing plugins. It turned out to be relatively easy and only requires some understanding of XML and Linux shell code. I will soon post more details about this ...