How to prevent HP SIM from scanning the VMotion port

If you run your ESXi hosts on HP hardware and have a look at the vmkernel.log files from time to time then you have probably already stumbled over messages like these:

vmkernel: cpu4:2741)MigrateNet: vm 2741: 1982: Accepted connection from <x.x.x.x>
vmkernel: cpu4:2741)MigrateNet: vm 2741: 2052: dataSocket 0x41002750c600 receive buffer size is 563560
vmkernel: cpu4:2741)WARNING: Migrate: 215: Invalid message type for new connection: 542393671.  Expecting message of type INIT (0).

Instead of x.x.x.x you will see an IP address that might be familiar to you: It belongs to a server that has the HP System Insight Manager (SIM) software installed for inventorying / managing / monitoring hardware devices in your network. This software is commonly used in HP shops, because it does a fairly good job with monitoring the hardware health of your servers and alerting if something is wrong, and this basic functionality is for free.

HP SIM identifies a device on the network by scanning known ports and trying to connect to known services with credentials that you have configured. It can monitor the hardware of an ESXi hosts by querying the HP CIM providers (that are installed with the HP ESXi Offline Bundles, see my HP & VMware links page) via the WBEM protocol. However, for identifying the ESXi host it will also try to connect via http to some known ports - including port 8000 - and that is used by vMotion!

So these vmkernel.log messages are produced by the vMotion code that expects to receive an incoming migration request from another host on port 8000 and complains about the partner not properly initiating the connection. How can we avoid this? A while ago VMware published KB2002969 about this issue - the resolution that you can find here is very easy, but also completely unacceptable: ... remove the reference to the ESXi host on the HP SIM server. What?! This means that you stop monitoring the host with HP SIM!

I looked into the configuration of HP SIM to find a way to disable the http connection attempts, and yes indeed: You can disable http and https in the Global Protocol Settings. However, as the name suggests this is really a global setting, and if you disable it then HP SIM will no longer be able to properly identify and monitor other hardware devices like iLO boards and Onboard Administrators (OA) of Blade enclosures, because this requires the http(s) protocol ...

How about preventing the connection on the ESXi host? Yes, the ESXi firewall comes to our rescue! It can not only block access to a network port completely, but it can also limit the range of allowed connection partners!

Limiting access to the vMotion port

(The picture shows how to configure this in the legacy vSphere client. You can also use the Web Client of vSphere 5.1, esxcli commands or host profiles to configure the firewall.)

Unfortunately you can not just blacklist the IP address of the HP SIM server, you can only define whitelists. However, if you have followed VMware's recommendation to use a dedicated VLAN (and IP subnet range) for the vMotion network then it is easy to allow connections only from this address range.

This way you get rid of the annoying error messages caused by HP SIM, but this method has another important benefit: In the past there were multiple known problem with port scanning software causing services on ESX(i) hosts to fail (see e.g. KB2010626, KB1010672). This can be related to malicious software and DoS attacks, but also "friendly" software like HP SIM or automatic application discovery and mapping products. If you have not already isolated your VMkernel networks using external firewall devices or software then the ESXi firewall is also a good option to protect your hosts against such "attacks".

---

---