ESXi embedded vs. ESXi installable FAQ

With the rise of ESXi - starting with version 3.5 in 2007/8 and now being the exclusive type-1 hypervisor architecture of VMware - there has always been some confusion about the meaning of the terms "ESXi embedded" vs. "ESXi installable".

With this blog post I will try to answer the FAQs around this topic and provide some useful background information.

What is not different?

Both ESXi embedded and installable

• are installed with the same installation media/CD and processes ...
• which also means that they share exactly the same code base and functionality ...
• and use the same license codes.

Specifically the free ESXi version (vSphere Hypervisor) supports both installation modes.

What makes a difference?

During the ESXi installation process you will never be asked whether you want to install in embedded or installable mode. It solely depends on the type and size of your target installation media:

• If you install ESXi on a USB key drive or SD card then you will always end up with ESXi embedded.
• If you install ESXi on a hard disk (or iSCSI/SAN/FCoE partition) that has a size of at least 5 GB then you will end up with ESXi installable.
• If the installation target media (no matter what type) is smaller than 5 GB then you will end up with ESXi embedded.

So what is the difference, and why?

The ESXi installer creates several partitions on the target disk that all serve different purposes:

#	Partition Name	Description	Size
1	Boot loader	the syslinux boot loader and EFI boot files	4 MB
2	1st boot bank	ESXi system files	250 MB
3	2nd boot bank	Copy of ESXi system files	250 MB
4	Diagnostic	for storing ESXi core dumps in case of PSODs	110 MB
5	Store	the VMware Tools ISOs	286 MB
6	Scratch(*)	ESXi logs and temporary files	4 GB
7	VMFS(*)	for storing VMs, if the remainder of the disk is at least	1 GB+

(*) exists in ESXi installable only!

The first five of these partitions are created for both ESXi embedded and installable. It is noticable that write access to these partition happens only in exceptional cases:

• at installation time (boot loader, primary boot bank and store)
• when the system is updated (store and secondary boot bank - after the update the two boot banks swap their roles and the old primary one is still available for a quick roll back)
• when the system crashes (then a core dump is written to the diagnostic partition)

However, during normal operations these partitions are only read or not accessed at all. At boot time the ESXi root file system is constructed inside a RAM disk which is populated by unpacking the software packages of the primary boot bank. After that the boot bank is no longer accessed, unless you decide to update ESXi.

Why is this important? Because flash based media like USB key drives and SD cards have only limited write cycles available (compared to hard disks). That means that the system architecture of ESXi embedded is specifically tailored towards these conditions and ensures a long life time of the installation media. More details on the ESXi embedded architecture can be found in the White Paper The architecture of ESXi.

The difference with ESXi installable now is that it adds an on-disk scratch partition for storing ESXi logs and temporary files. ESXi embedded keeps these files in a separate RAM disk, avoiding writing to the system media - at the price of volatility: Once the system reboots or crashes these files are lost - unless you have taken preventative measures ...:

Special considerations for ESXi embedded

For ESXi embedded it is a good practice and a recommendation by VMware to create a persistent scratch location (outside of the installation disk, e.g. on a SAN LUN). KB1033696 explains how to do this for ESXi 4.x and 5.x. This prevents log files from being lost and all sorts of other failure scenarios that can occur just because the scratch RAM disk is too small.

Another (and commonly implemented) way to preserve log files is to send them to a remote syslog server. For instructions see Enabling syslog on ESXi 3.5 and 4.x resp. Configuring syslog on ESXi 5.x.

The third kind: Autodeployed / PXE booted ESXi

Nowadays embedded and installable are not the only ESXi deployment variants. With vSphere 5.0 VMware introduced Autodeploy to use PXE boot with ESXi hosts. In this scenario ESXi is not installed at all: The boot bank files are downloaded from a network boot server and unpacked into the system RAM disk, eliminating the need for any local media.

The recommendations that apply to ESXi embedded also apply to PXE booted hosts, and there are more: An external diagnostic partition should be created as per KB2004299. Alternatively you can have ESXi hosts send diagnostic dumps to the Network dump collector that is part of vCenter 5.x.

Last but not least you can also redirect the store partition, that means the location where the VMware Tools ISO files are located. KB2004018 explains how to do this for PXE booted hosts, but it may be useful to do this for embedded and installable installations, too, as I outlined earlier in this blog post.

Still confused? What do I have after all?

I got the idea for compiling this FAQ post when I recently stumbled over yet another VMware KB article: KB2014558 explains how to find out what type of installation you are actually using. It is as simple as running the command

   esxcfg-info -e

Please comment on this post if you have additional questions or other comments about this topic. I will be happy to add any missing information!