Introduction

In the old days data centers were full of physical servers doing nothing or at most running at only 10% capacity, this was a waste of money (power, cooling, support contracts, etc) and space. Companies are always looking to reduce the overall costs and that is were virtualization comes in.

Virtualization has been around since the 1970's, it was not until the late 1990 that virtualization became a hot topic once again, in 1999 a company called VMware released VMware Workstation which was designed to run multiple Operating Systems at the same time on a desktop PC. In 2001 VMware release two servers versions called VMware GSX server (requires a host O/S to run) this was later renamed VMware server and VMware ESX server this had it's own VMKernel (known as the hypervisor) and was run directly on the hardware, also a new filesystem was created called VMware Machine File System (VMFS). Since the first release we had had ESX server 2.0 and ESXi 3.5 (2007) which brings us to the latest version VMware ESXi server 4.1 (2010).

VMware ESXi server has seen many changes over the years and the latest versions includes support for the following, I will be going into much more depth with the below topics.

• High Availability (clustering)
• VMotion
• Fault Tolerance
• Storage VMotion
• Distributed Resource scheduler (DRS)
• Distributed Power Management (DPM)
• Data Recovery

Going back to my first paragraph, virtualizing many servers reduces costs, administration and space and as most companies are now trying to reflect a green image, virtualizing whole environments (including Production environments) seem to be the way things are going.

So what is virtualization, what it is not is emulation or simulation

The latest CPU's from Intel and AMD assist in virtualization having there own technologies, see the appropriate web site for more information

• Intel - Intel VT
• AMD - AMD-V

VMware ESXi Architecture

VMware decide to remove the service console (COS) from its latest kernel (hypervisor), which allows the hypervisor not to have any dependencies on the operating system, which improves reliability and security and the need for many updates (patches). The result is a much more stream-lined O/S (90MB approx) which means that it can be embedded onto a hosts flash drive and thus elimating the need for a local disk drive (more greener environment).

The heart of a ESXi is the VMKernel, this controls the access to the physical hardware, it is similar to other O/S, were processes will be created, file systems are used. The VMkernel is designed for running virtual machines, it focuses on resource scheduling, device drivers, and I/O stacks. You can communicate with the VMkernel via the vSphere API which the vSphere client or vCenter can use.

VMware ESXi server has a number processes that are started, I managed to obtain information on most of them but there a few that I need to investigate further

There are a number of ports that are used by VMware ESXi, here is a list of some of the common ones used

ESXi can be deployed in two formats

ESXi now comes with two powerful command-line utilities vCLI and PowerCLI and also the new Tech Mode Support (TSM) which allows low-level access to the VMkernel so that you can run diagnostic commands.

ESXi Server has the below features, although ESXi is free some features will require a license.

• supports 64-bit only
• supports up to 512GB memory
• supports either 6 or 12 cores per physical processor
• support for fibre channel, fibre channel over Ethernet (FCOE) and iSCSI
• support for booting from a SAN (providing the network cards/HBA support this)
• supports thin provisioning which is designed to provide a higher level of storage utilization
• support dynamic data growth using available data storage , which includes growing Virtual Machine File Systems (VMFS)
• has it own patch manager to allow for easy patching
• fully supports HA
• supports both virtual machine and storage vMotion
• supports fault tolerance (FT)
• supports distributed resource scheduler (DRS)
• can support a mixed setup including any of the above features
• supports USB pass through (single virtual host only)
• support for serial connections over the network to virtual hosts (vSPC)
• supports Load-Based Teaming (LBT)

The VMware vNetwork Distributed Switch (dvSwitch) provides centralized configuration of networking for hosts within your vCenter server data center. This means that you can make changes in the vCenter can can then be applied to a number of ESXi hosts or virtual machines. Network I/O control is a new network traffic management feature for dvSwitches it implements a software scheduler within the dvSwitch to isolate and prioritize traffic types on the links that connect your ESXi server to the physical network. It can recognize the following types of traffic

• Virtual Machine
• Management
• iSCSI
• NFS
• Fault Tolerance logging
• vMotion

Network I/O control uses shares and limits to control traffic leaving the dvSwitch, which can be configured on the resource Allocation tab. Limits are imposed before shares and limits apply over a team of NIC's. Shares on the over hand, schedule and prioritize traffic for each physical NIC in a team

• shares - specify the relative importance of a traffic type being transmitted to the hosts physical NICs, shares settings work the same way a CPU and memory resources .
• limits - are used to specify the maximum limit that can be used by a traffic type

VMware also uses Load-Based Teaming (LBT), which is used to avoid network congestion on a ESXi server, it adjusts manually the mapping of virtual ports to physical NIC's to balance network load leaving and entering the dvSwitch. LBT will attempt to move one or more virtual ports to a less utilized link within the dvSwitch.

The vStorage API for Array Integration (VAAI) is a new API available for storage partners to use as a means of offloading specific storage functions in order to improve performance. It supports the following

Hosts profiles are used to standardize and simplify how you manage your vSphere host configurations, you can capture a policy that contains the configuration of networking, storage, security settings and other features from a properly configured host, you can then use this policy against other hosts to maintain consistency.

Although I am only covering ESXi 4.1 the table below shows the differences between older versions of VMware.

ESXi Boot Process

ESXi can be installed on a flash drive, small local or remote disk drive and as such ESXI differs from other O/S. The system partitions for ESXI are summarized below, this may differ slightly depending if installed on a flash drive or hard disk

• Bootloader Partition - This is a small 4MB partition and contains SYSLinux, which is used as a bootloader to start ESXi
• Boot Bank Partition - This 250MB partition stores the required files to boot ESXI, it is also know as hypervisor 1.
• Alt Boot Bank Partition - This 250MB is initially empty, the first time you patch ESXi, the new system image is stored here. The partition is also known as Hypervisor2.
• Core Dump Partition - This 100MB partition is normally empty but the VMKernel will store a memory dump image if the server crashes. You can manage this partition using the vCLI command vicfg-dumppart.
• Store Partition - this 285MB partition is used to store system utilities such as the ISO images for VMware Tools and floppy disk images for virtual device drives, it is also known as Hypervisor3.

When your ESXi server first starts, SYSLinux is loaded, SYSLinux looks at the file boot.cfg which is located both on hypervisor1 (mounted as /bootbank) and hypervisor2 (mounted as /altbootbank), SYSLinux uses the parameters build, updated and bootstate to determine which partition to use to boot ESXi, if you have not upgraded then it will use /bootbank if you have upgraded then it will use /altbootbank, this then reverses when you update again. If there is a problem after a upgrade you can always boot from the other previous partition. At the initial loading VMware Hypervisor screen you can load a prior version by pressing Shift+R, then press shift+Y to revert back, you should see the message "Fallback hypervisor restored successfully".

After SYSlinux determines which system image to boot, boot.cfg is read to determine the files that are used to boot the VMKernel, once loaded into memory the storage is not accessed again. if you run df -h you get a listing of the filesystems that ESXi has mounted, listed first is visorfs which is the RAM disk that ESXi has created, the four vfat partitions are bootbank, altboot, scratch and store.

The command vdf -h is new and gives details on the RAM disks, the listing below shows the tardisks that ESXi has extracted to create the filesystem, these entries correspond to the Archive and State file types in boot.cfg. The four mounts are MAINSYS is the root folder, hoststats is used to store realtime performance data on the host and updatestg is used as storage space for staging patches and updates.