Volumes connected to ESXi hosts via SAS to Compellent SCv2000 don't survive a reboot

Symptom:

ESXi 6.0 hosts connected to Dell Compellent SCv2000 via SAS experience an issue where by the LUNs do not survive a reboot. A rescan of the HBA's doesn't detect any LUNs.

Removing and re-assigning the LUN mappings for the hosts in Compellent Enterprise Manager works until the hosts are rebooted again.

Turns out that the 12Gb SAS HBA supplied by Dell is manufactured by Avago. The ESXi 6.0 2b image from the VMware website doesn;t have any inbox drivers from Avago so instead uses the LSI-MPT3 SAS driver. Whilst this appears to work from the outset, a reboot of the hosts proves that connectivity to the array isn't re-established after a reboot.

The solution is to remove the LSI-SAS VIB from the boot bank and install the Avago driver which can be downloaded from the VMware website: 

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI60-LSI-SCSI-MPT3SAS-10000000-1OEM&productId=491

After the package has been downloaded you will need to extract the offline bundle file with the ZIP extension and copy it to a VMFS datastore, in my example this is 'vol1'

Use putty etc to SSH onto the host(s). The first command removes the LSI-MSGPT3 VIB, a reboot is required afterwards. The second command installs the Avago VIB

esxcli software vib remove –n lsi-msgpt3
esxcli software vib install -d /vmfs/volumes/vol1/mpt3sas-10.00.00.00-6.0-offline_bundle-2803883.zip

VMware and ESXi with 4k Disks

EqualLogic firmware 7.0.0 now gives us the option to create volumes with 4k sectors rather than the legacy 512e sector size. I thought it would be interesting to create two EqualLogic volumes, one 4k, one 512e. Then create to VMs and IO meter to see if there are any performance gains.

The two volumes could be seen by the ESXi 5.5 host but 

I am surprised that considering 4096 byte sectors have been around since 2010 that VMware has no compatibility for this disk format.

To conclude:

ESXi + 4K disks = :(

Resetting the VMware ESXi trial period

Sometimes I am asked to build a vSphere environment on a trial license from VMware. By default this is 60 days. However after building the environment, updating and testing the trial period will have dropped a number of days. The customer rightly pointed out that this has reduced their trail of the product.

You can use this command at the ESXi shell to reset the trail period back to 60 days.

rm -r /etc/vmware/license.cfg
cp /etc/vmware/.#license.cfg /etc/vmware/license.cfg
/etc/init.d/vpxa restart

Some thoughts on interrupt moderation

Most 1GbE or 10GbE Network Interface Cards (NICs) support a feature called
interrupt moderation or interrupt throttling, which coalesces interrupts from the NIC
to the host so that the host does not spend all its CPU cycles processing interrupts.
However, for latency-sensitive workloads, the time that the NIC delays the delivery
of an interrupt for a received packet or for a packet that has successfully been sent
on the wire, is time adding to the latency of the workload.
Most NICs also provide a mechanism, usually with the ethtool command, to
disable interrupt coalescing. VMware recommends to disable physical NIC interrupt
moderation on the VMware ESXi™ host as follows:

# ethtool -C vmnicX rx-usecs 0 rx-frames 1 rx-usecs-irq 0 rx-framesirq 0

Where vmnicX is the physical NIC as reported by the ESXi command:

# esxcli network nic list

You can verify that your settings have taken effect by issuing the command:

# ethtool -c vmnicX

Note that although disabling interrupt moderation on physical NICs is extremely
helpful in reducing latency for latency-sensitive virtual machines, it can lead to
some performance penalties for other virtual machines on the ESXi host, as well as
higher CPU utilization to deal with the higher rate of interrupts from the physical
NIC.
Disabling physical NIC interrupt moderation can also defeat the benefits of Large
Receive Offloads (LRO), because some physical NICs (such as Intel 10GbE NICs)
that support LRO in hardware automatically disable it when interrupt moderation is
disabled, and the ESXi implementation of software LRO has fewer packets to
coalesce into larger packets on every interrupt. LRO is an important offload for
driving high throughput for large message transfers at reduced CPU cost, so this
trade off should be considered carefully. Additional details are available in the
knowledge base article, Poor TCP performance can occur in Linux virtual machines
with LRO enabled: kb.vmware.com/kb/1027511.

If the ESX host is restarted the above configurations must be reapplied.