Enabling Jumbo Frames on ESXi
Enabling jumbo rames on ESX/ESXi is needed for full throughput on 10Gbit links and the settings are not exposed through the GUI
Reason for Jumbo Frames
The default maximum size for data transmission on Ethernet is 1500 bytes for legacy compatibility reasons, unfortunately for newer high-speed networks breaking up data into 1500 byte chunks called frames creates too much overhead and prevents one from utilizing the full capabilities of the network hardware. For example in our testing on a 10Gbit link we were only able to transmit about 3-4Gbit of data between machines using 1500 byte frames due to the overhead of processing so many small frames of data while we were able to easily saturate 10Gbit when using jumbo frames. Modern full-featured network equipment can transmit larger frame sizes if configured to do so, usually up to 9000 bytes, allowing for full utilization of the available resources. Jumbo frames are usually only used in a controlled and segmented environment, for example an iSCSI SAN, as they can cause additional overhead from the need for packet fragmentation at the point when converting between a 9000 byte frame network and a standard 1500 byte frame network.
Configuring Jumbo Frames on a Cisco interface
Set the maximum transmit unit (MTU) to the largest available value, usually 9216 on both the physical interfaces and VLAN interfaces, if any.
interface TenGigabitEthernet7/1
mtu 9216
interface Vlan1234
mtu 9216
Configuring Jumbo Frames on an ESX VM server
Support for jumbo frames is included in ESX and ESXi but is not exposed in the GUI for the default vSwitch implementation, in addition remote configuration for vSwitches is not enabled on ESXi using the free license from the API using PowerCLI or VMware CLI. Support for jumbo frames is configurable in the GUI when using a Distributed vSwitch which is available in the Enterprise licensing levels. One can configure jumbo frames using the service console command line, on ESXi 3.5 and 4.0 this can be accessed by enabling unsupported tech support mode and on 4.1 by enabling SSH in the Security configuration.
Example for vSwitch
esxcfg-vswitch --list
esxcfg-vswitch --mtu 9000 vSwitch0
Example for VMkernel interface (for vMotion or iSCSI)
esxcfg-vmknic --list
esxcfg-vmknic --mtu 9000 "VMkernel Portgroup Name"
Note: When configuring a VMkernel interface for Jumbo Frames make sure that the underlying vSwitch is also configured for Jumbo Frames.
Configuring Jumbo Frames on ESX VM guest
The default emulated Intel E1000 network device presented to guests in VMware ESX does not support Jumbo Frames, one must install VMware Tools and enable a VMXNET3 interface to take advantage. With Jumbo Frames one should be able to easily saturate 10Gbit using iperf between VM guest hosts running on different VM servers.