anexinet ISG Blog Site: Understanding NUMA and Virtual NUMA (vNUMA) in vSphere

Working with a recent customer, we had the experience of designing a solution involving a number of very large (average 12-16 vCPU) machines. In order to maxime the performance of these VMs, we needed to fully understand the intricacies of server resource management technologies NUMA and vNUMA. Failing to understand how they worked could have cost the customer performance gains that these technologies offer.

So what are NUMA and vNUMA, exactly? And how does the proper understanding of them benefit an administrator’s virtual environment?

So, what is NUMA?

“NUMA,” which is short for “Non-Uniform Memory Access,” describes a system with more than one system bus. CPU resources and memory resources are grouped together into a “NUMA node.”

The memory in a NUMA node is thus much more easily accessed by an associated CPU. A CPU that needs to access memory in a different node (“Remote Access”) will experience much higher latency, and thus reduced application performance.

NUMA is, in short, an alternative approach to server architecture that links several small, high-performing nodes together inside a single server case.

So, why NUMA?

So long as the memory and CPU being used falls within the bounds of the NUMA node, local communication within a NUMA node allows a CPU much faster access to memory than in an ordinary system layout. This is especially important in the multi-GHz era of today, when CPUs operate significantly faster than the memory they are using. NUMA helps keep CPUs from entering a stalled state, waiting for data, by allowing the fastest possible access to memory.

How do I determine the size of my NUMA nodes?

According to Microsoft, “In most cases you can determine your NUMA node boundaries by dividing the amount of physical RAM by the number of logical processors (cores).” This can be considered a very loose guideline. Further information on determining the specific setup for your NUMA nodes can be found here:

Virtual NUMA

ESX has been NUMA-aware since at least 2002, with VMware ESX Server 1.5 introducing memory management features to improve locality on NUMA hardware. This has worked well for placement of VMs and memory locality for resources being used by that virtual machine, particularly for virtual machines that are smaller than the NUMA node. Large VMs, however, could benefit from extra help when it comes to scheduling their applications.

When enabled, vNUMA exposes a VM operating system to the physical NUMA topology. This allows for performance improvements within the VM by allowing the operating system and applications to take advantage of NUMA optimizations. This allows VMs to benefit from NUMA, even if the VM itself is larger than the physical size of the NUMA nodes.

A few quick points:

· An administrator can adjust, enable, or disable vNUMA on VMs using advanced vNUMA controls.

· If a VM has more than eight vCPUs, vNUMA is automatically enabled.

· If you enable CPU HotAdd, vNUMA is disabled.

See section 14, “Using NUMA Systems with ESXi” in the vSphere Resource Management Guide for more details.

What happens regarding vNUMA during a vMotion between hosts?

A VM’s vNUMA topology will mimic the topology of the host on which it is initally placed; this topology does not adjust if a VM moves to a different host unless the VM is restarted. This is another excellent argument for keeping your hardware consistent within an ESXi cluster, as moving the VM to an ESXi host with a different NUMA topology could result in lost CPU/Memory locality and reduced performance.

For more information, consult the "Guest Operating Systems" section of the VMware Performance Best Practices guide.

In conclusion

Regardless of your virtualization platform, NUMA plays an important part in understanding performance within a virtual environment. VMware, in ESXi versions 5.0 and beyond, has extended the capabilities of large VMs by intelligent NUMA scheduling and improving VM-level optimization with vNUMA. It is important to understand both your NUMA and vNUMA topologies when sizing your virtual machines.