A peek into Extended Page Tables

Back in 2001-02 when virtualization started to garner interest in the IT world, I wondered about running different Operating systems simultaneously on a server, I remember setting up a small environment in the lab using VMware GSX server and trying to run multiple operating systems side by side to each other. I had to take my focus out of virtualization after that due to change in my job role. I seldom spent much time looking into virtualization technology till I accepted another new role after Six years later.

A lot of development had happened over these years, with virtualization widely accepted among IT techs and management as a instrument to save money on IT expenditure, decrease the TCO, increase ROI and spend money wisely n ever shrinking IT budget. Processor technology moved mutlicore with more than Two cores available on a server and very few software which could take advantage of all those increased number of threads the multicore provided by processors, consolidating physical servers in form of virtual machines on a multicore processor based server helped IT to leverage additional threads and run their datacenters much cooler reducing the number of physical servers. Virtualization also brings in other goodies in terms of redundancy and disaster recovery. Since there are tons of material available on virtualization technology, I will stop here and won’t dwell deeper on virtualization.

As we all know the hypervisor also referred as Virtual Machine Monitor handles all the hardware resource slicing for the virtual machines running on top of it, providing identical execution environment. While VMM takes of time sharing hardware resources and allocating processor, memory and I/O slices to the virtual machines, it introduces significant latency and overhead since it has to translate every request concerning wit CPU/Memory and I/O and pass it on the actual physical device to  complete the request. This has been a Achilles heel for virtualization technology, IT organizations try to keep critical applications running in physical server since they fear the latency and overhead of VMM would bring down the performance of the application if virtualized. Intel introduced hardware assisted support for Virtualization within the processor in year 2005 called as Intel-VT technology. While Intel VT technology alleviated many performance issues associated with Processor virtualization solving part of the problem, memory overhead still remains.

Intel released new Intel Xeon 55XX series processor in March 2009; the new Xeon’s brings a many new technologies. Among the list of new things Xeon 55XX series brought in. One feature called Extended Page Tables also called as EPT. As I discussed earlier Hardware assisted virtualization support with Intel VT alleviated processor overheads, EPT takes care of memory overheads and provides virtual machines to perform much faster than software or VMM translated memory access. I would spend some time to discuss the three modes of memory addressing. A) How a normal process access memory on a physical machine. B) How software assisted memory management in Virtual machines with EPT support. C) How memory management is done when EPT is enabled.

Memory Management in Native Machines

In a native system the task of managing logical memory page numbers to Physical memory page numbers is handled by Operating System. The Operating system accomplishes this by storing the entries in something called as page table structures. When a process of any application access the logical address of the memory where it thinks the data is stored the hardware goes through the table structure to find out the physical address location of where the data is stored. Frequently accessed Logical page number to Physical page numbers of memory address locations are cached by the hardware system in Translation Look aside Buffer also called as TLB.  TLB is a small cache on the processor which accelerates the memory Management process by providing faster LPN to PPN mappings of frequently accessed memory locations.  Picture A shows Memory Management on a native machine.

Memory Management using VMM

When Virtual Machines are run on a hypervisor, the guest operating systems won’t have access to the hardware page tables like the natively run operating systems. The Virtual Machine Monitor emulates the page tables for the Guest operating systems and gives the guest Operating systems an illusion that they are accessing actual physical page numbers when mapping from Logical Page Numbers from the processes running.  The VMM actually runs a page table of it’s own called Shadow Page tables which is visible to the system hardware. So whenever the guest OS makes a request for virtual address translation to physical memory address the request is trapped by the VMM, which In turn run through its shadow page tables and provides the address of physical memory location. Picture B shows Memory Management using VMM.

While the VMM handles the LPN to PPN mapping quite efficiently, there are times when the page fault occurs considerably slowing down the application and the operating systems. The major penalty comes when the guest OS adjusts its logical mapping, this will trigger the VMM to adjust it’s shadow pages to keep in sync with the logical mappings of the Guest OS. For any memory intensive application running inside the guest OS , this process of syncing pages causes a hefty drop in performance due to the overhead of virtualization.

Memory Management using EPT

Hardware assisted memory management using EPT makes life easier for VMM. With EPT the TLB cache assumes an additional role to keep track of virtual memory and physical memory as seen to the guest OS. The individual virtual machines are tracked by the TLB by assigning them with an address space identifier.  Using the address space identifier the TLB can track the virtual machine address space and need not have to flush the TLB cache if one VM switches it space.

The advantage of having EPT manage memory for Virtual machine reduces the need for VMM to keep syncing the shadow pages eliminating the overhead, since the number of times the Shadow pages needs to be synced depends on the number of virtual machines running the server, elimination of sync produces tremendous increase in performance for server with larger number of virtual machines. In addition to this, the benefits of EPT scales with the number of virtual processors assigned to a particular VM, since the rise in processor count also increases the shadow page syncs. Using EPT to eliminate shadow page syncs enables the CPU’s to just sync TLB as the changes occur in the virtual pages, this process is close to achieving management of memory on a natively run operating systems. The only possible downside of managing memory using EPT is that the additional overhead it when there is a TLB miss, typically by many number of TLB stressing applications running on the same physical server. However the Hypervisors take an approach to reduce TLB misses by using large page tables.

Picture C shows Memory Management using EPT.

As a follow-up to my blog, I am setting up a quick lab environment to verify the EPT advantages. I think I will be able to post the results from my quick hands-on experiment in couple of week’s time.

-Bhaskar Gowda.