Collaboration Brings Persistent Memory to vSphere

Recently, at VMworld* in Las Vegas, I had the pleasure of presenting a talk on persistent memory technology with VMware’s Rich Brunner, CTO of Server Platform Technologies. The focus of the talk was on two things: The upcoming availability of Intel® Optane™ DC persistent memory and the related support in VMware vSphere. Intel and VMware have collaborated on this for quite some time, years in fact. So, Rich and I were very excited to present the latest milestones of that close collaboration.

Rich and I each gave our definitions of persistent memory, which I would summarize with the following qualities:

  • Persistent – holds contents when powered off
  • Accessed as memory, byte-addressable by programs
  • Storage APIs are not required to use it (although are also supported)
  • Performance close enough to DRAM that load/store access makes sense (no paging)

Intel® Optane™ DC Persistent Memory

In August, Intel began shipping the first revenue units of our persistent memory product, known as Intel® Optane™ DC persistent memory (the DC stands for Data Center), to our CSP and OEM partners.

Intel Optane DC Persistent Memory
Fig. 1

One thing that often surprises people hearing about the Intel persistent memory module for the first time is the capacity. As shown in the graphic above (Fig. 1), the modules, which plug into the memory DIMM slots, come in sizes of 128GB, 256GB, and 512GB. That’s up to half a terabyte per module, which can enable two-socket server systems to support 6 TB of memory or more with complementary DRAM installed on the machine. This nicely summarizes why persistent memory is expected to be a disruptive technology, causing people to re-think how they configure their servers and how server applications use the resources available to them.

From Minutes to Seconds

To illustrate one of the potential ways a server application takes advantage of persistent memory, at VMworld* I showed a screen shot from a recent Intel demonstration:

Intel Optane DC Persistent Memory
Fig. 2

Using a popular in-memory database, which keeps its entire database and some computed data structures like indexes in memory, we determined the amount of time to reboot with a large database was 2,100 seconds (35 minutes). Assuming a typical system is rebooted every couple of weeks to install security patches, updates etc., this led to an expected availability of 99.8%, as shown on the right side of the above picture (Fig. 2). By adding Intel® Optane™ DC persistent memory, data structures like the indexes were made persistent, even though they live in memory and are accessed as in-memory data structures. As a result, the time it takes to rebuild those indexes was eliminated and the reboot time was now only 17 seconds for the same very large database. The left side of the picture above (Fig. 2) shows this resulted in an expected availability of 99.999% (five nines). This is just one example, but it shows why server-class application providers are already converting their code to use persistent memory when it is available.

Storage Standards

The key to enabling software vendors to make these changes is the SNIA Persistent Memory Programming Model.

Intel Optane DC Persistent Memory
Fig. 3

SNIA (Storage Networking Industry Association) is an organization focused on storage standards. When it was clear persistent memory was on the horizon, a collection of member companies decided to form a working group to define a common programming model. The goal was to make sure ISVs had a single model for using persistent memory, rather than having to choose among different, incompatible models from different vendors. Key operating systems vendors participated in the workgroup, and that’s why we now see the model available in Windows, Linux, and VMware’s vSphere.

In the graphic above (Fig. 3), the box on the bottom represents persistent memory installed in the system. A set of drivers exposes the persistent memory in multiple ways, including as standard storage (the middle of the picture), where unmodified applications and file systems can use it as if it were simply a very fast SSD.

On the right side of the picture is where all the excitement is: an application uses a new feature in the OS known as DAX (direct access) to map the persistent memory directly into the application’s address space. This provides the application with the ability to use processor load and store instructions directly to the persistence, reading and writing persistent data structures where they live on the media, instead of the traditional method requiring data movement into DRAM before the program can access it.

VMware vSphere Natively Supports Persistent Memory

The graphic above (Fig. 3) really focuses on a simple bare-metal system, where the persistent memory is plugged in and the application gets access. But the exciting part of our VMworld presentation was to point out that, since VMware was closely involved in the programming model definition from the beginning and optimizing its software for it, everything I sketched in the graphic above is also available inside a VMware guest virtual machine!

Intel Optane DC Persistent Memory
Fig. 4

As you can see in the graphic above (Fig. 4), there are multiple ways to use persistent memory with vSphere 6.7. Both transparent usages (where software just accesses storage and the system uses persistent memory to make it very fast) and non-transparent usages (where the software writes directly to the persistent memory programming model) are supported. Rich also showed the slide below (Fig. 5), which illustrates the full Windows* and Linux stacks (working inside the virtual machine) and getting the benefit of trademark vSphere features, like Live VM Migration.

Intel Optane DC Persistent Memory
Fig. 5

The actual talk provided quite a bit more detail, of course, but in this article I wanted to summarize the major points that Rich and I were making. First, that VMware vSphere natively supports persistent memory now and it will be supporting Intel® Optane™ DC persistent memory the first day it is available. This includes features people have come to expect from vSphere, like running multiple workloads on a single physical machine, live migration of VMs across servers with different types of persistent memory, and ways for unmodified legacy workloads to see the benefits that persistent memory has to offer. Second, we described how persistent memory is about to explode in capacity due to Intel® Optane™ technology, and how the ecosystem has come together to provide a unified model for using persistent memory. As a result, application writers are shifting their thinking from the tradition two tiers, where data structures live (memory and storage) to include a new, third tier with persistent memory.

For more on Intel® Optane™ DC persistent memory, read Lisa Spelman’s overview of the technology from the announcement earlier this year and visit the Intel® Optane™ technology section of our website.

Published on Categories Data CenterTags , , ,
Andy Rudoff

About Andy Rudoff

Andy Rudoff is a Senior Principal Engineer at Intel, focusing on Non-Volatile Memory programming. He is a contributor to the SNIA NVM Programming Technical Work Group and author of the Persistent Memory Developer Kit hosted at His more than 30 years industry experience includes design and development work in operating systems, file systems, networking, and fault management at companies large and small, including Sun Microsystems and VMware. Andy has taught various Operating Systems classes over the years and is a co-author of the popular UNIX Network Programming text book.