Intel SSD P3700 Series – NVMe Efficiency

Intel SSD P3700 Series - NVMe Efficiency

SSD_PCIe_3700_Addin_Card_SM.pngIntel recently announced our SSD Data Center Family for PCIe products. You can find a link to the main page for the new PCIe drives on here.  Needless to say, things in the SSD world are exciting and one of the best things about these new SSDs is the NVMe protocol (non-volatile memory express). You can learn more about NVMe at For this blog, we’ll focus specifically on the number of CPU cycles it takes to generate an IO and the efficiency of the NVMe protocol. The NVMe website has some great slide decks and the graphic below is from one of Intel’s principal engineer’s presentations on NVMe.

In a nutshell, this graphic (below) outlines the latency in microseconds spent at each stage of an IO request, divided into media (drive latency), controller (HBA), and software (protocol). The great part about NVMe is that it’s extremely thin at about 6 microseconds. In addition, NVMe removes controller latency completely and uses less CPU to drive IO operations. Less CPU cycles per IO opens up a realm of new possibilities.


We decided we should test this out in the lab so we grabbed a dual-socket Intel Xeon E5-2690v2 Intel server, a 12Gb and 6Gb SAS controller, and 64GB of RAM. We paired these SAS controllers and the onboard SATA controller with a couple of 400GB SAS and SATA SSDs, then added a brand new 400GB Intel SSD DC P3700 Series drive to the mix. Using vanilla CentOS 6.5 plus current updates, we then then ran some FIO workloads to evaluate how each drive handled the same workload in identical hardware configurations. FIO is available in most Linux distros and our configuration consisted of eight workers with a queue depth of four, a block size of 4k, and random read pattern across the entire span of the SSDs under test. These tests were run on un-partitioned and unformatted devices specifically to look at the CPU utilization under test without a file system in play. We then prepped the drives by overwriting them four times with a sequential 4k workload and random data. This makes sure that we’ve actually overwritten the SSD a couple of times including the ‘spare’ area or ‘overprovisioning’ you can read more about overprovisioning in my blog on endurance.  In addition to all this preparation, to prevent frequency changes in the CPU we turned off ‘Turbo Boost’, ‘Power Management’, and ‘Hyperthreading’ options in the BIOS of the server.

Here’s the FIO syntax we used:

fio --ioengine=libaio --description=100Read100Random --iodepth=4 --rw=randread --blocksize=4096 --size=100% --runtime=600 --time_based --numjobs=1  --norandommap --name=/dev/nvme0n1 --name=/dev/nvme0n1 --name=/dev/nvme0n1 --name=/dev/nvme0n1 --name=/dev/nvme0n1 --name=/dev/nvme0n1 --name=/dev/nvme0n1 --name=/dev/nvme0n1 2>&1 | tee -a NVMeONpciE.log

Those familiar with IO testing will immediately notice that this workload is not strenuous, and we designed it that way on purpose. We’re not pushing queue depth and only 8 worker threads on a machine with 20 physical cores certainly won’t max out the box. Our objective was to put a relatively mild workload on the server so we could isolate and really look at IO efficiency. Below are our results.

When we looked at relative efficiency (below), using the 12Gb SAS controller and 12Gb SAS SSD as the baseline of 1.0 (largest CPU/IOP), we observed that other SAS and SATA combinations are roughly equivalent but the Intel PCIe/NVMe drive was 2.3 times more efficient than the other connections/protocols. This means the Intel SSD DC P3700 Series we used in our testing scenario generates 2.3 times more IOPS per percent CPU utilization!


During testing we also looked at latency (below). Using the 6Gb SAS controller and 6Gb SAS SSD as the baseline of 1.0 (highest average latency), we observed that other SAS and SATA combinations vary up to 20% lower (better) than the baseline. We also notice that the Intel SSD DC P3700 Series drive was almost half the average latency of the 6Gb SAS SSD at just over 300µs (microseconds!) with our random workload that spanned the entire logical drive!


So why does this matter? For the IT professional, this really opens some doors especially in the caching and IO tiering space. You could easily implement one of the Temp, Tier, or Cache methodologies I talk about in this blog without concern about overburdening your CPU.  With NVMe and PCIe, you get high bandwidth, extremely low IO latencies, and much more efficient CPU utilization per IO. In a nutshell, this means we can keep doing what Enterprise IT does best… do even more, with less and less. So… where could one of these Intel SSDs accelerate an application and benefit your organization?

- Chris

Christian Black is a Datacenter Solutions Architect covering the HPC and Big Data space within Intel’s Non-Volatile Memory Solutions Group. He comes from a 23 year career in Enterprise IT.

Follow Chris on Twitter at @RekhunSSDs.

Read more of Chris's SSD blogs