SGI has built a revolutionary system for NVMe storage scaling at SC14!

Intel launched its Intel® Solid-State Drive Data Center Family for PCIe  based on the NVMe specification in June, 2014. But in the world of amazing possibilities with these products, we still want more. Why? A half million IOPS and three gigs per second out of a single card is not enough for Super Compute workloads, right? Not always, and not for every application. Here are a couple of reasons why we need more performance and how that’s possible.  We really want to scale both the performance and the density.

Consistent performance is the answer to the question. Intel SSDs help to deliver consistent performance across different workloads, including mixed ones, which is worst-case scenario for a drive. That’s applicable to a wide range of Data Center products no matter SATA or PCIe. Performance scaling of SATA SSDs is limited by HBA or RAID controller performance, SAS topology and related interface latency. You can scale it linearly in a limited range until the threshold is reached. After that you realize nothing but increased access latency for the RAID configuration. The single Intel PCIe SSD (our P3700) product line can outperform at least 6 SATA SSDs (S3700) on a range of 4K random workloads while maintaining a lower latency than a single SATA SSD. (See the diagram below)


1 Source: Intel. Measurements made on Hanlan Creek (Intel S5520HC) system with two Intelâ Xeon X5560@ 2.93GHz and 12GB (per CPU) Mem running RHEL6.4 O/S, Intel S3700 SATA Gen3 SSDs are connected to LSI* HBA 9211, NVMe* SSD is under development, data collected by FIO* tool

But then how does the performance scale with multiple drives within one system? Given the benefit of latency reduction due to transition to PCIe interface, NVMe protocol and high QoS of the P3x00 product line, it’s hard to predict how far we can take this.

Obviously, we have a limited amount of PCIe lanes per CPU, which depends on the generation and CPU architecture as well as system, thermal and power architecture. Each P3700 SSD takes PCIe Gen3 x4. In order to evaluate the scaling of NVMe SSDs we would like to avoid using PCIe switches and multiplexers. How about a big multi-socket scale-up system based on 32 Xeon E7 CPUs as a test platform? Looks very promising for the investigation of the NVMe scaling.

SGI has presented an interesting All-Flash concept at SC14. It includes a 32 socket system with 64 Intel® Solid-State Drive DC P3700 800GB SSDs, running a single SLES 11 SP3 Linux OS.

That offers a great opportunity to see what happens with the performance scaling inside this massive single image system. Turns out, it’s a true record of 30M IOPS on 4K RR workload! Let’s have a look at the scaling progression here:


The data above was measured at SGI Labs on concept platform based on 32 E7 Xeon CPUs.

This chart represents IOPS and GB/s to the number of SSDs scaling on 4k random read workloads (blue line) and 128K random read workload (red line), from the testing done at SGI’s labs. Each SSD on the add-in-card form factor of PCIe card works independently. It’s like its own controller. We’re not worried about an additional software RAID overhead, and are only interested to see the raw device performance. Dotted lines represent linear approximation while the solid lines connect the dots from experimental tests.

Hard to believe? Come to SGI’s booth 915 and talk to them about it.

Published on Categories Archive
Andrey Kudryavtsev

About Andrey Kudryavtsev

Andrey Kudryavtsev is SSD Solution Architect in the NVM Solution Group at Intel. His main focus is the HPC area, where he helps end-customers and eco system partners to utilize the benefits of modern storage technologies and accelerate the SSD adoption for NVMe. He holds more than 12 years of total server experience, the last 10 years working for Intel. He is the guru of engineering creativity and is an influence in his field. He graduated from Nizhny Novgorod State University in Russia by Computer Science in 2004. Outside of work, he is the owner and coauthor of many experimental technologies in music, musical instruments, and multi-touch surfaces.