Record Performance of All Flash NVMe Configuration – Windows Server 2016 and Storage Spaces Direct

The previous blog in our series on performance of Windows Server* 2016 with Storage Spaces Direct we presented you the benefit of moving to an all flash based solution compared to a Hybrid NVMe SSD + HDD configuration. In this blog, we present the results of all-NVMe flash based configuration based on the testing in the Intel labs where we use NVMe SSDs on both caching and capacity tier and show you how well Storage Spaces Direct is optimized to support all-NVMe based configuration. Let’s start with the test configuration:

The Configuration

Four node cluster using Supermicro* Superserver1028U-TN10RT+ was setup as the test bed. The configuration for each Superserver1028U-TN10RT+ server consisted of:

Processor: 

2x Intel® Xeon® Processor E5-2699 v4 (55M Cache, 2.20 GHz, 22 cores, 145W)

Storage:

Cache Tier: 2x 800G Intel® SSD DC P3700 Series (NVMe)

Capacity Tier: 8x 2TB Intel® SSD DC P3500 Series (NVMe)

Network:

1 x 40 GbE dual-port Chelsio* T580 adapter

1x 40 GbE Extreme Networks* Summit X770-32x switch for cluster networking

Capture

To keep in sync with the other blogs in the series, let’s take a closer look at how the storage capacity is configured.

  • There is 64 TB in the cluster [(16TB/node)*4]
  • Using 3-way mirroring available in Storage Spaces Direct:
    • 21.3 TB of total space (64 TB /3 = 21.3 TB)
    • 5.3 TB per node of available storage (21.3 TB/4 nodes = 5.3 TB)
  • Total shared space of 4*4 (=16 TB) + 2 TB = 18 TB

Each Azure-like VM deployed on the cluster was assigned 2 cores, 3.5 GB RAM and 60 GB capacity. Each was equipped with 60 GB Data VHD (11.17 TB for 44VMs/Node and 6.09 TB for 24 VMs/Node total space used from the shares) containing 4*10 GB for Diskspd files. A total of 24 VMs were deployed in each node for the first test scenario and 44 VMs on each node for the second test scenario.

VM Configuration

Test Scenario 1:

  • 24x Azure-like VMs per node
  • 60 GB OS VHD + 60 GB Data VHD per VM [6.09 TB total space used from the shares]
  • 4*10GB Diskspd files per VM

Test Scenario 2:

  • 44x Azure-like VMs per node
  • 60 GB OS VHD + 60 GB Data VHD per VM [11.17 TB total space used from the shares]
  • 4*10GB Diskspd files per VM

Results

We performed 2 overall tests with Windows Server 2016 RTM build, one with 24 VMs per node to see performance when not all the resources in the cluster are subscribed and another with 44 VMs per node when the cluster resources are in full subscription.

Scenario 1: With 24 VMs per node, for a total of 96 VMs, DISKSPD (version: 2.0.17) was used for IOPS testing. For 4K 100% Random 100% Reads we achieved 3,031,190 aggregate IOPS and average CPU utilization of 92%. For 8K 70/30 Read/Write scenario we measured aggregate IOPS of 930,811 with average CPU utilization of 55%.

4K – 100% Random 100% Reads (24 VMs/Node):

CSVFS Reads/sec

L_17DF.tmp

4K Random Reads 8K 70/30 RW
VMs 96 VMs 96 VMs
Aggregate IOPS 3,031,190 930,811
Avg. CPU Utilization (%) 92 55

Scenario 2: With 44 VMs per node, for a total of 176 VMs, we ran DISKSPD (version: 2.0.17) for 4K 100% Random 100% Reads we achieved 2,675,763 aggregate IOPS and average CPU utilization of 93%. For 8K 70/30 Read/Write scenario we measured aggregate IOPS of 905,660 with average CPU utilization of 58%.

4K – 100% Random 100% Reads (44 VMs/Node):

CSVFS Reads/sec

Capture2

4K Random Reads 8K 70/30 RW
VMs 176 VMs 176 VMs
Aggregate IOPS 2,675,763 905,660
Avg. CPU Utilization (%) 93 58

Conclusion

Windows Server* 2016 with Storage Spaces Direct is very well optimized to support all-NVMe SSD flash based configuration deployments for IOPS and latency sensitive business processing applications that demand the best quality of service (QoS).

We see a consistent balanced performance across the nodes when all the resources are utilized with an aggregate of ~2.67 Million IOPS and an aggregate of ~3.03 Million IOPS when not all the resources are utilized with resources to grow further. The results highlight how well Storage Spaces Direct is optimized to support all NVMe SSD based solutions in Data Centers.  The Storage Spaces Direct cluster scales linearly while maintaining consistent performance, helping Data Centers to grow their storage as needed for IOPS and latency sensitive workloads, illustrating the best cost-performance storage with an all-NVMe flash based configuration for Storage Spaces Direct.

Disclaimers

Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as HammerDB, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Source: Internal Testing.* Other names and brands names may be claimed as the property of others.