The Dawn of “O+Q” for BOTH Lower Cost AND Higher Performance Software Defined Storage – Part I Ceph* Block Storage

Imagine waking up tomorrow and your favorite coffee tasted better and cost less, or finding out that flights to Hawaii are now shorter and first class seats cost the same as coach. That would be amazing! As technologists, that is what you and I do! We develop technologies that increase performance and reduce costs both in the hardware and software spaces.

I am excited about two technologies that, when combined, offer BOTH higher performance AND lower cost Software Defined Storage (SDS) solutions. Internally at Intel, we affectionately refer to this combination of new technologies as “O+Q”. The “O” being Intel® Optane™ DC SSDs and “Q” being Intel® QLC 3D NAND SSDs. Note that we also refer to TLC NAND, such as the Intel® SSD DC P4510 TLC 3D NAND SSDs as “T”. In this blog post, we will highlight the value of “O+Q” when run on 2nd Generation Intel® Xeon™ Scalable processors.

A Brief Overview of “O” and “Q”

Before we discuss the solution, here are short descriptions of what “O” and “Q” are, where they fit within a Ceph* solution, and why they offer such a compelling value proposition. Follow the links to learn more about each technology.

Intel® Optane™ SSD DC P4800X offers an industry-leading combination of high throughput, low latency, high QoS, and high endurance. This SSD is optimized to break through data-access bottlenecks, and is best suited for the Ceph Metadata (RocksDB & WAL) tier. Usually only a small amount of Intel® Optane™ technology is needed within the solution.

Intel® SSD D5-P4320 QLC 3D NAND SSDs offer 4 bits per cell compared to TLC NAND, which offers 3 bits per cell. Meaning the NAND is denser and less expensive making it best suited for the Ceph Object Storage Daemon* (OSD) data tier. This enables lower cost all-flash arrays that can displace HDD arrays needing more performance, as well as displace TLC all-flash arrays that are cost sensitive and need more density.

“O+Q” Ceph Block Storage Benchmarking with 2nd Gen Intel® Xeon™ Scalable Processors

As mentioned above, within a Ceph Cluster, the ”O+Q” solution involves using Intel® Optane™ SSD DC P4800X SSDs for the Metadata (RocksDB & WAL) and Intel® SSD D5-P4320 QLC 3D NAND for the OSD Data. When comparing to the previous generation of Intel technologies, this solution offers significant benefits as seen in the table shown here:

Figure 1: See Footnote #1 for configuration, testing, and pricing details

The data in the figure above was collected in Intel’s Performance Lab. As you can see, when compared to the previous generation all TLC solution, “O+Q” offers twice the storage for 10% less cost per node. This is possible because of the densely packed QLC NAND which enables less expensive and denser nodes. Additionally, the solution provides 10% better IOPS and 67% better P9999 tail latency.

Less expensive. Higher bandwidth. Lower Latency. With this solution, you can have your cake and eat it too.

Mapping Your Workload to Intel’s Solutions

In helping end users with their deployments, we usually start with two questions:

1) Is your primary use case block or object storage?

2) Are you performance bound or cost sensitive?

Then we use the chart below to map the answers to the above questions to the correct solution. As you can see, the “New Way” adds “O+Q” as the best solution for warm data (cost sensitive block storage or performance bound object storage). We recommend using this chart as a starting point when evaluating your software defined storage needs:

Figure 2: Mapping your workload to an Intel solution

My Recommendation for Block Storage SDS

Intel can’t offer you better tasting coffee or faster flights to Hawaii, but we do offer the technologies to bring greater performance and value to your SDS solution. If you are planning to build a CEPH cluster for data you want to keep “warm”, and yet want the best value, Intel can help you deploy Ceph block or object storage with “O+Q” today.

Final Comments

In this blog post I provided “O+Q” performance and pricing data for the “Value All Flash Array for Block Storage” solution shown in the chart above. In a future blog post, I will cover the “Performance All Flash Array for Object Storage” solution.
A note of thanks to Performance Engineer Orlando Moreno for running the benchmarks, Principal Engineer Tushar Gohad for providing technical reviews, and Intel Senior Solution Architect Karl Vietmeier for his contributions to the heat map chart (Figure 2).

See other ways in which Intel® Optane™ DC SSDs are making an impact in the data center.

__________________________________
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.com/benchmarks. Performance results are based on testing as of the date set forth in the configurations and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure.

Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.

1 Source - Intel MSRP and 3rd party sources. See Appendix B for detailed pricing information. System BOM costs used for $/GB calculations as follows: Baseline = $32,429; Intel Optane DC SSD + Intel QLC 3D NAND = $28,129
2 Source – Intel tested. See Appendix A for detailed system test configurations.

Appendix A – Solution Value for Ceph* Block Storage
Performance results are based on testing on February 20, 2019 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure.

1. Workload: Distributed Object Infrastructure Testing on Ceph* 13.2.4 Mimic

P4800X + P4510: Tested by Intel on 2/20/2019, 5-nodes, 2x Intel® Xeon Gold 6252 on WolfPass with 12 x 16GB 2666MHz DDR4 (total 384GB), NIC: 25x2 GbE Mellanox Conect-4 Lx CX4121A, Storage: Intel® SSD DC S3610 1.5TB, Application drive: 1x Intel® Optane™ SSD DC P4800X (375GB) + 6x Intel® SSD DC P4510 (4TB), Bios: SE5C620.86B.0D.01.0250.112320180145, ucode: 0x4000010 (HT=ON, Turbo=ON), OS: RedHat 7.6, Kernel: 3.10.0-957.el7.x86_64, Benchmark: Ceph 13.2.4 Mimic, QD= 64, Results: 4KB read = 1313300 IOPS & Latency (99.99th Percentile)= 320.13 ms, 4KB write = 291006.67 IOPS & (99.99th Percentile)= 499.68 ms, 4KB read/write (70/30) = 656517.67 IOPS & (99.99th Percentile)= 519.43 ms. Intel® SSD DC P4510 test results captured on 4TB model, while cost calculations are based on 8TB model pricing. Drive capacity is not material to test results in this benchmarking scenario.

P4800X + P4320: Tested by Intel on 2/26/2019, 5-nodes, 2x Intel® Xeon Gold 6252 on WolfPass with 12 x 16GB 2666MHz DDR4 (total 384GB), NIC: 25x2 GbE Mellanox Conect-4 Lx CX4121A, Storage: Intel® SSD DC S3610 1.5TB, Application drive: 1x Intel® Optane™ SSD DC P4800X (375GB) + 6x Intel® SSD D5-P4320 (7.68TB), Bios: SE5C620.86B.0D.01.0250.112320180145, ucode: 0x4000010 (HT=ON, Turbo=ON), OS: RedHat 7.6, Kernel: 3.10.0-957.el7.x86_64, Benchmark: Ceph 13.2.4 Mimic, QD= 64, Results: 4KB read = 1142650 IOPS & Latency (99.99%)= 323.58 ms, 4KB write = 274374.5 IOPS & Latency (99.99%)= 444.415 ms, 4KB read/write (70/30) = 596530 IOPS & Latency (99.99%)= 456.67 ms

Baseline: Tested by Intel on Feb 20, 2019, 5-nodes, 2x Intel® Xeon Gold 6152 on WolfPass with 12 x 16GB 2666MHz DDR4 (total 384GB), NIC: 25x2 GbE Mellanox Conect-4 Lx CX4121A, Storage: Intel® SSD DC S3610 1.5TB, Application drive: 1x Intel® SSD DC P4600 (2TB) + 6x Intel® SSD DC P4500 (4TB), Bios: SE5C620.86B.0D.01.0250.112320180145, ucode: 0x4000010 (HT=ON, Turbo=ON), OS: RedHat 7.6, Kernel: 3.10.0-957.el7.x86_64, Benchmark: Ceph 13.2.4 Mimic, QD= 64, Results: 4KB read = 1149766.67 IOPS & (99.99th Percentile)= 381.58 ms, 4KB write = 230116.67 IOPS & (99.99th Percentile)= 556.35 ms, 4KB read/write (70/30) = 536652.33 IOPS & (99.99th Percentile)= 574.75 ms

Client: Tested by Intel on 2/20/2019, 8x RBD Clients over 2 rack units with Intel Xeon E5-2699v4, 22 cores, 2.2GHz on Taylor Pass, MB Bios SE5C610.86B.01.01.0022.062820171903, 8x32GB 2134 DDR4, 256 GB total memory capacity, Network Intel® 1x25 GbE, storage Intel® SSD DC S3700 400GB, OS Ubuntu 16.04.2 LTS, Kernel 4.4.0-62-generic, Ceph 13.2.4 Mimic, FIO 2.20

Appendix B – Solution value for Ceph* Block Storage, Costing

Detailed pricing as of 3/28/2019:

Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.

*Other names and brands may be claimed as the property of others.
© Intel Corporation. All rights reserved.

Published on Categories StorageTags , , , ,
Justin Elkow

About Justin Elkow

Justin Elkow, Solution Architect at Intel, responsibilities include architecting software defined storage solutions utilizing Intel® technology, defining the business strategy for bringing scalable solutions to market, and executing to drive sales and solution adoption. This work includes positioning solutions to customers, influencers, and software partners, as well as train a world-wide sales force to scale business opportunities. He also evangelizes Intel technologies in public forums, conferences, and various customer events to help drive infrastructure change and business scale while connecting key technical solutions into industry verticals, such as Cloud Service Providers, Financial Services, Retail, Healthcare, Life Sciences, and Energy.