Lustre is accelerating but do you know if you have speed bumps?

Lustre has gone through some big enhancements in the last few years.  ZFS has been added as a back-end filesystem, Multiple Metadata targets allow high levels of performance and whole file snapshots provide a new mechanism for data protection.    The Lustre 2.11 release will provide a huge boost in small file performance with support with Data-on-metadata (DOM) to enable an entirely new category of analytics workloads on Lustre systems.

While Lustre was getting better, hardware was also improving.  Hard-drives alone no longer suffice to meet the performance and capacity needs for Metadata.  The Intel® SSD DC Family for PCIe provide superior processing and memory capability with a fraction of the footprint, power consumption, TCO and storage latency compared to legacy HDDs.  The Intel® Optane™ SSD DC P4800X Series provides the necessary capacity and performance for a high performance ZFS Data on Metadata service for billions of files and tens of thousands of IOPS per drive.

A typical HDD deployment is 60-90 Drivers per OSS and 24 HDDs on the Metadata Server OSS per 60 drive with 2TB

Stretching and sharing the new condensed capacity: Beyond density and raw NVMe performance, Attala Systems, Supermicro and Intel have collaborated to create a solution optimized for Lustre 2.11 and beyond.  The Attala highly-available and high-performance storage solution is based on an Intel FPGA-based scale-out fabric that leverages standard Ethernet to connect Lustre MDTs and OSTs to Intel NVMe SSDs residing in one or more 1U JBOFs with minimal added latency or costs.  The solution enables:

  • Near-native NVMe SSD performance for the higher IOPS driven by small files
  • “Parallel data sharing” including shared volumes to enable MDT/OST node HA/failover
  • The ability to “carve and isolate” from the concentrated, scale-out NVMe SSD capacity to create logical NVMe-over-fabric SANs for each MDT/OST active-active node pairs
Attala architecture with Supermicro systems

At the 2017 Supercomputer Conference in Denver, the three companies showed how this solution is used for two commonly used HPC cluster compute and file solutions:

  • Lustre for small file performance - While most data reside in big files, most files (70%) are small with sizes smaller than the default Lustre 1MB stripe size. The Lustre 2.11 increases speeds up small file performance – by up to 4X - via Data-on-MDT (Meta Data Target) on same storage NVMe based tests.  The performance and high-availability are enabled via Attala Storage and Intel® Optane™ SSD DC P4800X Series with “parallel data sharing” to enable Lustre fail-over and scale-out capability
  • Multi-tenant Spark – To support multiple users (departments, test/dev, QA, production) and the need to spin-up instant clusters, virtualization (e.g. KVM) solves half the problem by avoiding need to dedicate CPU/memory for different apps. Attala Storage solves the other half via the ability to instantly provision, “carve and isolate” Intel® SSD DC Family for PCIe* resources for multiple users.  The result is an unprecedented multi-tenant solution for instant-clusters with lower cost & better-than-bare-metal performance.


Taufik Ma, CEO, Attala Systems
Keith Mannthey, Solutions Architect, Intel
Andrey Kudryavtsev, SSD Solution Architect, Intel

Published on Categories StorageTags , ,
Andrey Kudryavtsev

About Andrey Kudryavtsev

Andrey Kudryavtsev is SSD Solution Architect in the NVM Solution Group at Intel. His main focus is the HPC area, where he helps end-customers and eco system partners to utilize the benefits of modern storage technologies and accelerate the SSD adoption for NVMe. He holds more than 12 years of total server experience, the last 10 years working for Intel. He is the guru of engineering creativity and is an influence in his field. He graduated from Nizhny Novgorod State University in Russia by Computer Science in 2004. Outside of work, he is the owner and coauthor of many experimental technologies in music, musical instruments, and multi-touch surfaces.