Lustre in 2015, A Quick Look Back

This has been an important year for Lustre. The file system now appears in over 70 percent of the top 100 supercomputers listed in the Top500 of the fastest systems in the world. And it’s in nine of the top ten machines. That’s quite a testimonial about Lustre’s performance and its return to widespread use in the HPC community.

Lustre is a critical technology with a strong position in the High Performance Computing (HPC) market. It continues to lead on the performance side and has matured nicely, being more reliable and stable with richer features than ever. We believe the surge for incorporating Lustre in customer designs, besides performance, is because it is open source, it’s available, it runs on any platform, and it’s supported by focused experts backed by Intel’s commitment to the technology.

As the democratization of HPC gains momentum and demand for high performance data analytics (Hadoop workloads) increases, enterprise companies, not just the labs, are taking a look at the use of HPC in their data centers. And, they’re looking at Lustre, realizing the benefits of an extreme speed, open source parallel file system over alternatives, such as NFS and GPFS.

For example, the Bank of Italy has just put a Lustre-based system in production that converges both their scientific computing workloads and enterprise office automation and file sharing. TATA Consultancy Services in India has run some interesting testing with real financial workloads—not canned benchmarks—with Hadoop running on an HPC cluster, using Intel Enterprise Edition for Lustre* software as the file system and the adaptors for Hadoop that allow Hadoop’s data analytics workloads to run on HPC and Lustre. They’ve seen a dramatic performance improvement with Hadoop on HPC and Lustre. We’re also talking with a customer who wants to deploy in an ongoing manner, a significant number of petabytes per quarter using Lustre for machine learning and data streaming workloads. So, there’s a growing interest in Lustre in enterprise environments.

As many of you know, I started Whamcloud after my time at Lawrence Livermore Labs. Since the acquisition of Whamcloud, I’ve gotten to experience firsthand the changes to Lustre through what Intel is doing, the Community, and watching the vendor, partner, and customer acceptance of this parallel file system.

Lustre is incredibly complex, which is one of the reasons Lustre experts are few and far between; Intel employs most of them, and Intel continues to be the leading contributor to the Lustre code and community. We are staying committed to the technology, and we look forward to our efforts throughout 2016.