Enabling new applications for HPC – Intel Enterprise Edition for Lustre software

One of the advantages of software developed by open source communities is that having many eyes and hands on the software improves quality, security, and functionality. But in the case of Lustre* (the open source parallel file system widely used in high-performance computing HPC) the advantages go even beyond that. The Lustre community is not just making Lustre better, it’s enabling users to apply HPC and Lustre to solve entirely new classes of problems. Over seventy of the world’s 100 fastest computers already rely on Lustre. It’s used in scientific applications like modeling weather systems. But as the coming International Supercomputing Conference (ISC) in Frankfurt demonstrates, advances in Lustre are now enabling new applications in enterprises as well.

These new enterprise applications are leveraging HPC and the massive amounts of data it creates for new products and services, speed time to market, or create competitive advantage. For example, Big Data analytics increasingly relies on machine learning techniques to identify patterns in massive amounts of unstructured data, to develop models that represent the patterns, and to apply the models to make decisions or predictions. Companies are using machine learning to teach autonomous vehicles how to drive, robots how to balance, walk, and climb on two feet, wearable devices how to perceive surroundings and navigate through them. HPC and Lustre are driving new applications in oil and gas simulation, securities trading, and in hundreds of other applications where businesses need to find and extract insight from the exploding volume of data. Cool stuff.

Intel® Enterprise Edition for Lustre* software—a key element of the Intel® Scalable System Framework—provides a powerful Lustre distribution for mission-critical HPC applications. And as Lustre functionality grows to address these new applications, so does the community of developers and partners.  HPC and technical computing solution providers continue to introduce products based on Intel® EE for Lustre.* Atos/Bull is an early adopter, as is DDN. Supermicro announced its Total Solution for Lustre on Oracle’s ZFS Storage technology. HPE announced the HPE Apollo 4520 designed to support Intel® EE for Lustre*. Microsoft Azure and Amazon AWS offer Intel® Cloud Edition for Lustre* software to their customers. And Seagate—who until recently offered their own distribution—announced it will incorporate Intel® EE for Lustre* into its ClusterStor storage architecture for HPC and that it will now adopt Intel® EE for Lustre* as its baseline Lustre distribution.

The ISC event will highlight many of these new application and solutions with more than a dozen presentations and demos, at least seven partner booths showing Lustre-based products or services, and a Lustre partner training day led by Intel experts. For example, DDN Storage will present on accelerating parallel I/O with non-volatile memory express (NVMe) devices and will demonstrate an NVMe device integrated with Intel® EE for Lustre*. Edinburgh Parallel Computing Centre will examine next-generation I/O architectures for NVDIMM. Altair will discuss I/O performance of their OptiStruct and FEKO solvers using Intel® EE for Lustre*. NEC will demonstrate Lustre on ZFS with Omni-Path. RSC Group will show their ultra-compact RSC Tornado liquid-cooled cluster solution based on Intel® Xeon® E5-2600 v4 and Intel® Xeon Phi™ and running Lustre. Ellexus will discuss the role of high-performance I/O in profiling genome pipelines in cancer research. Grau Data AG will highlight their hierarchical storage management solutions for HPC. And Boston Limited will show how to accelerate visual effects workloads with their dataScaler-L using Intel® EE for Lustre*. These are all in the Intel Collaboration Hub at ISC.

These new enterprise-oriented solutions and the availability of HPC services based on Intel® EE for Lustre* in Microsoft Azure and Amazon AWS are letting businesses and institutions access the kind of computing power that previously was only available in costly supercomputers. Enterprises are discovering new applications for HPC and Lustre in almost every industry. So if you’re attending ISC June 19-23 in Frankfurt, visit our booth and those of our partners to learn how you might apply HPC and Lustre to drive innovation in your business. Lustre is going to keep getting better and solving new kinds of problems. They might as well be yours.