High Performance Computing for Data-Driven Discovery

Alison Sokoloff is a blogger and enterprise software marketeer with tenure at companies such as Intel, Oracle and Sun Microsystems. She tweets @alisongs and @hadoopintel

Data intensive applications including weather forecasting and seismic analysis have been part of the HPC industry from the early days.  Performance of today’s systems and parallel software tools has made it possible to create larger and more complex simulations. However, with unstructured data accounting for 80 percent of all data, and growing 15 times faster than other data1, the HPC industry is looking to tap into all of this information to gain valuable insights.

Intel is addressing this need with the announcement of the Intel® HPC Distribution for Apache Hadoop* software (Intel® HPC Distribution) that combines the Intel® Distribution for Apache Hadoop software with Intel® Enterprise Edition of Lustre* software to deliver an enterprise-grade solution for storing and processing large data sets. This powerful combination allows users to run their MapReduce applications, without change, directly on shared, fast Lustre-powered storage, making it fast, scalable and easy to manage.

The Intel® Cloud Edition for Lustre* software is a scalable, parallel file system that is available through the Amazon Web Services Marketplace* and allows users to pay-as-you go to maximize storage performance and cost effectiveness. The software is ideally suited for dynamic applications, including rapid simulation and prototyping. In the case of urgent or unplanned work that exceeds a user's on-premise compute or storage performance, the software can be used for cloud bursting HPC workloads to quickly provision the infrastructure needed before moving the work into the cloud.

With numerous vendors announcing pre-configured and validated hardware and software solutions featuring the Intel Enterprise Edition for Lustre, at SC'13, Intel and its ecosystem partners are bringing turnkey solutions to market to make big data processing and storage more broadly available, cost effective and easier to deploy. Partners announcing these appliances include Advanced HPC*, Aeon Computing*, ATIPA*, Boston Ltd.*, Colfax International*, E4 Computer Engineering*, NOVATTE* and System Fabric Works*.

This is a synergistic fit for Intel as Intel's HPC technologies are once again featured throughout the 42nd edition of the Top 500 list, demonstrating how the company's parallel architecture continues to be the standard building block for the world's most powerful supercomputers. Intel-based systems account for more than 82 percent of all supercomputers on the list and The complete Top500 list is available at www.top500.org.