Intel Omni Path Architecture for the Tri Labs CTS-1

The market momentum for Intel Omni-Path Architecture (Intel OPA) HPC Fabric continues to grow. This is the first full quarter of Intel OPA sales and shipments, and we can already point to dozens of wins, including the National Nuclear Security Administration’s Tri Labs, Texas Advanced Computing Center, the University of Hull, the University of Cambridge, the Alfred Wegener Institute, Pittsburgh Supercomputing Center, and the University of Colorado. Intel OPA won a head-to-head competitive battle for the interconnect and was selected for the Capacity Technology Systems (CTS-1) program for the Tri Labs project at Sandia National Laboratories, Los Alamos National Laboratory, and Lawrence Livermore National Laboratory with partner Penguin Computing.  CTS-1 is a multi-year award that represents a total $39 million to provide over 7 Petaflops (quadrillion floating operations per second) of “capacity” computing capability.

According to Matt Leininger, the Deputy for Advanced Technology Projects at LLNL and acquisition manager of systems under the CTS program, “Over the procurements, these clusters have evolved to allow us to maintain the advanced computing resources we need. The Labs adopted InfiniBand Architecture in previous clusters when it came out in early 2000. InfiniBand DDR was part of the first procurement. In the second procurement, the Tri-Labs used Intel® True Scale fabric adapters on InfiniBand QDR. We saw phenomenal scalability with Intel True Scale. For CTS-1, we selected Intel OPA over the alternative. This decision was based on our performance benchmarking and assessment of any technical and schedule risks. Our expectation is that Intel OPA will continue to provide improved scalability in the new clusters.”

Penguin Computing is building the CTS-1 clusters based on their Tundra Extreme Scale cluster platform, using both Intel OPA and the latest generation Intel® Xeon processor. Selected CTS-1 shipments joins Bridges at the Pittsburgh Supercomputing Center and many others as latest-generation HPC clusters built with Intel OPA.  These clusters are being delivered, installed and becoming operational this quarter.

Sid Mair at Penguin Computing, who is head of their Federal Systems Division, said that the Tri-Labs chose Intel for CTS-1, because of the reliability and performance of our products, and the ability to deliver Intel OPA in the timeframe CTS-1 needed it. Penguin also likes the 48-port switch, which helps reduce the number of switches required to build the clusters for CTS-1 deployments, both reducing complexity and cost, according to Sid.

An added collaboration out of the CTS-1 build is a new 12-volt version of the Intel OPA switch that will plug into the Open Computing Project (OCP) framework for clusters. Tundra is built on OCP, which has a 12-volt rail inside the racks. Sid says they can increase density of the system, including the switches, by about 50 percent.

The first cluster in the CTS-1 build is currently being installed at Lawrence Livermore. We are excited to have Intel OPA, a component of the Intel Scalable Systems Framework (Intel SSF) in these systems and others to come. Congratulations to the Penguin Computing team.

In summary, Intel Omni-Path Architecture is off to great start with excellent acceptance by many customers.  Most of the customer awards were based on benchmark results where they found Intel OPA performance more than competitive as well as offering very good price/performance.  Based on this initial level of customer acceptance, we have high expectations for Intel OPA this year.

For more information, please reference the following: