Cooling a Supercomputer: Cool Liquid in Cherry Creek at SC13

By Geoff Lyon, CoolIT Systems

Geoff Lyon is the CEO & CTO at CoolIT Systems in Calgary Alberta, Canada where he has been working to develop liquid cooling technologies and products to enable wide-spread adoption in today’s datacom industry. He is a member of IMAPS, an associate member of ASHRAE and has presented at Interpack, IEEE Semi-Therm, The Server Design Summit and various other forums. He holds a number of issued and pending patents in the field of liquid cooling and has a degree in aerospace engineering.


If you follow the posts in this blog, you will have noticed lots of mentions of the Cherry Creek system hitting #400 on the top 500 list published for November.  What you may have missed is that the system has a very unique cooling system.  A number of weeks ago, Intel contacted CoolIT with a very ambitious goal of collaborating on a Top500 Supercomputer.  What makes this particularly special was that this heterogeneous system was to fit in 2 standard racks, run on a popular Super Micro form-factor (the Fat Twin Chassis system) and that it was going to run, fully loaded, on the floor at SC13.

Which brings us to the reason that Intel called CoolIT.  The system was running a little warm and the deadline was fast approaching to build the full system and submit the performance in time to be ranked in the November Top500 list.

At CoolIT, we fully recognize the relationship between processor power and performance  - you want more performance? That could mean more power.  Intel has a fantastic, scalable offering for HPC with the Phi co-processor combining with the Xeon CPU.  The formula for performance from a hardware standpoint is pretty straightforward – put as many Xeon Phi and Xeon CPU's into a rack as possible.  The other relationship we are familiar with is: the more power that goes in, the more heat that needs to come out.  Intel's vision was simple – cool the full system with liquid, on an exhibition floor where the temperature may vary, and make it quiet enough that it is not a distraction on the floor.  Alrighty then, let’s review the quick details.

The Super Micro Fat Twin platform can support fairly high density hardware.  Each 4U Chassis can fit 12 Xeon Phi (300 Watts each) and 8 Xeon CPU's (150W Each).  Some quick math reveals that we are dealing with 4800W of just processing power and the remaining power would likely add up to about 1000-1200W, depending on power supply efficiencies, memory, voltage regulators etc.   This is a perfect set-up to use liquid cooling.  CoolIT has liquid cooling solutions for the Phi and Xeon CPU's.  The trick was to create an appropriate design for the liquid path within the server nodes,design some mechanical hardware pieces to ensure everything stayed where it belonged, and hook it up to a self-contained CoolIT DCLC™ AHx configuration.  The real challenge with this project was the schedule.  Long story short, we got it done in the nick of time.


The amazing technology, herculean efforts from all those involved in the hardware, logistics, software and coordination resulted in the final submission to the Top500 being time stamped with less than 1 minute to spare.  The system was then transported to Denver where it ran flawlessly for the duration of the show.

While we are still compiling the full system report of the various temperatures, which we are looking forward to sharing, the performance of the test using only 48 nodes was good enough to rank 400 on the Top500.  What is probably more impressive is the fact that the system ranked 41 on the Green500 consuming less than 75kW of power.  Not bad for a demonstration system!

To me, the creation of the Cherry Creek system was not really about scoring high in the rankings but to showcase that a world-class supercomputer can be assembled using readily available, off-the-shelf components.  By leveraging Direct Contact Liquid Cooling, the system was fast, cool, quiet and a pleasant shade of green.