As SC14 approaches next week, we have invited industry experts to share their views on high performance computing and life sciences. Below is a guest post from Charles Shiflett, senior software engineer at Aspera. Charles will be sharing his thoughts on ultra high speed WAN transfers during SC14 at the Intel booth (#1315) on Tuesday, Nov. 18 at 12:30 p.m. in the Intel Theater and on Wednesday at 2 p.m. in the Intel Community Hub.
Research is running into a problem where the amount of data that is being generated is growing faster than what can be analyzed and stored using traditional tools and architectures. This has led to an explosion of technologies and tools in processing data that take advantage of the parallelism inherent both in compute clusters and/or cloud environments. What hasn't improved at a commensurate rate is improvements to performance in storage or network throughput.
The reason storage and network throughput hasn't improved is in part a sense of complacency. In our traditional computing model, people think of internet (or network) as slow, disks as somewhat faster, and memory as screaming fast. This is completely wrong in an HPC environment, but it is the world we are stuck in when go through traditional interfaces which were designed with this paradigm in mind. Achieving breakthrough performance requires a new approach.
Using commodity Intel® hardware we were able to develop a novel solution (termed next generation Aspera FASP) which bypasses traditional kernel layers in Storage and IO. While this solution is still under development we have already been able to show over 90 percent utilization of a transfer solution using two 40 Gbit/s network cards for a total of 80 Gbit/s (see photo above). This equates to disk to disk performance at about 8 GB/s or the ability to transfer 1TB of data in about 2 minutes. At Super Computing 14, Aspera will be showcasing next generation FASP in operation over a WAN environment where we will transfer 250 GB of data in about 20 seconds from the show floor to Chicago and back.
As the developer of this solution, what excites me most is the benefit our customers will get from a high speed block based transfer solution that not only solves WAN transfer needs but does it in a way that is secure (every packet is encryped with AES-128), runs on commodity Intel hardware, and is equally applicable in both LAN and WAN environments. Our future plans are to provide this solution in a way that integrates with high performance compute packages (such as Spark or Hadoop), high performance storage (think Lustre or GPFS), while continuing to build upon the Intel and IBM Aspera technologies that have made this solution possible.
What questions do you have?