How Intel QuickAssist Technology Helps Storage and Big Data Deal With Explosive Data Growth

I recently got the opportunity to discuss the security and network optimization applications of Intel QuickAssist Technology with Allyson Klein for her “Chip Chat” podcast. I enjoy listening to Chip Chat, so it was a great experience to be a part of the podcast.

The interview was part of our launch of the new Intel® Xeon® Processor E5-2600 v3 and Intel Communications Chipset 8900 featuring QuickAssist Technology.

QuickAssist Technology provides hardware-assisted compression and cryptography for Xeon-based platforms that allows system manufacturers to implement real time compression and encryption algorithms with minimal utilization of the CPU.  Thus, they get the flexible, high performance encryption and compression while preserving processing cores for revenue-generating applications.

On the podcast, I spoke a lot about the security benefits of this technology, ranging from better performance on the ciphers that protect data transmission in 3G and 4G/LTE networks to the use of secure sockets layer (SSL) encryption on a growing range of websites and web services. QuickAssist technology provides performance boost and increased efficiency for a wide range of these applications.

But the podcast lasts only 12 minutes and what I didn’t get to discuss was the growing need for the compression capabilities of QuickAssist Technology in storage and big data analytics applications -- We are living through a time of dramatic growth in data. IDC reports that the data that is created and copied will double annually reaching 44 zettabytes (equivalent to 44 trillion gigabytes) by 2020.**  Where this hits particularly hard is with big data and storage applications and QuickAssist Technology allows system manufacturers to implement real time compression and encryption algorithms that can keep up with network and CPU performance.

Let’s look at how QuickAssist is being used in storage and big data applications:

Storage: Network performance and processor performance have increased dramatically in recent years, but hard disk drive (HDD) read/write performance has not kept pace. Thus more data centers are tiering their storage systems to put their most active data on the solid-state drives (SSDs) and less active on HDDs and using compression to get more space at every tier. Using Intel QuickAssist provides the computing power for to compress data in real-time, making it realistic to use for even the highest-performance storage tiers.

Big Data: 2015 is predicted by many to be a big year for big data, driven by the success of Hadoop, the leading big data framework. Many enterprises are running trials with Hadoop to garner useful analytics from their big data and will convert those trials into production systems in the coming year. Early big data projects focused on batch processing of data at rest, but now capabilities are evolving that enable processing and analyzing of streaming data in real-time. This evolution is unlocking a variety of exciting new use cases and applications in healthcare, telecom and capital markets.

Big data applications need compression for a couple key reasons: Hadoop is designed to run on large data sets residing in compute and storage clusters.  Those clusters rely on compression to preserve network bandwidth and storage capacity, and in turn optimize network utilization and system cost. Real time data compression delivers on these objectives, while also improving the run time of end-to-end Hadoop processing. Intel QuickAssist technology built into the storage devices and Hadoop servers will help the next generation of full scale deployments to succeed.

Compression and encryption are such fundamental processes for some of the key security and data growth challenges of our digital universe.  It’s exciting to see the new use cases for these processes that are enabled when QuickAssist Technology is used to drive new levels of performance.

What do you think? Find me on Twitter @jeni_p and let me know.