Optimizing Big Data Infrastructure

Managing the Changing IT Landscape: Big Data Infrastructure

Big data, mobility, and cloud represent the great technology trio that enterprises consider top IT priorities. However, right up there with these high-visibility initiatives is updating data center infrastructure—a more mundane initiative but critically important to the success of the other three. Intel’s 2013 big data survey found that 50 percent of respondents listed modernizing infrastructure as one of their top three priorities.

Why Infrastructure Matters

IT Strategic Priorities.jpg

In fact, our survey showed infrastructure updates topped the strategic priority list, beating out big data, mobility, and cloud—because each of these three initiatives require a newly designed data center. Innovating with older architecture can be difficult and expensive and can add to complexity.

A data center infrastructure upgrade optimizes performance; delivers more capacity; and balances necessary compute, networking, and storage resources. Newer infrastructure featuring servers based on the Intel® Xeon® processor E5 family was designed for this purpose. These new servers pack more power into a smaller footprint, handle virtualization more efficiently, scale elastically to adapt to fluctuating workloads, and support 10 gigabit Ethernet (10 GbE) and intelligent storage solutions.

Real-time Big Data Processing

Another insight from our research is a shift in the way data is processed by IT and delivered to business leaders. The importance of real-time processing is growing, as you can see in the charts below. This is an excellent example of the changing demands on our infrastructure and the need for a balanced approach that improves performance and makes real-time analytics a reality.

How Business Analytics is Changing.jpg

The benefits of a balanced infrastructure are significant. Intel recently evaluated how a new and balanced infrastructure dramatically reduced big data processing time from four hours to seven minutes. There were four steps involved in achieving this result.

The infographic featured here shows how each of these infrastructure upgrades contribute to the processing time reduction.

The testing team began with a basic configuration of older Intel servers. Processing time benchmark: four hours.

Baseline Configuration.png

Step 1: Upgrading to new servers based on the Intel Xeon processor E5 family cut processing time in half.

Step 1.png

Step 2: Deploying new storage solutions using Intel Solid-State Drives (SSDs) shaved significantly more processing time.

step 2.png

Step 3: Implementing new networking with Intel 10 GbE solutions were faster still.

Step 3.png

Step 4: Deploying new data analytics software running the Intel Distribution for Apache Hadoop* helped achieve the final processing time: seven minutes.

step 4.png

Discover all the details of how these technologies were used together to deliver balanced infrastructure for real-time, big data analytics in the Intel IT Center white paper Big Data Technologies for Near-Real-Time Results.

From the Lab to the Business

A growing number of companies are experiencing the advantages of a balanced approach to upgrading infrastructure.

Public cloud service provider MiCloud optimized cloud performance to address increasing big data demands by building an architecture that included a similar approach to what I outlined above. They boosted internal processing speed and reduced data transmission delays by 80 percent for storage and 40 percent for network adapters.

BMW estimates it will have 10 million cloud-connected cars generating 100 million requests and 1 terabyte of data per day by 2018. The company is in the process of implementing a two-phase plan using Open Data Center Alliance* usage models for upgrading their cloud infrastructure to handle the demand.

If your organization is thinking about deploying a big data solution, is infrastructure modernization part of the plan? Be sure to join the conversation and share your challenges and successes.

Chris Peters

Chris Peters is a business strategist with more than 21 years of experience ranging from Information Technology, manufacturing, supply chain, nuclear power and consumer products.

Find him on LinkedIn.

Follow him on Twitter (@Chris_P_Intel)

Check out his previous posts and discussions

#ITCenter #Bigdata #IntelXeon