Cisco Validated design for Intel Distribution of Hadoop

Cisco Validated design for IDH

In a previous blog, we discussed Intel’s partnership with Cisco to provide a complete end to end solution for Hadoop platform using Intel’s distribution of Hadoop (IDH) on the Cisco’s UPA architecture. To further solidify the partnership, Cisco recently released a Cisco validated design document (CVD) for IDH which is intended to facilitate highly reliable and predictable customer deployments.

The intended audience for the Cisco validated design (CVD) document is typically sales engineers, solution architects, field consultants, partner engineering teams, IT managers and customers alike. The CVD describes the architecture and deployment procedures of IDH on a 64 node cluster based on Cisco’s Common Platform Architecture (CPA) version 2 for big data platforms. 

Hardware Platform

The Cisco’s CPA as described here is a field tested hardware architecture framework for Big data solution and consists of primarily the Cisco UCS C240 M3 Rack-Mount Servers, the Cisco UCS 6200 Series Fabric Interconnects, the Cisco UCS 2200 Series Fabric Extenders and the Cisco UCS Manager components. The CVD provides detailed hardware configuration description to help an IT manager or solution architect designing a Hadoop cluster, to narrow down to the optimal configuration quickly. It also includes clear instructions on cabling and setting up the network fabric and rack extensions for the servers and disks. With clearly defined system configuration steps using the UCS Manager specified in the CVD, it is possible to get a cluster platform ready for Hadoop deployment within hours.

Software Stack

At the lowest level of the software stack, the UCS Manager allows an IT systems administrator to define the size and capacity of the cluster with simple User Interface (UI) driven framework. It also provides the facility to create profile pools of nodes that can be used to replenish the nodes of the Hadoop cluster.

The UCS Manager also lets the system administrator install the Operating System on all nodes of the cluster. The Supported OS on the CPA is RedHat Enterprise Linux (RHEL). For a complete list of supported versions of RHEL refer to the CVD here.

Currently, Intel Hadoop version 2.5.1 has been tested and certified to be supported on top of the software stack for this CVD. The Intel Manager which is part of IDH provides the ability to install, configure, administer and monitor the Hadoop cluster. IDH also provides an enhanced Hbase and Hive distribution that brings near realtime performance for big data analytics with a SQL like interface which makes it an ideal platform for both analytics and transaction processing workloads for big data. In addition, the formal, comprehensive security framework across all components of the Hadoop distribution that IDH brings to this solution makes it a unique solution out of the box, for a variety of enterprise IT big data problems.

Testing and Validation

As part of the Validation process, a complete cluster was built as documented in the CVD and relevant Hadoop workloads were run to ensure that all components that were installed ran successfully. Further details on features of the total solution are described in the CVD.