Simplifying and securing Apache Hadoop deployments in virtualized data centers


For enterprises looking to deploy big data analytics workloads on an agile, elastic, and trustworthy infrastructure, the Intel® Distribution for Apache Hadoop* (IDH) validated on VMware* vSphere* with Big Data Extensions (BDE) provides industry-leading performance, security, and manageability. Optimized and tested on a cluster of more than 500 virtual machines running on Dell* PowerEdge* servers, the joint solution demonstrates multi-tenancy and resource elasticity at scale.

Intel and Dell encourage enterprises that want to test and develop data-driven services with agility and reliability to bring their big data use cases to the Dell Solution Center for a proof-of-concept with IDH on VMware vSphere.


  • Industry-leading security with enhanced authentication, authorization, and auditing mechanisms built in to Apache Hadoop, along with hardware-enhanced encryption/decryption and VM-based isolation
  • Hardware-enhanced performance based on Intel® Architecture and software optimizations such as native task execution and Hive/HBase enhancements
  • Reliability based on open-source disaster recovery in Apache Hadoop and vSphere fault-tolerance and high availability
  • Proven at scale on more than 500 VMs running on Dell PowerEdge servers based on Intel® Xeon® E5 processors in a test environment at the Dell Solution Center

Solution Overview

The integration of IDH for Apache Hadoop with VMware vSphere Big Data Extensions provides enterprise customers with a software platform for big data analytics that can be deployed on a virtualized infrastructure. The solution consists of IDH software optimized, tested, and validated on VMware vSphere with Big Data Extensions. The combination delivers both agility and security, taking advantage of the automation, fault-tolerance, and isolation of vSphere software and the performance, security, and manageability of IDH, which is integrated via vSphere Big Data Extensions.

IDH is a software platform that provides distributed data processing and data management for enterprise applications that analyze massive amounts of diverse data. IDH includes all essential components of open-source Apache Hadoop enhanced by Intel, along with management components originating from Intel. Proven in production at some of the most demanding enterprise deployments in the world, IDH is supported by experts at Intel with deep performance optimization experience in the Apache Hadoop software stack as well as knowledge of the underlying processor, storage, and networking components.

VMware vSphere, which is a virtualization platform designed for organizations that want to virtualize the entire data center and deliver IT as a service, delivers efficiency through higher infrastructure utilization and automation. VMware vSphere Big Data Extensions (BDE) is a feature within vSphere to support Apache Hadoop workloads. BDE provides an integrated set of management tools to help enterprises deploy, run, and manage Hadoop on a common virtual infrastructure. Through the vSphere management tool, vCenter, enterprises are able to manage and scale Hadoop seamlessly through the vSphere platform. BDE enable enterprises to deploy Hadoop clusters quickly and take advantage of the isolation, automation, and fault-tolerance inherent in vSphere.

With the integration of Intel Distribution with vSphere BDE, enterprises can simplify the deployment and management of Apache Hadoop in their virtualized data centers. In addition, Intel Distribution uses Intel® Xeon® processor features—such as Intel® Advanced Encryption Standard – New Instructions (AES-NI®)—exposed through vSphere to provide hardware-enhanced security. Along with faster encryption and decryption, Intel Distribution includes robust authentication, access control, and auditing mechanisms that enhance the security of the software platform. Intel Distribution also improves the performance of Hadoop jobs by making better use of Intel Xeon processors, solid state storage, and networking.

The combination of Intel Distribution for Apache Hadoop with VMware vSphere delivers a big data software platform that is agile, elastic, and trusted by enterprises worldwide. Intel, VMware, and Dell have collaborated to test this integration at scale on up to 850 virtual machines (VMs) at the Dell Solution Center.

Key Capabilities of the VMware vSphere with Big Data Extensions

Virtualizing Hadoop using VMware vSphere gives enterprises new levels of agility to help deploy, run, and manage Hadoop clusters while maintaining system performance on par with physical deployments. VMware vSphere Big Data Extensions, or BDE, is the feature within vSphere that enables enterprises to run clustered, scale-out Hadoop applications through vSphere.

Virtualizing Hadoop on vSphere using BDE frees enterprises from buying dedicated hardware for Hadoop. It also allows for the separation of data and compute, which paves the way for mixed workload environments and the ability to run different workloads simultaneously on the same physical host. In addition, BDE delivers increased reliability and security by leveraging proven vSphere technology, and improves agility with the vCenter user interface, vSphere’s management tool, which enables enterprises to manage and scale Hadoop seamlessly through the vSphere platform.

Along with enabling easy deployment and management of Hadoop clusters on a common virtual infrastructure, VMware vSphere BDE enables enterprises to:

  • Elastically scale Hadoop clusters and automatically rebalance resources to meet performance SLAs
  • Separate data and compute nodes through VM deployment to create a true multi-tenant environment
  • Control data access through VM-based isolation to manage resource, version, and data isolation
  • Leverage reliability of vSphere HA/FT to offer a more comprehensive availability solution across multiple single points of failure

Where to Learn more…

To discover more about what the Intel Distribution for Apache Hadoop (IDH) and Virtualizing Hadoop using VMware vSphere Big Data Extensions (BDE) can do for your organization, please check out

Until next time,