Accelerate Deep Learning Workloads on Apache Spark with BigDL

Apache Spark* has emerged as a game-changer for big data processing in recent years. According to Wikibon, the adoption of Apache Spark is gaining momentum and will reach a growth rate of 72% by 20191. With its open-source, in-memory architecture, Spark can process and analyze not only batch data but massive volumes of streaming data in real-time. Another key benefit of Spark is its ability to train machine and deep learning models at scale as well as handle an entire analytics pipeline.

Intel has made significant technical contributions to the Apache Spark community including an Intel-led open source initiative called BigDL. BigDL is a distributed library for deep learning applications. Unlike a number of other libraries for building deep learning applications, BigDL is native to Apache Spark. With BigDL, you can write deep learning applications as standard Spark programs that run on existing Spark or Hadoop* clusters. By using infrastructure already in place instead of deploying a new cluster with an unfamiliar architecture, BigDL accelerates time to value, reduces TCO, and improves ease of use.

BigDL enables developers and data scientists to build deep learning applications while leveraging their existing investments in Spark and Hadoop infrastructure. BigDL has numerous supporters in the industry including Microsoft Azure, Cloudera, AWS, JD.com, Databricks, Cray, and GigaSpaces, among others.

In a recent webinar hosted by Intel (“Simplifying Deep Learning and AI Innovation”) we invited speakers from GigaSpaces and Intel to discuss how simplified deep learning (BigDL), unified analytics, and in-memory computing can empower AI-driven applications. GigaSpaces showcased a demo utilizing natural language processing and search to deliver a highly capable, real-time interactive voice response (IVR) agent. Such a system can analyze customer sentiment and prioritize support more rapidly and scale more efficiently than a traditional telephone support system. Ultimately, it can enhance customer experiences by providing personalized support more rapidly.

Here are a few highlights from the webinar:

  1. Deep Learning Enters Mainstream. Enterprises of all types and sizes are building deep learning workloads that span fraud detection, image recognition, text classification, sentiment analysis, chatbots, product recommendation, and weather forecasting to name a few.
  2. BigDL’s Ease of Use. GigaSpaces and Intel spoke to BigDL’s ability to facilitate deep learning on existing Spark infrastructure, bridging the gap between the big data and the artificial intelligence communities.
  3. Marrying BigDL and In-memory Compute. BigDL’s integration with GigaSpaces’ InsightEdge Platform enables data to be analyzed as soon as it is created, delivering actionable insights more rapidly and providing the level of responsiveness that real-time interactive voice response agents require.

 To learn more, we invite you to attend the on-demand webinar “Simplifying Deep Learning and AI Innovation.”