Apache Spark

1

Agenda

• Introduction• Features -

• Supports all major Big Data Environments• General platform for major Big Data tasks• Access diverse data sources • Speed• Ease of use

• Architecture• Resilient Distributed Datasets

• In-memory Computing• Performance

Apache Spark 2

Spark Introduction

• Apache Spark is an – open source, – parallel data processing framework – master slave model– complements Hadoop to make it easy to develop fast, unified Big Data applications

• Cloudera offers commercial support for Spark with Cloudera Enterprise.

• It has over 465 contributors making it most active Big Data Project in Apache Foundation, started at UC Berkley in 2009

Apache Spark 3

Supports All major BigData Environments

Runs Everywhere -• Standalone Cluster Mode• Hadoop Yarn• Apache Mesos• Amazon EC2

• Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS).

Apache Spark 4

General platform for all major Big Data tasks

• Common ETL (Sqoop)• SQL and Analytics (Pig and Hive)• Real Time Streaming (Storm)• Machine Learning (Mahout)• Graphs (Data Visualization) • Both Interactive and Batch mode processing• Reuse the same code for batch and stream processing, even joining

streaming data to historical data

Apache Spark 5

Access Diverse Data Sources

Read and write anywhere -• HDFS• Cassandra• Hbase• TextFile• RDBMS• Kafka

Apache Spark 6

Speed

• Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.

• Spark has an advanced DAG execution engine that supports cyclic data flow and in-memory computing.

Logistic regression in Hadoop and Spark

Apache Spark 7

Ease of Use

• Write application in – Java– Scala– Python

• 80 high-level operators that make it easy to build parallel apps

Apache Spark 8

Unified Analytics with Cloudera’s Enterprise Data Hub

Faster Decisions (Interactive) Better Decisions (Batch) Real-Time Action (Streaming and Applications)

Web Security

Why is my website slow? What are the common causes of performance issues?

How can I detect and block malicious attacks in real-time?

Retail

What are our top selling items across channels?

What products and services to customers buy together?

How can I deliver relevant promotions to buyers at the point of sale?

Financial Services

Who opened multiple accounts in the past 6 months?

What are the leading indicators of fraudulent activity?

How can I protect my customers from identity theft in real-time?

Apache Spark 9

ARCHITECTURE

Apache Spark 10

Apache Spark 11

Resilient Distributed Datasets

• A read-only collection of objects partitioned across a set of machines

• Immutable. You can modify an RDD with a transformation but the transformation returns you a new RDD whereas the original RDD remains the same.

• Can be rebuilt if a partition is lost.• Fault tolerant because they can be recreated & recomputed

RDD supports two types of operations:• Transformation• Action

Apache Spark 12

• Transformation: don't return a single value, they return a new RDD. Nothing gets evaluated when you call a Transformation function, it just takes an RDD and return a new RDD.– Some of the Transformation functions are map, filter, flatMap,

groupByKey, reduceByKey, aggregateByKey, pipe, and coalesce.• Action: operation evaluates and returns a new value. When an

Action function is called on a RDD object, all the data processing queries are computed at that time and the result value is returned.– Some of the Action operations are reduce, collect, count, first, take,

countByKey, and foreach.

Apache Spark 13

How in-memory computing Improve Performance

• Intermediate data is not spilled to disk

Apache Spark 14

Performance improvement in DAG executions

• New shuffle implementation, sort-based shuffle uses single buffer which reduces substantial memory overhead and can support huge workloads in a single stage. Earlier concurrent buffers were used

• The revamped network module in Spark maintains its own pool of memory, thus bypassing JVM’s memory allocator, reducing the impact of garbage collection.

• New external shuffle service ensures that Spark can still serve shuffle files even when the executors are in GC pauses.

Apache Spark 15

• Timsort, which is a derivation of quicksort and mergesort. It works better than earlier quicksort in partially ordered datasets.

• Exploiting cache locality reduced sorting by factor of 5. sort_key record sort_key location

The Spark cluster was able to sustain 3GB/s/node I/O activity during the map phase, and 1.1 GB/s/node network activity during the reduce phase, saturating the 10Gbps link available on these machines.

Apache Spark 16

10b 10b100b 4b

• The memory consumed by RDDs can be optimized per size of data• Performance can be tuned by caching• MR supports only Map and Reduce operations and everything (join,

groupby etc) has to be fit into the Map and Reduce model, which might not be the efficient way. Spark supports 80 other transformations and actions.

Apache Spark 17

References

• http://spark.apache.org/• http://en.wikipedia.org/wiki/Apache_Spark• http://opensource.com/business/15/1/apache-spark-new-world-recor

d• http://

www.cloudera.com/content/cloudera/en/products-and-services/cdh/spark.html

• http://vision.cloudera.com/mapreduce-spark/• http://

thesohiljain.blogspot.in/2015/03/apache-spark-introduction.html

Apache Spark 18

Thank You

Apache Spark 19