Leverage Mesos for running Spark Streaming production jobs

Iulian DragoșLuc Bourlier

Agenda

● Intro to Mesos

● Spark deployment modes (coarse-fine cluster client)

● Fault-tolerance○ driver failures

○ --supervise○ WAL

○ Kafka direct

● Streaming backpressure

● profit !2Leverage Mesos for running Spark Streaming production jobs

Mesos

3Leverage Mesos for running Spark Streaming production jobs

General (a “distributed kernel”)

Efficient resource management

Proven technology (in production at Apple and Twitter)

Typesafe & Mesosphere maintain the Spark/Mesos framework

Why Apache Mesos?

4

“Program against your datacenter as a single pool of resources”

Frameworks running on Mesos

HDFS

Marathon

Cassandra

ElasticSearch

Yarn (Myriad)

and of course, Spark5

Resource Scheduling with Mesos

6

2-level scheduling

Mesos offers resources to frameworks

Frameworks accept or reject offers

Frameworks are free to schedule tasks on those resources as they see fit

Offers include CPU cores, memory, ports, disk

Spark Deployment modes

7

Deployment modes

8

Driver placementClient mode:

on the machine starting the job

(developer machine)

quick and easy

if the machine stops, the job is lost

Cluster mode:

inside the cluster

managed by the framework

no need to keep the developer machine online

less convenient

Spark UI may be hard to find/unavailable

spark-shell and stdout/stderr are not available 9

Fine-grained:

Each Spark task is a Mesos task. Resources are acquired and freed as needed.

better resource sharing

slower startup time

good for batch and static streaming

Coarse-grained:

Mesos executors are acquired at the beginning of a job.Spark tasks are dispatched immediately.

fast startup for tasks

better for interactive sessions

resources are locked for the duration of the app

10

Executor policy

Spark Streaming

11Leverage Mesos for running Spark Streaming production jobs

12

Fault tolerance

13

Node dies

14

Write ahead log (WAL)

15

Driver failure

16

Driver restart

17

on restart, the driver can recover everything from checkpoint dir

can: you need to code it that way

Standalone and Yarn have --supervise for this purpose

On Mesos, use Marathon

Driver restarts

18

Recommended setup

19

use cluster-mode deployment

enable WAL or use a reliable source (Kafka)

driver restarts à la --supervise using Marathon

enable back-pressure (see next)

Back pressure inSpark Streaming

20Leverage Mesos for running Spark Streaming production jobs

Resilient system ≠ stable system

Data may arrive faster than Spark can process

⇒ scheduling delay

⇒ memory spill

⇒ driver stopped

21Leverage Mesos for running Spark Streaming production jobs

Congestion support in Spark 1.4

Static rate limit

● conservative

● difficult to find the right limit (depends on cluster size)

● one limit to rule all stream

22Leverage Mesos for running Spark Streaming production jobs

Back pressure

● (also, flow control)

● a slow consumer should slow down the producer

○ classic example: TCP

● Reactive Streams

23Leverage Mesos for running Spark Streaming production jobs

Back pressure in Spark 1.5

● dynamic rate limit

● rate estimator

○ estimates the number of element that can be safely processed by system

during the batch interval

● rate sent to the receivers

● rate limiter

○ relies on TCP to slow down producers

24Leverage Mesos for running Spark Streaming production jobs

Rate estimator

● each BatchCompleted event contains

○ processing delay, scheduling delay

○ number of element in mini-batch

● the rate is (roughly) elements/processingDelay

● but what about accumulated delay?

25Leverage Mesos for running Spark Streaming production jobs

Rate estimator

Propotional-Integrale-Derivative

● P, I, D constants can change

convergence, overshooting,

oscillations

26Leverage Mesos for running Spark Streaming production jobs

https://en.wikipedia.org/wiki/PID_controller

27Leverage Mesos for running Spark Streaming production jobs

https://en.wikipedia.org/wiki/PID_controller

Back pressure in Spark 1.5

● each input has its own estimator

● work with all stream receivers,

including KafkaDirectInputStream

● configuration:

○ spark.streaming.backpressure.enabled true

○ spark.streaming.backpressure.minRate R

28Leverage Mesos for running Spark Streaming production jobs

Limitations

● linearity assumption

● records with similar execution times

● TCP back pressure accumulates in the TCP channel

29Leverage Mesos for running Spark Streaming production jobs

Back pressure withReactive Streams

30Leverage Mesos for running Spark Streaming production jobs

Reactive Streams

● standard for asynchronous stream processing

● consumer controls rate by asking for elements from producers

with Spark Streaming

● more direct back pressure control

31Leverage Mesos for running Spark Streaming production jobs

32Leverage Mesos for running Spark Streaming production jobs

Reactive Streams

● support expected with Spark 1.6

● SPARK-10420 (Spark Streaming Receiver)

33Leverage Mesos for running Spark Streaming production jobs

Thank you

34