Building Stream Processing as a Service (SPaaS)

Building Stream Processing as a Service (SPaaS)

Steven Wu @stevenzwu

Why stream processing?

Unbounded user activity stream

Alice

Bob

Jane

Unbounded data - batch

Alice

Bob

Jane

Feb 25 Feb 26


Alice

Bob

Jane

Feb 25 Feb 26


Alice

Bob

Jane

Feb 25 Feb 26


Alice

Bob

Jane

Feb 25 Feb 26

Unbounded data - stream

Alice

Bob

Jane

Agenda

● Introduction● Apache Flink primer● SPaaS Overview● Keystone Router● Custom Stream Processing Applications● Backfill and Rewind

Agenda


Real Time Data Infrastructure

Producers Consumers

Messaging As A Service (Kafka)

Stream Processing As A Service (Flink)

Keystone Data Pipeline

Topics Routing Transformations(filter, projection, UDF)

Schema / Hygiene

Stream Processing

Producers Consumers

Messaging As A Service (Kafka)

Stream Processing As A Service (Flink)

Keystone Data Pipeline

Topics Routing Transformations(filter, projection, UDF)

Schema / Hygiene

EventProducers

Sinks

highly available ingest pipelines - the backbone of a real-time data infrastructure

Agenda


Exact-once semantics for stateful computation

Source: http://flink.apache.org/

Flexible windowing


Event time semantics


State backends and checkpointing

Available

● Memory

● File system

● RocksDB (support

incremental checkpoint)



Checkpoint is lightweight

Source: Stephan Ewen

Agenda


Offerings by complexity

● Simple drag and drop: filter, projection, data hygiene

○ Available now via Keystone router

● Medium: SQL, UDF (User Defined Function)

○ Coming 2018

● Advanced: custom stream processing applications

○ Available now

Ease of use v.s. capability

difficult

easy

Limited Full feature

Color legend

Available now

Coming 2018Ease

of u

se

Capability


difficult

easy


Color legend

Available now

Coming 2018Ease

of u

se

Capability

KeystoneRouter


difficult

easy


Custom SPaaS App

Color legend

Available now

Coming 2018Ease

of u

se

Capability

KeystoneRouter


difficult

easy


KeystoneRouter

Custom SPaaS App

SQL

Color legend

Available now

Coming 2018Ease

of u

se

Capability


difficult

easy


Custom SPaaS App

Color legend

Available now

Coming 2018Ease

of u

se

Capability

KeystoneRouter + UDF

SQL + UDF

SPaaS running on Titus(Netflix’s in-house container runtime)

Job isolation: single job

Job Manager

Task Manager

Task Manager

Task Manager

...

Titus Job #1

Titus Job #2

Flink standalone

cluster

Agenda


Events are published to fronting Kafka directly or via proxy

KS

Gat

eway

Stream Consumers

EventProducer

Keystone Management

FrontingKafka

FlinkRouter

Consumer Kafka

HTTP /gRPC

Events land up in fronting Kafka cluster

KS

Gat

eway

Stream Consumers

EventProducer

Keystone Management

FrontingKafka

FlinkRouter

Consumer Kafka

HTTP /gRPC

Events are polled by router, filter and projection applied

KS

Gat

eway

Stream Consumers

Keystone Management

FrontingKafka

FlinkRouter

Consumer Kafka

EventProducer

HTTP /gRPC

Router sends events to destination

KS

Gat

eway

Stream Consumers

Keystone Management

FrontingKafka

FlinkRouter

Consumer Kafka

EventProducer

HTTP /gRPC

Keystone pipeline system boundary

FlinkRouterFronting

Kafka

EventProducer

Keystone Management

KS

Gat

eway

Consumer Kafka

Stream Consumers

HTTP /gRPC

EventProducers

Sinks

highly available ingest pipelines

Keystone scale

● >1,000,000,000 unique events

ingested per day

● >99.9999% of delivery rate

Demo: provision a data stream (mini pipeline)

https://docs.google.com/file/d/1axD_Jz2I7y1o9l1e1opxYG1Qbn4sqLCc/preview

Configure outputs

Drag-and-drop Keystone router

● Stateless and embarrassingly parallel

● ~2,000 jobs in prod

● Self serve and fully managed

● At least once delivery semantics

● Isolation

Agenda


Out-Of-The-Box Functionality

● Templates (Java /

Scala)

● Build and Deployment

tooling

● Connectors

● Dashboards

● Logs

● Alerts

● Titus Integration

● Capacity

Management

Demo: SPaaS project bootstrap

https://docs.google.com/file/d/1AoxJNWNPP70kvwfGAhnzaXvwKoL7ltbZ/preview

Skeleton code

Add business logic

Demo: create a new Flink job

https://docs.google.com/file/d/164PcycZSIILzNTQSXvdWzUCjvY3cwwXr/preview

Override source config

Override Kafka cluster VIP

kafka-test:2181 kafka-prod:2181

Override job config

Configure resources

Configure multiple sources or sinks

kafka-test:2181 kafka-prod:2181

Deep links

Duplo blocks

● Filter

● Projector

● Data Hygiene

● Connectors

Sources

● Kafka

● Hive

Supported Source and Sink Connectors

Sinks

● Elasticsearch

● Kafka

● Hive

● Keystone

Agenda


Things can go wrong

Application bug

Flink Streaming

JobSinkSource

Sink failure

Flink Streaming

JobSinkSource

Dependency service failure

Flink Streaming

JobSink

MicroService

Source

Data Enrichment

How to recover

● Backfill (available now)

● Rewind Flink job (coming soon)

How to recover

● Backfill

● Rewind Flink job

Live job continues

Live Job

SinkKafka

TimeNow ->outage period

Hive as backfill source

Live Job

Sink

Backfill Job

Kafka

TimeNow ->outage period

Choose Hive source

Configure Hive source

Not a lambda architecture

● Single streaming code base

● Just switch source from Kafka to Hive

Hive backfill probably not for stateful jobs

● Warm-up issue

● Ordering issue


● Warm-up issue

● Ordering issue

Stateful stream processor

Image adapted from Stephen Ewen

Warm-up period

SinkBackfill Job

Timeoutage periodWarm-up period

No output emit during warm-up

SinkBackfill Job

Timeoutage periodWarm-up period

Emit output No output


● Warm-up issue

● Ordering issue

Kafka: messages ordered within a partition

Source: kafka.apache.org

Hadoop input split

f0 f1 f2 f3 f4files

Hadoop input split

f0 f1 f2 f3 f4

s0 s1 s2 s3 s4 s5 s6 s7 s8 s9

files

splits

Hadoop input split

f0 f1 f2 f3 f4

s0 s1 s2 s3 s4 s5 s6 s7 s8 s9

files

splits

JobManager

TaskManager

TaskManager

TaskManager

s0 … s9

Split calculation

Hadoop input split

f0 f1 f2 f3 f4

s0 s1 s2 s3 s4 s5 s6 s7 s8 s9

files

splits

JobManager

TaskManager

TaskManager

TaskManager

s0 … s9

s0

s1

s2

s3

s4

s5

s6

s7

s8

s9

Split calculation

Split Assignment

Where is the order?

f0 (hour=0) f1 f2 (hour=23) f3 (hour=12) f4 (hour=3)

s0 s1 s2 s3 s4 s5 s6 s7 s8 s9

files

splits

TaskManager

s0 (hour=0)

...

s3 (hour=23) s6 (hour=12) s9 (hour=3)

...

Does time/ordering matters?

● Probably not for stateless computation

● Probably important for stateful

computation

Window with allowed lateness

DataStream<T> input = ...;

input

.keyBy(<key selector>)

.window(<window assigner>)

.allowedLateness(<time>)

.<windowed transformation>(<window function>);

Source: flink.apache.org

How to recover

● Backfill

● Rewind Flink job

Flink checkpoint and fault tolerance

TimeCheckpoint x-1 Checkpoint x Now

Flink checkpoint and fault tolerance

TimeCheckpoint x-1 Checkpoint x

Flink rewind

TimeCheckpoint y Checkpoint x

outage window

Checkpoint x+1

Now

Flink rewind

TimeCheckpoint y Checkpoint x

outage window

Checkpoint x+1

Kafka retention

Timeoutage window NowKafka retention

As far as we can go back

Hive backfill v.s. Flink rewind

Hive backfill Flink rewind

Warm-up issue Yes No

Ordering issue Yes No

Data retention Months Hours or days

Applicability Stateless Stateless and stateful

Pros for Hive backfill source

● Long-term storage (a few months)

● Fast recovery

○ S3 is very scalable

○ Runs in parallel with live job

Stateless Stateful


Today’s recommendation

Stateless Stateful


Is this the future?

Stateless Stateful

Flink rewind

Or is this the future?

Caveats for reprocessing

● Does not overwhelm external services

● Non-retractable sink output

● Non-replayable dependencies

Does not overwhelm external services

Flink Streaming

JobSink

MicroService

Source

10x load

10x load

● Duplicates are ok

● Idempotent sink

● Cleanable sink

○ e.g. drop Hive partition with bad data

Non-retractable sink output

Non-replayable dependencies

Flink Streaming

Job

Sink

A/BService

Source

Time


Flink Streaming

Job

Sink

A/BService

Source

Time

Process live msg X


Flink Streaming

Job

Sink

A/BService

Source

Time

Process live msg X

Alice?


Flink Streaming

Job

Sink

A/BService

Source

Time

Process live msg X

Alice? Cell A


Flink Streaming

Job

Sink

A/BService

Source

Time

Allocation change

Process live msg X


Flink Streaming

Job

Sink

A/BService

Source

Timeoutage period

Allocation change

Process live msg X


Flink Streaming

Job

Sink

A/BService

Source

Timeoutage period

RewindAllocation change

Process live msg X


Flink Streaming

Job

Sink

A/BService

Source

Timeoutage period


Process live msg X Reprocess

old msg X


Flink Streaming

Job

Sink

A/BService

Source

Timeoutage period


Process live msg X Reprocess

old msg X

Alice? Cell B

Convert table to stream

Flink Streaming

Job

Sink

A/BService

Source

Table lookup

Convert table to stream

Flink Streaming

Job

Sink

A/BSource

Source

State

A/B data becomes part of app state

Stream Kong

Putting together

Amazon EC2

Titus Container Runtime

Stream Processing Platform(Flink Streaming Engine, Config Management)

Reusable ComponentsSource & Sink Connectors, Filtering, Projection, etc.

Keystone Routers(with UDF)

Streaming Jobs

Man

agem

ent S

ervic

e & U

I

Met

rics &

Mon

itorin

g

Stre

amin

g Jo

b Dev

elopm

ent

Dash

boar

ds

SQL(with UDF)

SPaaS Layered Cake

Geico caveman, https://memegenerator.net

Thank you!

@stevenzwu

Documents

Building Stream Processing as a Service (SPaaS)