View
167
Download
0
Category
Preview:
Citation preview
Big thanks to everyone!
The convergence ofreal-time analytics and
event-driven applications@StephanEwen
Flink Forward San FranciscoApril 11, 2017
2
3
2016 was the year when streaming technologies became mainstream
2017 is the year to realize the full spectrum
of streaming applications
Some large scale streaming applications
4
5
Detecting fraud in real time
As fraudsters get better, need to update models without downtime
Live 24/7 service
Credit card transactions
Notificationsand alerts
Evolving fraudmodels built bydata scientists
@
6
@ Athena X SQL to define metrics Thresholds and actions to trigger Blends analytics and
actionsStreams from Hadoop, Kafka, etc
SQL, thresholds,
actions
AnalyticsAlerts
Derived streams
7
Route events to Kafka, ES, Hive Complex interaction sessions rules Mix of stateless / small state / large state
Stream Processing as a Service• Launching, monitoring, scaling, updating• DSL to define jobs
@
8
Blink based on Flink A core system in Alibaba Search
• Machine learning, search, recommendations• A/B testing of search algorithms• Online feature updates to boost conversion rate
Alibaba is a major contributor to Flink Contributing many changes back to open source
@
9
@
Complete social network implementedusing event sourcing andCQRS (Command Query Responsibility Segregation)
What can we learn from these?
10
All these applications run on Flink Applications, not just analytics
• Not just finding out what the data means but acting on that at the same time
Workloads going beyond the traditional Hadoop realm• Hadoop is possible deploy, source, and sink• Container engines and other storage systems
increasingly popular with Flink
So, what is data streaming?
11
First wave for streaming was lambda architecture• Aid batch systems to be more real-time
Second wave was analytics (real time and lag-time)• Based on distributed collections, functions, and
windows
The next wave is much broader:A new architecture for event-driven applications
12
Event–driven applications
Events, State, Time, and Snapshots
14
f(a,b)
Event-driven functionexecuted distributedly
Events, State, Time, and Snapshots
15
f(a,b)
Maintain fault tolerant local state similar toany normal application
Events, State, Time, and Snapshots
16
f(a,b)
wall clock
event time clock
Access and react tonotions of time and progress,handle out-of-order events
Events, State, Time, and Snapshots
17
f(a,b)
wall clock
event time clock
Snapshot point-in-timeview for recovery,rollback, cloning,versioning, etc.
Event–driven applications
18
Event-drivenApplications
Stream Processing
Batch Processing
Stateful, event-driven,event-time-aware processing
(event sourcing, CQRS, …)
(streams, windows, …)
(data sets)
The APIs
19
Process Function (events, state, time)
DataStream API (streams, windows)
Table API (dynamic tables)
Stream SQL
Stream- &Batch Processing
Analytics
StatefulEvent-DrivenApplications
Process Function
20
class MyFunction extends ProcessFunction[MyEvent, Result] {
// declare state to use in the program lazy val state: ValueState[CountWithTimestamp] = getRuntimeContext().getState(…)
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]): Unit = { // work with event and state (event, state.value) match { … }
out.collect(…) // emit events state.update(…) // modify state
// schedule a timer callback ctx.timerService.registerEventTimeTimer(event.timestamp + 500) }
def onTimer(timestamp: Long, ctx: OnTimerContext, out: Collector[Result]): Unit = { // handle callback when event-/processing- time instant is reached }}
Data Stream API
21
val lines: DataStream[String] = env.addSource( new FlinkKafkaConsumer09<>(…))
val events: DataStream[Event] = lines.map((line) => parse(line))
val stats: DataStream[Statistic] = stream .keyBy("sensor") .timeWindow(Time.seconds(5)) .sum(new MyAggregationFunction())
stats.addSink(new RollingSink(path))
Table API & Stream SQL
22
Streaming Architecturefor Event-driven Applications
23
Compute, State, and Storage
24
Classic tiered architecture
Streaming architecture
database
layer
computelayer
application state+ backup
compute+
stream storageand
snapshot storage(backup)
application state
Performance
25
synchronous reads/writesacross tier boundary
asynchronous writesof large blobs
all modificationsare local
Classic tiered architecture
Streaming architecture
Consistency
26
distributed transactions
at scale typicallyat-most / at-least once
exactly onceper state
=1 =1snapshot consistency
across states
Classic tiered architecture
Streaming architecture
Scaling a Service
27
separately provision additionaldatabase capacity
provision computeand state together
Classic tiered architecture
Streaming architecture
provision compute
Rolling out a new Service
28
provision a new database(or add capacity to an existing one)
provision compute
and state together
simply occupies someadditional backup
space
Classic tiered architecture
Streaming architecture
Time, Completeness, Out-of-order
29
?
event time clocksdefine data
completenessevent time timers
handle actions for
out-of-order data
Classic tiered architecture
Streaming architecture
Repair External State
30
Streaming architecture
streams(lets say Kafka etc) live application external state
wrong results
backed up data(HDFS, S3, etc.)
Repair External State
31
Streaming architecture
live application external state
overwritewith correct results
streams(lets say Kafka etc)
backed up data(HDFS, S3, etc.) application on backup
input
Repair External State
32
Streaming architecture
live application external state
overwritewith correct results
streams(lets say Kafka etc)
backed up date(HDFS, S3, etc.)
Each service doubles as a batch job!
application on backup input
33
Streaming has outgrown the Hadoop Stack
Event-driven applications and realtime analytics converge with Apache Flink
Event-driven applications become easierto manage, faster, and more powerful following a
streaming architecture implemented with Flink
Recommended