View
637
Download
1
Category
Preview:
Citation preview
Google Cloud Dataflowthe next generation of managed big data service based on the Apache Beam programming model
Sub Szabolcs Feczak, Cloud Solutions Engineer
9th Cloud & Data Center World 2016 - 한국 IDG
You leave here understanding the fundamentals of
the Apache Beam model and the Google Cloud Dataflow managed service
We have some fun.
1
Goals
2
Background and Historical overview
The trade-off quadrant of Big Data
CompletenessSpeed
Cost Optimization
Complexity
Time to Answer
MapReduce
Hadoop
Flume
Storm
Spark
MillWheel
Flink
Apache Beam
*
Batch
Streaming
Pipelines
Unified API
No Lam
bda
Iterative
Interactive
Exactly Once
State
Timers
Auto-A
wesom
e
Waterm
arks
Window
ing
High-level API
Managed Service
Triggers
Open Source
Unified Engine*
*O
ptimizer
* * **
*
* *
* *
*
Deep dive, probing familiarity with the subject
1M Devices
16.6K Events/sec
43B Events/month
518B Events/year
Before Apache Beam
Batch
Accuracy
Simplicity
Savings
Stream
Speed
Sophistication
Scalability
OROROROR
After Apache Beam
Batch
Accuracy
Simplicity
Savings
Stream
Speed
Sophistication
Scalability
ANDANDANDAND
Balancing correctness, latency and cost with a unified batch
with a streaming model
http://research.google.com/search.html?q=dataflow
Apache Beam (incubating)
Java https://github.com/GoogleCloudPlatform/DataflowJavaSDK
Python (ALPHA)
Scala /darkjh/scalaflow
/jhlch/scala-dataflow-dsl
SoftwareDevelopment Kits Runners
http://incubator.apache.org/projects/beam.htmlThe Dataflow submission to the Apache Incubator was accepted on February 1, 2016, and the resulting project is now called Apache Beam.
Spark runner@ /cloudera/spark-
dataflow
Flink runner @ /dataArtisans/flink-dataflow
• Movement
• Filtering
• Enrichment
• Shaping
• Reduction
• Batch computation
• Continuous computation
• Composition
• External orchestration
• Simulation
Where might you use Apache Beam?
AnalysisETL Orchestration
Why would you go with a managed service?
GCP
Managed Service
User Code & SDKWork Manager
Deploy & Schedule
Monitoring UI
Job Manager
Cloud Dataflow Managed Service advantages (GA since 2015 August)
Progress & Logs
Deploy Schedule & Monitor Tear Down
Worker Lifecycle ManagementCloud Dataflow Service
❯ Time & life never stop
❯ Data rates & schema are not static
❯ Scaling models are not static
❯ Non-elastic compute is wasteful and can create lag
Challenge: cost optimization
Auto-scaling800 QPS 1200 QPS 5000 QPS 50 QPS
10:00 11:00 12:00 13:00
Cloud Dataflow Service
100 mins. 65 mins.
vs.
Dynamic Work RebalancingCloud Dataflow Service
● ParDo fusion○ Producer Consumer○ Sibling○ Intelligent fusion
boundaries● Combiner lifting e.g. partial
aggregations before reduction
● http://research.google.com/search.html?q=flume%20java
...
Graph OptimizationCloud Dataflow Service
C D
C+D
consumer-producer
= ParallelDo
GBK = GroupByKey
+ = CombineValues
sibling
C D
C+D
A GBK + B
A+ GBK + B
combiner lifting
Deep dive into the programming model
The Apache Beam Logical Model
What are you computing?
Where in event time?
When in processing time?
How do refinements relate?
What are you computing?
● A Pipeline represents a graph
● Nodes are data processing
transformations
● Edges are data sets flowing
through the pipeline
● Optimized and executed as a
unit for efficiency
What are you computing? PCollections ● is a collection of homogenous
data of the same type
● Maybe be bounded or unbounded in size
● Each element has an implicit timestamp
● Initially created from backing data stores
Challenge: completeness when processing continuous data
9:008:00 14:0013:0012:0011:0010:00
8:00
8:008:00
8:00
What are you computing? PTransforms
transform PCollections into other PCollections.
What Where When How
Element-Wise(Map + Reduce = ParDo)
Aggregating(Combine, Join Group)
Composite
GroupByKey
Pair With Ones
Sum Values
Count
❯ Define new PTransforms by building up subgraphs of existing transforms
❯ Some utilities are included in the SDK• Count, RemoveDuplicates, Join,
Min, Max, Sum, ...
❯ You can define your own:• DoSomething, DoSomethingElse,
etc.
❯ Why bother?• Code reuse• Better monitoring experience
Composite PTransformsApache BeamSDK
Example: Computing Integer Sums
What Where When How
What Where When How
Example: Computing Integer Sums
Key 2
Key 1
Key 3
1
Fixed
2
3
4
Key 2
Key 1
Key 3
Sliding
123
54
Key 2
Key 1
Key 3
Sessions
2
43
1
Where in Event Time?
● Windowing divides data into event-time-based finite chunks.
● Required when doing aggregations over unbounded data.
What Where When How
What Where When How
Example: Fixed 2-minute Windows
What Where When How
When in Processing Time?
● Triggers control when results are emitted.
● Triggers are often relative to the watermark.Pr
oces
sing
Tim
e
Event Time
WatermarkSkew
What Where When How
Example: Triggering at the Watermark
What Where When How
Example: Triggering for Speculative & Late Data
What Where When How
How do Refinements Relate?
● How should multiple outputs per window accumulate?
● Appropriate choice depends on consumer.
Firing Elements
Speculative 3
Watermark 5, 1
Late 2
Total Observ 11
Discarding
3
6
2
11
Accumulating
3
9
11
23
Acc. & Retracting
3
9, -3
11, -9
11
What Where When How
Example: Add Newest, Remove Previous
1. Classic Batch 2. Batch with Fixed Windows
3. Streaming 5. Streaming with Retractions
4. Streaming with Speculative + Late Data
Customizing What Where When How
What Where When How
The key takeaway
Optimizing Your Time To Answer
More time to dig into your data
Programming
Resource provisioning
Performance tuning
Monitoring
ReliabilityDeployment & configuration
Handling Growing Scale
Utilization improvements
Data Processing with Cloud DataflowTypical Data Processing
Programming
How much more time?
You do not just save on processing, but code complexity and size as well!
Source: https://cloud.google.com/dataflow/blog/dataflow-beam-and-spark-comparison
What do customers have to say aboutGoogle Cloud Dataflow
"We are utilizing Cloud Dataflow to overcome elasticity challenges with our current Hadoop cluster. Starting with some basic ETL workflow for BigQuery ingestion, we transitioned into full blown clickstream processing and analysis. This has helped us significantly improve performance of our overall system and reduce cost."
Sudhir Hasbe, Director of Software Engineering, Zullily.com
“The current iteration of Qubit’s real-time data supply chain was heavily inspired by the ground-breaking stream processing concepts described in Google’s MillWheel paper. Today we are happy to come full circle and build streaming pipelines on top of Cloud Dataflow - which has delivered on the promise of a highly-available and fault-tolerant data processing system with an incredibly powerful and expressive API.”
Jibran Saithi, Lead Architect, Qubit
"We are very excited about the productivity benefits offered by Cloud Dataflow and Cloud Pub/Sub. It took half a day to rewrite something that had previously taken over six months to build using Spark"
Paul Clarke, Director of Technology, Ocado
“Boosting performance isn’t the only thing we want to get from the new system. Our bet is that by using cloud-managed
products we will have a much lower operational overhead. That in turn means we will have much more time to make
Spotify’s products better.”
Igor Maravić, Software Engineer working at Spotify
Demo Time!
Let’s build something - Demo!
Ingest stream from Wikipedia edits https://wikitech.wikimedia.org/wiki/Stream.wikimedia.org
Inspect the result set in our data warehouse (BigQuery)
Create a pipeline and run a Dataflow job to extract the top 10 active editors and top 10 pages edited
Extract words from a Shakespeare corpus, count the occurrences of each word, write sharded results as blobs into a key value store (Cloud Storage)
1.
2.
Thank You!cloud.google.com/dataflowcloud.google.com/blog/big-data/cloud.google.com/solutions/articles#bigdatacloud.google.com/newsletterresearch.google.com
Recommended