Upload
alluxio-inc
View
622
Download
1
Embed Size (px)
Citation preview
M e m o r y - S p e e d V i r t u a l D i s t r i b u t e d S t o r a g e : L a t e s t F e a t u r e s a n d U s e C a s e s
August 2016 @ Strata+Hadoop World Beijing
Bin Fan, Yupeng Fu
1
About Us
• Bin Fan
• Software Engineer @ Alluxio, Inc.
• Alluxio PMC
• Worked at Google, MSR
• CMU, CUHK, USTC
• Wechat @ apc999_fb
• Yupeng Fu
• Software Engineer @ Alluxio, Inc.
• Alluxio PMC
• Worked at Palantir, Google
• UCSD, Tsinghua
• Wechat @richbird9
2
• Team consists of Alluxio creators and top committers
• Invested by
• Committed to Alluxio Open Source
• http://www.alluxio.com
Alluxio Inc.
3
Outline
• What is Alluxio • Example use cases
1. Off-heap memory to alleviate resource pressure 2. Fast data sharing between jobs 3. Accelerate access to remote storage 4. Unified namespace
• Summary
4
Non-persistent data-storage
software.
ATTRIBUTES
Memory-Speed Virtual Distributed Storage
Scale out architecture.
Virtualized across different storage types under a unified namespace.
Memory-speed access to data.
5
OPEN SOURCE ALLUXIO: History
• Started in the AMPLab at UC Berkeley – Summer of 2012
• Open Sourced as Tachyon (renamed to Alluxio) – Apache 2.0 License
– Latest Release: version 1.2.0 (July, 2016)
6
OPEN SOURCE ALLUXIO
v0.2 v0.3 v0.4 v0.5
v0.6 v0.7
v0.8
v1.0
v1.1
7
OPEN SOURCE ALLUXIO
The fastest growing open-source project in the big data ecosystem. Currently over 300 contributors from over 100 organizations.
8
Outline
• What is Alluxio • Example use cases
1. Off-heap memory to alleviate resource pressure 2. Fast data sharing between jobs 3. Accelerate access to remote storage 4. Unified namespace
• Summary
9
Application 1
Alluxio
CASE 1: Off-heap Memory Storage to Alleviate Resource Pressure
10
HDFS / Amazon S3
Consolidating Memory
Spark Job1
Spark Memory
block 1
block 3
Spark Job2
Spark Memory
block 3
block 1
block 1
block 3
block 2
block 4
storage engine & execution engine same process
Data duplicated at memory-level
11
Spark Job1
Spark mem
Spark Job2
Spark mem
HDFS / Amazon S3 block 1
block 3
block 2
block 4
HDFS disk
block 1
block 3
block 2
block 4 Alluxio
In-Memory
block 1
block 3 block 4
Data not duplicated at memory-level
Consolidating Memory
storage engine & execution engine same process
12
Data Resilience during Crashes
Spark Task
Spark Memory block manager
block 1
block 3
HDFS / Amazon S3 block 1
block 3
block 2
block 4
Process crash requires network and/or disk I/O to re-read the data
storage engine & execution engine same process
13
Data Resilience during Crashes
Crash
Spark Memory block manager
block 1
block 3
HDFS / Amazon S3 block 1
block 3
block 2
block 4
Process crash requires network and/or disk I/O to re-read the data
storage engine & execution engine same process
14
HDFS / Amazon S3
Data Resilience during Crashes
block 1
block 3
block 2
block 4
Crash
storage engine & execution engine same process
Process crash requires network and/or disk I/O to re-read the data
15
Spark Task
Spark Memory block manager
HDFS disk
block 1
block 3
block 2
block 4 Alluxio
In-Memory
block 1
block 3 block 4
Process crash only needs memory I/O to re-read the data
Data Resilience during Crashes
storage engine & execution engine same process
16
Crash
Process crash only needs memory I/O to re-read the data
HDFS disk
block 1
block 3
block 2
block 4 Alluxio
In-Memory
block 1
block 3 block 4
Data Resilience during Crashes
storage engine & execution engine same process
17
CASE STUDY
Thanks to Alluxio, we now have the raw data immediately available at every iteration and we can skip the costs of loading in terms of time waiting, network traffic, and RDBMS activity. - Henry Powell, Barclays
RESULTS
• Barclays workflow iteration time decreased from hours to seconds
• Alluxio enabled workflows that were
impossible before • By keeping data only in memory, the I/O cost of
loading and storing in Alluxio is now on the order of seconds
Relational Database
Share Data Across Jobs at Memory Speed
• Reducing time from hours to
seconds
• Solving issues of Teradata as
under storage
18
Barclays Previous Architecture
Spark
Teradata
Slow ETL, 30+ minutes
Each context restart requires ETL process
again
h"ps://dzone.com/ar1cles/Accelerate-‐In-‐Memory-‐Processing-‐with-‐Spark-‐from-‐Hours-‐to-‐Seconds-‐With-‐Tachyon
19
Barclays Architecture with Alluxio
Spark
Teradata
Alluxio
ETL only once Data still available in Alluxio memory after Job crash
20
CASE 2: Fast Data sharing between apps
Application 1
Alluxio
Application2
HDFS
21
Spark Job
Spark Memory
block 1
block 3
Hadoop MR Job
YARN
HDFS / Amazon S3 block 1
block 3
block 2
block 4
storage engine & execution engine same process
Data Sharing Between Frameworks
Inter-process sharing slowed down by network and/or disk I/O
22
Data Sharing Between Frameworks
Spark Job
Spark Memory
Hadoop MR Job
YARN
HDFS / Amazon S3 block 1
block 3
block 2
block 4
HDFS disk
block 1
block 3
block 2
block 4 Alluxio
In-Memory
block 1
block 3 block 4
storage engine & execution engine same process
Inter-process sharing can happen at memory speed
23
CASE STUDY
We’ve been running Alluxio in production for over 9 months, resulting in 15x speedup on average, and 300x speedup at peak service times. - Xueyan Li, Qunar
RESULTS
• Multiple computing frameworks can share data
at memory speed with Alluxio • Improved the performance of their system with
15x – 300x speedups • Alluxio provides various user-friendly APIs,
which simplifies the transition to Alluxio.
Sharing Data Across Different Compute Frameworks
• 6 billion logs (4.5 TB) daily
• Mix of Memory + HDD
24
Qunar Previous Architecture
Flink Streaming Spark Spark
Streaming Flink
HDFS Cluster
Some jobs cannot finish
Slow access to the storage
25
Qunar Architecture with Alluxio
Flink Streaming
HDFS Cluster
Spark Spark Streaming Flink
Alluxio
Share in-memory data across frameworks
26
CASE 3: Accelerate access to remote storage
Application
Alluxio
Remote Storage
27
• Different compute and storage hardware requirements
• Scale compute and storage resources independently
• Cost effective object stores (Amazon S3, Google GCS, Microsoft Azure
Blob Store) are inherently decoupled
Decoupling Compute from Storage
Accessing data requires remote I/O!
28
Remote I/O problems
Application
Remote Storage
every data operation requires data transfer, sometimes over
the WAN
high latency, network throughput
29
Visualizing the Stack with Alluxio
FAST 104 - 105 MB/s
Application
Remote Storage
MODERATE 103 - 104 MB/s
SLOW 102 - 103 MB/s
Alluxio MEM Often
Only When Necessary
Alluxio SSD/HDD
Limited
30
What if memory capacity is still not enough?
31
Alluxio Manages All Local Storage
MEM
SSD
HDD
Faster
Higher Capacity
32
Configurable Storage Tiers
MEM only
MEM + HDD
SSD only
33
Pluggable Tier Management Policies
Evict stale data to slower tier
Promote hot data to faster tier
34
CASE STUDY
Baidu File System
The performance was amazing. With Spark SQL alone, it took 100-150 seconds to finish a query; using Alluxio, where data may hit local or remote Alluxio nodes, it took 10-15 seconds. - Shaoshan Liu, Baidu
RESULTS
• Data queries are now 30x faster with Alluxio • 1000-worker Alluxio cluster run stably,
providing over 50TB of RAM space • By using Alluxio, batch queries usually lasting
over 15 minutes were transformed into an interactive query taking less than 30 seconds
Accelerate Access to Remote Storage
• 1000+ nodes deployment
• 2+ petabytes of storage
• Mix of memory + HDD
• 1000-worker cluster
35
CASE 4: Unified Name Space
36
Application
Alluxio
AWS S3 HDFS
Common Scenarios
• At large organizations, data spans many storage systems
• Legacy storage systems
• Application logic needs to integrate with different storage systems
37
Unified Name Space An abstraction for applications to access different storage systems through the same interface • Benefits – Application written once with Allluxio API – Transparently add new storage systems – Seamlessly integrate with existing systems
38
Transparent Naming
Operations over persisted Alluxio objects mapped
transparently to underlying storage
alluxio://Data/Sales <-> hdfs://Data/Sales
Alluxio Storage System (HDFS, S3, …)
alluxio://host:port/
Data Users
Reports Sales Alice Bob
hdfs://host:port/
Data Users
Reports Sales Alice Bob
39
mount
Multiple Storage Systems
• Unified namespace for multiple data sources
alluxio://Data/Sales -> s3://bucket/Sales
alluxio://Users/Alice -> hdfs://Users/Alice
• API for on-the-fly mounting / unmounting
Alluxio Storage System A
alluxio://host:port/
Data Users
Alice Bob
hdfs://host:port/
Users
Alice Bob
Storage System B
s3://host/bucket
Reports Sales Reports Sales
40
mount
mount
A Proliferation of Under Storage Systems
MORE…
41
CASE STUDY
RESULTS
• Alluxio’s unified namespace enables different applications and frameworks to easily interact with their data from different storage systems
• Global data centers across North America and
Asia • Tiered storage feature manages various
storage resources including memory, SSD and disk
Transparently Manage Data Across Different Storage Systems
• Storage medium: MEM
• Clusters in different regions
An Internet Company
Machine Learning Pipelines
42 42
Outline
• What is Alluxio • Example use cases
1. Off-heap memory to alleviate resource pressure 2. Fast data sharing between jobs 3. Accelerate access to remote storage 4. Unified namespace
• Summary
43
Takeaway: When to use Alluxio
• Two or more jobs access the same dataset • Job(s) may not always succeed • Spark with memory/GC pressure • Jobs/Apps are pipelined • Resulting data does not need to be immediately
persisted • Interact with remote data • Data stored across different storage systems
44
Try out Alluxio 1.2.0 http://www.alluxio.org/releases
45
First China Meetups!
• Nanjing (Saturday, 7/30)
• Shanghai (Sunday, 7/31)
• Beijing (Sunday, 8/7)
46
Resources • Alluxio Project: http://www.alluxio.org
• Development: https://github.com/Alluxio/alluxio
• Meet Friends: http://www.meetup.com/Alluxio
• Alluxio, Inc.: http://www.alluxio.com
• Training: http://www.alluxio.com/alluxio-training/
• Contact us: [email protected]
• Alluxio Wechat: Alluxio 47