The Missing Piece of On-Demand Clusters

The Missing Piece of On-Demand Clusters

October 2016

Calvin Jia

About Me

• Calvin Jia

• Software Engineer @ Alluxio, Inc.

• Alluxio PMC

• #1 Alluxio Contributor

• Twitter: @JiaCalvin

2

Outline

• Cloud Architectures• Alluxio in Cloud Architectures• Takeaways

3

Cloud Architectures – Overview

• Mostly service based, from providers– Amazon Web Services– Google Cloud Platform

• Separate compute and storage clusters• Compute clusters are ephemeral

4

Cloud Architectures – Pros & Cons

Pros• Low maintenance • Pay as you go• Elastic and scalable• Cost effective storage

Cons• Lower Performance

5

Outline

• Cloud Architectures• Alluxio in Cloud Architectures• Takeaways

6

Alluxio in Cloud Architectures – Overview

• Deployed in compute clusters– Memory speed data access– Transparent data access to any storage

• Simple to deploy– Mount storage systems similar to local disks

7

Alluxio in Cloud Architectures – Example

8

Alluxio in Cloud Architectures – Benefits

9

• Remedies the performance drawback• Acceleration due to memory-speed I/O• Designed to improve the affinity of compute

and storage

Alluxio in Cloud Architectures – Data Path

10

FAST 104 - 105 MB/s

MODERATE 103 - 104 MB/s

SLOW 102 - 103 MB/s

Only when necessaryLimited

Often

SSDHDD

Mem

Takeaways – Experiment Results

11

0

100

200

300

400

500

600

Spark - No PersistSpark - PersistAlluxio

Runti

me

(sec

onds

)

Takeaways – Alluxio & Cloud Architectures

12

• Cloud architectures have significant upsides• Alluxio alleviates the major downsides

Try out Alluxio 1.2.0http://www.alluxio.org/releases

13

14

Read the blog at http://www.alluxio.com/2016/08/accelerating-on-demand-data-analytics-with-alluxio

Read the whitepaper at www.alluxio.com.

Check out www.alluxio.org.

Thank you!