GraphLab: A New Framework for Parallel Machine Learning YUCHENG LOW, JOSEPH GONAZLEZ, AAPO KYROLA, DANNY BICKSON, CARLOS GUESTRIN, JOSEPH M. HELLERSTEIN PRESENTED BY: STEPHEN MACKE

GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Download PDF Report

Upload
others
View
4
Download
0

Embed Size (px)

Citation preview

Page 1: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

GraphLab: A New Framework for Parallel Machine LearningYUCHENG LOW, JOSEPH GONAZLEZ, AAPO KYROLA, DANNY BICKSON, CARLOS GUESTRIN, JOSEPH M. HELLERSTEIN

PRESENTED BY: STEPHEN MACKE

Page 2: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

The Problem•Need to perform machine learning at scale•Hardware advances hitting power wall•Need to parallelize (multicore + distributed)•But parallelization is hard!

Page 3: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Low-‐level approaches•Pthreads•MPI•(OpenMP)•(CUDA, SSE instructions)

Page 4: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Solution: MapReduce?•Only works well for embarassingly parallel problems

•Iterative MapReduce typically requires global synchronization at a single reducer

Page 5: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Other Abstractions•Systolic, Dataflow• Nodes vertices, edges communication links• Bulk synchronous – bad for “Gauss-‐Seidel”-‐style schedules

•DAG Abstraction (Dryad, Pig Latin)• Dataflow graph depends on # iterations•Must be specified in advance – not suitable for iterative computation

•Graph-‐based messaging (e.g. Pregel)• Require users to explicitly manage communication

Page 6: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Digression: Jacobi vs. Gauss-‐Seidel schedules

Page 7: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Digression: Jacobi vs. Gauss-‐Seidel schedules

Page 8: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Digression: Jacobi vs. Gauss-‐Seidel schedules

Page 9: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

The Main Idea

Perform iterative updates in this “Gauss-‐Seidel” fashion, asynchronously.

Page 10: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Other Abstractions (cont’d)

Page 11: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

GraphLab Components

Page 12: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Data Graph

Page 13: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

User Update Functions

Page 14: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Sync Operation•“Fold” operation aggregates over vertices• Analogous to “Reduce” from MapReduce

•“Merge” operation for parallel tree reduction• Analogous to “Combine” from MapReduce

•“Apply” finalizes value (e.g. rescaling)

Page 15: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Let’s write Update for Pagerank!•Reminder: pagerank eqns

Page 16: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Let’s write Update for Pagerank!

Page 17: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Scope Rules

Page 18: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Scope Rules

Q. What consistency model do we need for pagerank?

Page 19: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Scheduling•Synchronous•Round-‐robin•Set scheduler

•Splash scheduler

Page 20: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Set Scheduling via Graph Coloring•For certain consistency models (q. which ones?), we can update all vertices of same color in parallel!

Ex. Gibbs sampling

Page 21: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Case Studies•MRF Parameter learning•Gibbs sampling•Co-‐Em•Lasso / Shooting•Compressed Sensing

Page 22: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Results (MRF parameter learning)

Page 23: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Results (Gibbs sampling, LBP)

Page 24: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Results (Co-‐Em)

Page 25: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Results (Lasso)

Page 26: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Results (Compressed Sensing)

Page 27: GraphLab:ANew) Framework)for)Parallel) Machine)Learningi.stanford.edu/~adityagp/courses/cs598/slides/graphlab.pdf · GraphLab:ANew) Framework)for)Parallel) Machine)Learning YUCHENGLOW,)JOSEPH)GONAZLEZ,

Conclusion•Does not say anything new about parallel programming fundamentals

•New interesting abstraction for certain types of problems• Shows need for rethinking existing abstractions (e.g. MR)• (particularly for iterative / ML algorithms)

•Commercialized as GraphLab Create• But mostly for toolkit / viz purposes• Actual mucking with update / sync functions somewhat niche• Open sourced in PowerGraph, GraphChi, etc.