Concurrency Control for Parallel Machine Learning

Concurrency Control for

Parallel Machine Learning

Dimitris Papailiopoulos

Xinghao Pan, Joseph Gonzalez, Stefanie Jegelka, Tamara Broderick, Dimitris Papailiopoulos, Joseph Bradley, Michael I. Jordan

Data

Model

State

Serial Inference

Model

State

Parallel Inference

Processor 1

Processor 2

Data

Data

Model

State

Parallel Inference

Processor 1

Processor 2

Concurrency:more machines = less time

Correctness:serial equivalence

?

Data

Model

State

Coordination Free Parallel

Inference

Processor 1

Processor 2

Ignore collisions

?

Concurrency:(almost) free

+

Speedup = #CPU

Correctness?Not always...

Correctness

Serial

Low High

Correctness

Concurrency

Coordination-

free

Serial

High

Low High

Low

Correctness

Concurrency

Coordination-

free

Serial

High

Low High

Low

Concurrency

ControlDatabase mechanismso Guarantee correctnesso Maximize concurrency Mutual exclusion Optimistic CC

Data

Model

State

Mutual Exclusion Through

Locking

Processor 1

Processor 2

Introduce locking (scheduling) protocols to prevent

conflicts.

Data

Model

State

Processor 1

Processor 2

✗

Enforce local serialization to avoid conflicts.

Mutual Exclusion Through

Locking

Data

Model

State

Optimistic Concurrency Control

Processor 1

Processor 2

Allow computation to proceed without blocking.

Kung & Robinson. On optimistic methods for concurrency

control.

ACM Transactions on Database Systems 1981

Data

Model

State

Optimistic Concurrency Control

Processor 1

Processor 2

✗ ✗

Validate potential conflicts.

Invalid Outcome

Kung & Robinson. On optimistic methods for concurrency

control.

ACM Transactions on Database Systems 1981

Data

Model

State

Optimistic Concurrency Control

Processor 1

Processor 2

✗ ✗Rollback and Redo

Take a compensating action.

Kung & Robinson. On optimistic methods for concurrency

control.

ACM Transactions on Database Systems 1981

Concurrency Control

14

Coordination Free:

Provably fast and correct under key assumptions.

Concurrency Control:

Provably correct and fast under key assumptions.

Systems Ideas to

Improve Efficiency

Clusteri

ng

Online

Facility

Location

Submodular

MaximizationSubset selection, diminishing

marginal gainsMax Graph

Cut

Set

Cover

Sensor Placement

Social Network

Influence

Propagation

Document

Summarization

Machine Learning + Concurrency

Control(Xinghao Pan et al.)

SportsFootball

Word Series

Giants

Cardinals

PoliticsMidterm

Obama

Democrat

Tea

FinanceQE

market

interest

Dow

Topic Modelling

Correlation

Clustering

Community

Detection

Deduplication

Clusteri

ng

Online

Facility

Location

Submodular

MaximizationSubset selection, diminishing

marginal gainsMax Graph

Cut

Set

Cover

Sensor Placement

Social Network

Influence

Propagation

Document

Summarization

SportsFootball

Word Series

Giants

Cardinals

PoliticsMidterm

Obama

Democrat

Tea

FinanceQE

market

interest

Dow

Topic Modelling

Correlation

Clustering

Community

Detection

Deduplication

Serial ML

algorithm

Sequence of

transactions

Identify potential

conflicts

Apply Concurrency

Control

mechanisms

Parallel ML

algorithm

Machine Learning + Concurrency

Control(Xinghao Pan et al.)

Application: Deduplication

Computer Science

Division – University of

California Berkeley CA

University of California at Berkeley

Department of

Physics Stanford

University California

Lawrence Berkeley National

Labs <ref>California</ref>

Computer Science

Division – University of

California Berkeley CA

University of California at Berkeley

Department of

Physics Stanford

University California

Lawrence Berkeley National

Labs <ref>California</ref>

Application: Deduplication

Serial Correlation Clustering

Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.

Journal of the ACM (JACM), 55(5):23, 2008.

Serially process vertices

Serial Correlation Clustering

Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.

Journal of the ACM (JACM), 55(5):23, 2008.

Serially process vertices

Serial Correlation Clustering

Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.

Journal of the ACM (JACM), 55(5):23, 2008.

Serially process vertices

Serial Correlation Clustering

Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.

Journal of the ACM (JACM), 55(5):23, 2008.

Serially process vertices

Serial Correlation Clustering

Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.

Journal of the ACM (JACM), 55(5):23, 2008.

Serially process vertices

Approximation 3 OPT (in expectation)

Parallel Correlation Clustering

Concurrency Control Correlation Clustering

(C4)Parallel Correlation ClusteringCannot introduce

adjacent cluster

centers

Resolve by

Mutual Exclusion

Concurrency Control Correlation Clustering

(C4)Common neighbor must be

assigned to earliest center

Resolve by

Optimistic Concurrency

Control

ResolutionAssign common neighbor to earliest cluster

Optimistic AssumptionNo other new cluster created

?

Properties of C4(Concurrency Control Correlation Clustering)

Theorem: C4 is correct.

C4 preserves same guarantees as serial algorithm (3

OPT).Co

rre

ctn

es

sC

on

cu

rren

cy

Theorem: C4 has provably small overheads.

= almost linear speedupExpected #blocked transactions < 2τ |E| / |V|.

τ ≡ diff in parallel cpu’s progress

Empirical Validation on Billion Edge

Graphs

Amazon EC2 r3.8xlarge instances

Multicore up to 16 threads

Real and synthetic graphs

100 runs (10 random orderings x 10 runs)

Graph Vertices Edges

IT-2004 Italian web-graph 41 Million 1.14 Billion

Webbase-2001 WebBase crawl 118 Million 1.02 Billion

Erdos-Renyi Synthetic random 100 Million ≈ 1.0 Billion

C4: Cost of Coordination

< 0.02% blocked

C4: Speed-up

Ideal

10x

speedu

p

ConclusionConcurrency Control

for Parallel ML

o Guarantee

correctness

o Maximize

concurrency

Code release in the works!

https://amplab.cs.berkeley.edu/projects/cc

ml/

xinghao@berkeley.edu

ApplicationsCorrelation Clustering

Submodular Maximization

Clustering

Online Facility Location

Feature Modeling