11

Scaling UpScaling UpData Intensive ScienceData Intensive Science

to Campus Gridsto Campus Grids

Douglas ThainDouglas Thain

Clemson UniversityClemson University

25 Septmber 200925 Septmber 2009

22

http://www.cse.nd.edu/~ccl

33

The Cooperative Computing LabThe Cooperative Computing Lab

We We collaborate with peoplecollaborate with people who have large who have large scale computing problems.scale computing problems.We We build new softwarebuild new software and systems to help and systems to help them achieve meaningful goals.them achieve meaningful goals.We run a We run a production computing systemproduction computing system used by people at ND and elsewhere.used by people at ND and elsewhere.We conduct We conduct computer science researchcomputer science research, , informed by real world experience, with an informed by real world experience, with an impact upon problems that matter.impact upon problems that matter.

44

What is a Campus Grid?What is a Campus Grid?

A campus grid is an aggregation of all A campus grid is an aggregation of all available computing power found in an available computing power found in an institution:institution:– Idle cycles from desktop machines.Idle cycles from desktop machines.– Unused cycles from dedicated clusters.Unused cycles from dedicated clusters.

Examples of campus grids:Examples of campus grids:– 700 CPUs at the University of Notre Dame700 CPUs at the University of Notre Dame– 9000-11,000 CPUs at Clemson University9000-11,000 CPUs at Clemson University– 20,000 CPUs at Purdue University20,000 CPUs at Purdue University

55

Provides robust batch queueing on a Provides robust batch queueing on a complex distributed system.complex distributed system.

Resource owners control consumption:Resource owners control consumption:– ““Only run jobs on this machine at night.”Only run jobs on this machine at night.”– ““Prefer biology jobs over physics jobs.”Prefer biology jobs over physics jobs.”

End users express needs:End users express needs:– ““Only run this job where RAM>2GB”Only run this job where RAM>2GB”– ““Prefer to run on machinesPrefer to run on machines

http://www.cs.wisc.edu/condor

66

77

88

99

1010

1111

Clusters, clouds, and gridsClusters, clouds, and gridsgive us access to unlimited CPUs.give us access to unlimited CPUs.

How do we write How do we write programsprograms that can that canrun effectively in large systems?run effectively in large systems?

1212

Example: Biometrics ResearchExample: Biometrics Research

Goal: Design robust face comparison function.Goal: Design robust face comparison function.

F

0.05

F

0.97

1313

Similarity Matrix ConstructionSimilarity Matrix Construction

1.0 0.8 0.1 0.0 0.0 0.1

1.0 0.0 0.1 0.1 0.0

1.0 0.0 0.1 0.3

1.0 0.0 0.0

1.0 0.1

1.0

Challenge Workload:

60,000 iris images1MB each.02s per F833 CPU-days600 TB of I/O

1414

I have 60,000 iris images acquired in my research lab. I want to reduce each one to a feature space, and then compare all of them to each other. I want to spend my time doing science, not struggling with computers.

I have a laptop.

I own a few machines I can buy time from Amazon or TeraGrid.

Now What?

1515

Non-Expert User Using 500 CPUsNon-Expert User Using 500 CPUsTry 1: Each F is a batch job.Failure: Dispatch latency >> F runtime.

HN

CPU CPU CPU CPUF F F FCPUF

Try 2: Each row is a batch job.Failure: Too many small ops on FS.

HN

CPU CPU CPU CPUF F F FCPUFFFF FF

FFFF

FFF

FFF

Try 3: Bundle all files into one package.Failure: Everyone loads 1GB at once.

HN

CPU CPU CPU CPUF F F FCPUFFFF FF

FFFF

FFF

FFF

Try 4: User gives up and attemptsto solve an easier or smaller problem.

1616

ObservationObservation

In a given field of study, many people In a given field of study, many people repeat the same repeat the same pattern of work many of work many times, making slight changes to the data times, making slight changes to the data and algorithms.and algorithms.If the system knows the overall pattern in If the system knows the overall pattern in advance, then it can do a better job of advance, then it can do a better job of executing it reliably and efficiently.executing it reliably and efficiently.If the user knows in advance what patterns If the user knows in advance what patterns are allowed, then they have a better idea are allowed, then they have a better idea of how to construct their workloads.of how to construct their workloads.

1717

AbstractionsAbstractionsfor Distributed Computingfor Distributed Computing

Abstraction: a Abstraction: a declarative specificationdeclarative specification of the computation and data of a workload.of the computation and data of a workload.

A A restricted patternrestricted pattern, not meant to be a , not meant to be a general purpose programming language.general purpose programming language.

UsesUses data structures instead of files. instead of files.

Provide users with a Provide users with a bright path..

Regular structure makes it tractable to Regular structure makes it tractable to model and model and predict performance.predict performance.

1818

Working with AbstractionsWorking with AbstractionsF

A1A2

An

AllPairs( A, B, F )

Cloud or Grid

A1A2

Bn

CustomWorkflow

Engine

Compact Data Structure

1919

All-Pairs AbstractionAll-Pairs Abstraction

AllPairs( set A, set B, function F )AllPairs( set A, set B, function F )

returns matrix M wherereturns matrix M where

M[i][j] = F( A[i], B[j] ) for all i,jM[i][j] = F( A[i], B[j] ) for all i,j

B1

B2

B3

A1 A2 A3

F F F

A1A1

An

B1B1

Bn

F

AllPairs(A,B,F)F

F F

F F

F

allpairs A B F.exe

2020

How Does the Abstraction Help?How Does the Abstraction Help?

The custom workflow engine:The custom workflow engine:– Chooses right data transfer strategy.Chooses right data transfer strategy.– Chooses the right number of resources.Chooses the right number of resources.– Chooses blocking of functions into jobs.Chooses blocking of functions into jobs.– Recovers from a larger number of failures.Recovers from a larger number of failures.– Predicts overall runtime accurately.Predicts overall runtime accurately.

All of these tasks are nearly impossible for All of these tasks are nearly impossible for arbitrary workloads, but are tractable (not arbitrary workloads, but are tractable (not trivial) to solve for a trivial) to solve for a specificspecific abstraction. abstraction.

2121

2222

Choose the Right # of CPUsChoose the Right # of CPUs

2323

Resources ConsumedResources Consumed

2424

All-Pairs in ProductionAll-Pairs in ProductionOur All-Pairs implementation has provided over Our All-Pairs implementation has provided over 57 CPU-years57 CPU-years of computation to the ND of computation to the ND biometrics research group over the last year.biometrics research group over the last year.

Largest run so far: Largest run so far: 58,396 irises58,396 irises from the Face from the Face Recognition Grand Challenge. The largest Recognition Grand Challenge. The largest experiment ever run on publically available data.experiment ever run on publically available data.

Competing biometric research relies on samples Competing biometric research relies on samples of 100-1000 images, which can miss important of 100-1000 images, which can miss important population effects. population effects.

Reduced computation time from 833 days to 10 Reduced computation time from 833 days to 10 days, making it feasible to repeat multiple times for days, making it feasible to repeat multiple times for a graduate thesis. (We can go faster yet.)a graduate thesis. (We can go faster yet.)

2525

2626

2727

Are there other abstractions?Are there other abstractions?

2828

All-Pairs AbstractionAll-Pairs Abstraction

AllPairs( set A, set B, function F )AllPairs( set A, set B, function F )

returns matrix M wherereturns matrix M where

M[i][j] = F( A[i], B[j] ) for all i,jM[i][j] = F( A[i], B[j] ) for all i,j

B1

B2

B3

A1 A2 A3

F F F

A1A1

An

B1B1

Bn

F

AllPairs(A,B,F)F

F F

F F

F

allpairs A B F.exe

2929

M[4,2]

M[3,2] M[4,3]

M[4,4]M[3,4]M[2,4]

M[4,0]M[3,0]M[2,0]M[1,0]M[0,0]

M[0,1]

M[0,2]

M[0,3]

M[0,4]

Fx

yd

Fx

yd

Fx

yd

Fx

yd

Fx

yd

Fx

yd

F

F

y

y

x

x

d

d

x F Fx

yd yd

Wavefront( matrix M, function F(x,y,d) )Wavefront( matrix M, function F(x,y,d) )

returns matrix M such thatreturns matrix M such that

M[i,j] = F( M[i-1,j], M[I,j-1], M[i-1,j-1] )M[i,j] = F( M[i-1,j], M[I,j-1], M[i-1,j-1] )

F

Wavefront(M,F)

M

3030

Some-Pairs AbstractionSome-Pairs Abstraction

SomePairs( set A, list (i,j), function F(x,y) )SomePairs( set A, list (i,j), function F(x,y) )

returnsreturns

list of F( A[i], A[j] )list of F( A[i], A[j] )

A1

A2

A3

A1 A2 A3

F

A1A1

An

F

SomePairs(A,L,F)F F

F

(1,2)(2,1)(2,3)(3,3)

3131

What if your application doesn’t What if your application doesn’t fit a regular pattern?fit a regular pattern?

3232

MakeflowMakeflow

part1 part2 part3: input.data split.py ./split.py input.data

out1: part1 mysim.exe ./mysim.exe part1 >out1

out2: part2 mysim.exe ./mysim.exe part2 >out2

out3: part3 mysim.exe ./mysim.exe part3 >out3

result: out1 out2 out3 join.py ./join.py out1 out2 out3 > result

3333

worker

workerworker

workerworker

workerworker

workqueue

afile bfile

put progput afileexec prog afile > bfileget bfile

100s of workersdispatched to

the cloud

makeflowmaster

queuetasks

tasksdone

prog

detail of a single worker:

Makeflow ImplementationMakeflow Implementation

bfile: afile prog prog afile >bfile

Two optimizations: Cache inputs and output. Dispatch tasks to nodes with data.

3434

Experience with MakeflowExperience with Makeflow

Reusing a good old idea in a new way.Reusing a good old idea in a new way.

Easy to test and debug on a desktop machine or Easy to test and debug on a desktop machine or a multicore server.a multicore server.

The workload says The workload says nothingnothing about the distributed about the distributed system. (This is good.)system. (This is good.)

Graduate students in bioinformatics running Graduate students in bioinformatics running codes at production speeds on hundreds of codes at production speeds on hundreds of nodes in less than a week.nodes in less than a week.

Student from Clemson got complex biometrics Student from Clemson got complex biometrics workload running in a few weeks. workload running in a few weeks.

3535

Putting it All TogetherPutting it All Together

WebPortal

DataRepository

Campus Grid

FY

ZX

Abstraction

3636

BXGrid SchemaBXGrid Schema

fileid = 24305size = 300Ktype = jpgsum = abc123…

replicaid=423state=ok

replicaid=105state=ok

replicaid=293state=creating

replicaid=102state=deleting

TypeType SubjectSubject EyeEye ColorColor FileIDFileID

IrisIris S100S100 RightRight BlueBlue 1048610486

IrisIris S100S100 LeftLeft BlueBlue 1048710487

IrisIris S203S203 RightRight BrownBrown 2430424304

IrisIris S203S203 LeftLeft BrownBrown 2430524305

Scientific Metadata

General Metadata

ImmutableReplicas

3737

3838

3939

4040

Results from Campus GridResults from Campus Grid

4141

BiocomputeBiocompute

4242

4343

Parallel BLAST MakeflowParallel BLAST Makeflow

4444

4545

Abstractions as a Social ToolAbstractions as a Social Tool

Collaboration with outside groups is how we Collaboration with outside groups is how we encounter the most interesting, challenging, and encounter the most interesting, challenging, and important problems, in computer science.important problems, in computer science.However, often neither side understands which However, often neither side understands which details are essential or non-essential:details are essential or non-essential:– Can you deal with files that have upper case letters?Can you deal with files that have upper case letters?– Oh, by the way, we have 10TB of input, is that ok?Oh, by the way, we have 10TB of input, is that ok?– (A little bit of an exaggeration.)(A little bit of an exaggeration.)

An abstraction is an excellent chalkboard tool:An abstraction is an excellent chalkboard tool:– Accessible to anyone with a little bit of mathematics.Accessible to anyone with a little bit of mathematics.– Makes it easy to see what must be plugged in.Makes it easy to see what must be plugged in.– Forces out essential details: data size, execution time.Forces out essential details: data size, execution time.

4646

ConclusionConclusion

Grids, clouds, and clusters provide enormous Grids, clouds, and clusters provide enormous computing power, but are very challenging to computing power, but are very challenging to use effectively.use effectively.

An abstraction provides a robust, scalable An abstraction provides a robust, scalable solution to a narrow category of problems; each solution to a narrow category of problems; each requires different kinds of optimizations.requires different kinds of optimizations.

Limiting expressive powerLimiting expressive power, results in systems , results in systems that are usable, predictable, and reliable.that are usable, predictable, and reliable.

Portal + Repository + Abstraction + GridPortal + Repository + Abstraction + Grid

= New Science Capabilities= New Science Capabilities

4747

AcknowledgmentsAcknowledgmentsCooperative Computing LabCooperative Computing Lab– http://www.cse.nd.edu/~cclhttp://www.cse.nd.edu/~ccl

Grad StudentsGrad Students– Chris MorettiChris Moretti– Hoang Bui Hoang Bui – Li YuLi Yu– Mike OlsonMike Olson– Michael AlbrechtMichael Albrecht

Faculty:Faculty:– Patrick FlynnPatrick Flynn– Nitesh ChawlaNitesh Chawla– Kenneth JuddKenneth Judd– Scott EmrichScott Emrich

NSF Grants CCF-0621434NSF Grants CCF-0621434

and CNS-0643229and CNS-0643229

UndergradsUndergrads– Mike KellyMike Kelly– Rory CarmichaelRory Carmichael– Mark PasquierMark Pasquier– Christopher LyonChristopher Lyon– Jared BulosanJared Bulosan– Kameron SrimoungachKameron Srimoungach– Rachel WittyRachel Witty– Ryan JansenRyan Jansen– Joey RichJoey Rich