Upload
yolanda-sullivan
View
47
Download
0
Embed Size (px)
DESCRIPTION
Scaling Up Data Intensive Science to Campus Grids. Douglas Thain Clemson University 25 Septmber 2009. http://www.cse.nd.edu/~ccl. The Cooperative Computing Lab. We collaborate with people who have large scale computing problems. - PowerPoint PPT Presentation
Citation preview
11
Scaling UpScaling UpData Intensive ScienceData Intensive Science
to Campus Gridsto Campus Grids
Douglas ThainDouglas Thain
Clemson UniversityClemson University
25 Septmber 200925 Septmber 2009
22
http://www.cse.nd.edu/~ccl
33
The Cooperative Computing LabThe Cooperative Computing Lab
We We collaborate with peoplecollaborate with people who have large who have large scale computing problems.scale computing problems.We We build new softwarebuild new software and systems to help and systems to help them achieve meaningful goals.them achieve meaningful goals.We run a We run a production computing systemproduction computing system used by people at ND and elsewhere.used by people at ND and elsewhere.We conduct We conduct computer science researchcomputer science research, , informed by real world experience, with an informed by real world experience, with an impact upon problems that matter.impact upon problems that matter.
44
What is a Campus Grid?What is a Campus Grid?
A campus grid is an aggregation of all A campus grid is an aggregation of all available computing power found in an available computing power found in an institution:institution:– Idle cycles from desktop machines.Idle cycles from desktop machines.– Unused cycles from dedicated clusters.Unused cycles from dedicated clusters.
Examples of campus grids:Examples of campus grids:– 700 CPUs at the University of Notre Dame700 CPUs at the University of Notre Dame– 9000-11,000 CPUs at Clemson University9000-11,000 CPUs at Clemson University– 20,000 CPUs at Purdue University20,000 CPUs at Purdue University
55
Provides robust batch queueing on a Provides robust batch queueing on a complex distributed system.complex distributed system.
Resource owners control consumption:Resource owners control consumption:– ““Only run jobs on this machine at night.”Only run jobs on this machine at night.”– ““Prefer biology jobs over physics jobs.”Prefer biology jobs over physics jobs.”
End users express needs:End users express needs:– ““Only run this job where RAM>2GB”Only run this job where RAM>2GB”– ““Prefer to run on machinesPrefer to run on machines
http://www.cs.wisc.edu/condor
66
77
88
99
1010
1111
Clusters, clouds, and gridsClusters, clouds, and gridsgive us access to unlimited CPUs.give us access to unlimited CPUs.
How do we write How do we write programsprograms that can that canrun effectively in large systems?run effectively in large systems?
1212
Example: Biometrics ResearchExample: Biometrics Research
Goal: Design robust face comparison function.Goal: Design robust face comparison function.
F
0.05
F
0.97
1313
Similarity Matrix ConstructionSimilarity Matrix Construction
1.0 0.8 0.1 0.0 0.0 0.1
1.0 0.0 0.1 0.1 0.0
1.0 0.0 0.1 0.3
1.0 0.0 0.0
1.0 0.1
1.0
Challenge Workload:
60,000 iris images1MB each.02s per F833 CPU-days600 TB of I/O
1414
I have 60,000 iris images acquired in my research lab. I want to reduce each one to a feature space, and then compare all of them to each other. I want to spend my time doing science, not struggling with computers.
I have a laptop.
I own a few machines I can buy time from Amazon or TeraGrid.
Now What?
1515
Non-Expert User Using 500 CPUsNon-Expert User Using 500 CPUsTry 1: Each F is a batch job.Failure: Dispatch latency >> F runtime.
HN
CPU CPU CPU CPUF F F FCPUF
Try 2: Each row is a batch job.Failure: Too many small ops on FS.
HN
CPU CPU CPU CPUF F F FCPUFFFF FF
FFFF
FFF
FFF
Try 3: Bundle all files into one package.Failure: Everyone loads 1GB at once.
HN
CPU CPU CPU CPUF F F FCPUFFFF FF
FFFF
FFF
FFF
Try 4: User gives up and attemptsto solve an easier or smaller problem.
1616
ObservationObservation
In a given field of study, many people In a given field of study, many people repeat the same repeat the same pattern of work many of work many times, making slight changes to the data times, making slight changes to the data and algorithms.and algorithms.If the system knows the overall pattern in If the system knows the overall pattern in advance, then it can do a better job of advance, then it can do a better job of executing it reliably and efficiently.executing it reliably and efficiently.If the user knows in advance what patterns If the user knows in advance what patterns are allowed, then they have a better idea are allowed, then they have a better idea of how to construct their workloads.of how to construct their workloads.
1717
AbstractionsAbstractionsfor Distributed Computingfor Distributed Computing
Abstraction: a Abstraction: a declarative specificationdeclarative specification of the computation and data of a workload.of the computation and data of a workload.
A A restricted patternrestricted pattern, not meant to be a , not meant to be a general purpose programming language.general purpose programming language.
UsesUses data structures instead of files. instead of files.
Provide users with a Provide users with a bright path..
Regular structure makes it tractable to Regular structure makes it tractable to model and model and predict performance.predict performance.
1818
Working with AbstractionsWorking with AbstractionsF
A1A2
An
AllPairs( A, B, F )
Cloud or Grid
A1A2
Bn
CustomWorkflow
Engine
Compact Data Structure
1919
All-Pairs AbstractionAll-Pairs Abstraction
AllPairs( set A, set B, function F )AllPairs( set A, set B, function F )
returns matrix M wherereturns matrix M where
M[i][j] = F( A[i], B[j] ) for all i,jM[i][j] = F( A[i], B[j] ) for all i,j
B1
B2
B3
A1 A2 A3
F F F
A1A1
An
B1B1
Bn
F
AllPairs(A,B,F)F
F F
F F
F
allpairs A B F.exe
2020
How Does the Abstraction Help?How Does the Abstraction Help?
The custom workflow engine:The custom workflow engine:– Chooses right data transfer strategy.Chooses right data transfer strategy.– Chooses the right number of resources.Chooses the right number of resources.– Chooses blocking of functions into jobs.Chooses blocking of functions into jobs.– Recovers from a larger number of failures.Recovers from a larger number of failures.– Predicts overall runtime accurately.Predicts overall runtime accurately.
All of these tasks are nearly impossible for All of these tasks are nearly impossible for arbitrary workloads, but are tractable (not arbitrary workloads, but are tractable (not trivial) to solve for a trivial) to solve for a specificspecific abstraction. abstraction.
2121
2222
Choose the Right # of CPUsChoose the Right # of CPUs
2323
Resources ConsumedResources Consumed
2424
All-Pairs in ProductionAll-Pairs in ProductionOur All-Pairs implementation has provided over Our All-Pairs implementation has provided over 57 CPU-years57 CPU-years of computation to the ND of computation to the ND biometrics research group over the last year.biometrics research group over the last year.
Largest run so far: Largest run so far: 58,396 irises58,396 irises from the Face from the Face Recognition Grand Challenge. The largest Recognition Grand Challenge. The largest experiment ever run on publically available data.experiment ever run on publically available data.
Competing biometric research relies on samples Competing biometric research relies on samples of 100-1000 images, which can miss important of 100-1000 images, which can miss important population effects. population effects.
Reduced computation time from 833 days to 10 Reduced computation time from 833 days to 10 days, making it feasible to repeat multiple times for days, making it feasible to repeat multiple times for a graduate thesis. (We can go faster yet.)a graduate thesis. (We can go faster yet.)
2525
2626
2727
Are there other abstractions?Are there other abstractions?
2828
All-Pairs AbstractionAll-Pairs Abstraction
AllPairs( set A, set B, function F )AllPairs( set A, set B, function F )
returns matrix M wherereturns matrix M where
M[i][j] = F( A[i], B[j] ) for all i,jM[i][j] = F( A[i], B[j] ) for all i,j
B1
B2
B3
A1 A2 A3
F F F
A1A1
An
B1B1
Bn
F
AllPairs(A,B,F)F
F F
F F
F
allpairs A B F.exe
2929
M[4,2]
M[3,2] M[4,3]
M[4,4]M[3,4]M[2,4]
M[4,0]M[3,0]M[2,0]M[1,0]M[0,0]
M[0,1]
M[0,2]
M[0,3]
M[0,4]
Fx
yd
Fx
yd
Fx
yd
Fx
yd
Fx
yd
Fx
yd
F
F
y
y
x
x
d
d
x F Fx
yd yd
Wavefront( matrix M, function F(x,y,d) )Wavefront( matrix M, function F(x,y,d) )
returns matrix M such thatreturns matrix M such that
M[i,j] = F( M[i-1,j], M[I,j-1], M[i-1,j-1] )M[i,j] = F( M[i-1,j], M[I,j-1], M[i-1,j-1] )
F
Wavefront(M,F)
M
3030
Some-Pairs AbstractionSome-Pairs Abstraction
SomePairs( set A, list (i,j), function F(x,y) )SomePairs( set A, list (i,j), function F(x,y) )
returnsreturns
list of F( A[i], A[j] )list of F( A[i], A[j] )
A1
A2
A3
A1 A2 A3
F
A1A1
An
F
SomePairs(A,L,F)F F
F
(1,2)(2,1)(2,3)(3,3)
3131
What if your application doesn’t What if your application doesn’t fit a regular pattern?fit a regular pattern?
3232
MakeflowMakeflow
part1 part2 part3: input.data split.py ./split.py input.data
out1: part1 mysim.exe ./mysim.exe part1 >out1
out2: part2 mysim.exe ./mysim.exe part2 >out2
out3: part3 mysim.exe ./mysim.exe part3 >out3
result: out1 out2 out3 join.py ./join.py out1 out2 out3 > result
3333
worker
workerworker
workerworker
workerworker
workqueue
afile bfile
put progput afileexec prog afile > bfileget bfile
100s of workersdispatched to
the cloud
makeflowmaster
queuetasks
tasksdone
prog
detail of a single worker:
Makeflow ImplementationMakeflow Implementation
bfile: afile prog prog afile >bfile
Two optimizations: Cache inputs and output. Dispatch tasks to nodes with data.
3434
Experience with MakeflowExperience with Makeflow
Reusing a good old idea in a new way.Reusing a good old idea in a new way.
Easy to test and debug on a desktop machine or Easy to test and debug on a desktop machine or a multicore server.a multicore server.
The workload says The workload says nothingnothing about the distributed about the distributed system. (This is good.)system. (This is good.)
Graduate students in bioinformatics running Graduate students in bioinformatics running codes at production speeds on hundreds of codes at production speeds on hundreds of nodes in less than a week.nodes in less than a week.
Student from Clemson got complex biometrics Student from Clemson got complex biometrics workload running in a few weeks. workload running in a few weeks.
3535
Putting it All TogetherPutting it All Together
WebPortal
DataRepository
Campus Grid
FY
ZX
Abstraction
3636
BXGrid SchemaBXGrid Schema
fileid = 24305size = 300Ktype = jpgsum = abc123…
replicaid=423state=ok
replicaid=105state=ok
replicaid=293state=creating
replicaid=102state=deleting
TypeType SubjectSubject EyeEye ColorColor FileIDFileID
IrisIris S100S100 RightRight BlueBlue 1048610486
IrisIris S100S100 LeftLeft BlueBlue 1048710487
IrisIris S203S203 RightRight BrownBrown 2430424304
IrisIris S203S203 LeftLeft BrownBrown 2430524305
Scientific Metadata
General Metadata
ImmutableReplicas
3737
3838
3939
4040
Results from Campus GridResults from Campus Grid
4141
BiocomputeBiocompute
4242
4343
Parallel BLAST MakeflowParallel BLAST Makeflow
4444
4545
Abstractions as a Social ToolAbstractions as a Social Tool
Collaboration with outside groups is how we Collaboration with outside groups is how we encounter the most interesting, challenging, and encounter the most interesting, challenging, and important problems, in computer science.important problems, in computer science.However, often neither side understands which However, often neither side understands which details are essential or non-essential:details are essential or non-essential:– Can you deal with files that have upper case letters?Can you deal with files that have upper case letters?– Oh, by the way, we have 10TB of input, is that ok?Oh, by the way, we have 10TB of input, is that ok?– (A little bit of an exaggeration.)(A little bit of an exaggeration.)
An abstraction is an excellent chalkboard tool:An abstraction is an excellent chalkboard tool:– Accessible to anyone with a little bit of mathematics.Accessible to anyone with a little bit of mathematics.– Makes it easy to see what must be plugged in.Makes it easy to see what must be plugged in.– Forces out essential details: data size, execution time.Forces out essential details: data size, execution time.
4646
ConclusionConclusion
Grids, clouds, and clusters provide enormous Grids, clouds, and clusters provide enormous computing power, but are very challenging to computing power, but are very challenging to use effectively.use effectively.
An abstraction provides a robust, scalable An abstraction provides a robust, scalable solution to a narrow category of problems; each solution to a narrow category of problems; each requires different kinds of optimizations.requires different kinds of optimizations.
Limiting expressive powerLimiting expressive power, results in systems , results in systems that are usable, predictable, and reliable.that are usable, predictable, and reliable.
Portal + Repository + Abstraction + GridPortal + Repository + Abstraction + Grid
= New Science Capabilities= New Science Capabilities
4747
AcknowledgmentsAcknowledgmentsCooperative Computing LabCooperative Computing Lab– http://www.cse.nd.edu/~cclhttp://www.cse.nd.edu/~ccl
Grad StudentsGrad Students– Chris MorettiChris Moretti– Hoang Bui Hoang Bui – Li YuLi Yu– Mike OlsonMike Olson– Michael AlbrechtMichael Albrecht
Faculty:Faculty:– Patrick FlynnPatrick Flynn– Nitesh ChawlaNitesh Chawla– Kenneth JuddKenneth Judd– Scott EmrichScott Emrich
NSF Grants CCF-0621434NSF Grants CCF-0621434
and CNS-0643229and CNS-0643229
UndergradsUndergrads– Mike KellyMike Kelly– Rory CarmichaelRory Carmichael– Mark PasquierMark Pasquier– Christopher LyonChristopher Lyon– Jared BulosanJared Bulosan– Kameron SrimoungachKameron Srimoungach– Rachel WittyRachel Witty– Ryan JansenRyan Jansen– Joey RichJoey Rich