Embed Size (px)
Citation preview
1
18-847F:SpecialTopicsinComputerSystems
FoundationsofCloudandMachineLearning
Infrastructure
2
Lecture2:OverviewandKeyConcepts
FoundationsofCloudandMachineLearning
Infrastructure
GraduateSeminarClass
3
(Almost)nolectures
Readingresearchpapers
Studentpresentations
ClassDiscussions
FinalResearchProject(NoExams!)
TODO
4
o Sign-upforpresentation
o Formgroupsforclassprojects
o Startthinkingaboutprojects
TopicsCovered
5
CloudCompu)ng DistributedStorage
MachineLearning
Modelreplica
PARAMETERSERVERw’=w–αΔw
Modelreplica
Modelreplica
w Δw
a b a+b
HistoryandOverview
6
CloudCompu)ng DistributedStorage
a b a+b
HistoryandOverview
7
CloudCompu)ngo MapReduce,Spark
o SchedulinginParallelComputing
o StragglerReplication
o TaskReplicationinQueueingSystems
Whatisthecloud?
8
Acollectionofserversthatcanfunctionasasinglecomputing
node,andcanbeaccessedfrommultipledevices
1960’s:TheMainframeEra
9
o Large,expensivemachineso Onlyoneperuniversity/institution
IBM704(1964)
1970’s:Virtualization
10
o IBMreleasedaVMOSthatallowedmultipleuserstosharethemainframecomputer
IBM704(1964)
1980’s-1990’s:InternetandPCs
11
o PCsbecomeaffordableo Internetconnectivitywentonimproving
o VirtualPrivateNetworks(VPNs)
o GridComputing:ConnectcheapPCsviatheInternet
o Onthetheoryside,queueingtheory,traditionallyusedinoperationsmanagementrebounded
AShortTutorialonQueueingTheory
12
13
Reference Textbooks
DesignQuestion1Whatifthearrivalratedoubles?
14
Servicerateμ=5Job Arrival
Rate λ = 3
MeanResponseTimeT=Wai)ng)meinQueue+ServiceTime
Q: If λ doubles, do you need a server of 2x rate to achieve the same E[T]?
DesignQuestion2Manyslow,oronefastserver?
15
3μλ
μ
μ
μ
Choose first idle server
λ
Q: Which of the two systems gives lower E[T]?
DesignQuestion3Howtoassignjobstoservers
16
Q: Which policy works the best?
μ
μ
μ
λ
Random, Round-robin, Shortest Queue, SITA, LWL
QueueingTerminology
17
ServicerateμArrival Rate λ
MeanServiceTime E[S]=1/μ
MeanWaitingTime E[W]
MeanResponseTime E[T]=E[W]+E[S]
Mean#CustomersinQueue E[N]
ServerUtilization ρ=λ/μ
Little’sLaw
18
Theorem:ForanyergodicopensystemwehaveE[N]=λE[T]
Some Variants E[Nw] = λ E[W] ρ = λ E[S]
Very general and hence powerful law • Any # of servers, scheduling policy, queue size limit
Little’sLaw:Quiz
19
A professor takes 2 new students in even-numbered years, and 1 new student in odd-numbered years. If avg. graduation time = 6 yrs, how many students will the professor have on average?
Little’sLaw:Answer
20
A professor takes 2 new students in even-numbered years, and 1 new student in odd-numbered years. If avg. graduation time = 6 yrs, how many students will the professor have on average?
E[N]=λE[T] =1.5*6 =9
Kendall’sNotation
21
ServicerateμArrival Rate λ
A/S/nARRIVALDIST.M:PoissonEk:ErlangG:General
SERVICEDIST.M:Exponen)alD:Determinis)c
G:General
Numberofservers
M/M/1Queue
22
WANTTOFIND
1. MeanResponseTimeE[T]
2. MeanWaitingTimeE[W]
ServicerateμArrival Rate λ
M/M/1:MarkovModel
23
0 2
λ
µ
1 4 3
λ
µ
λ
µ
λ
µ
where
M/M/1:MeanResponseTime
24
0 2
λ
µ
1 4 3
λ
µ
λ
µ
λ
µ
Quiz:DesignQuestion1Whatifthearrivalratedoubles?
25
Servicerateμ=5Job Arrival
Rate λ = 3
MeanResponseTimeT=Wai)ng)meinQueue+ServiceTime
Q: If λ doubles, do you need a server of 2x rate to achieve the same E[T]?
A: Service rate 6+2 = 8 is sufficient
M/M/nQueue
26
μ
μ
μ
Choose first empty queue
λ
WANTTOFIND
1. MeanResponseTimeE[T]
2. MeanWaitingTimeE[W]
M/M/nQueue
27
0 2
λ
µ
1 n
λ
2µ
λ
3µ
λ
nµ
λ
nµ
where
Erlang-CFormula
PQ =1X
i=n
⇡i
= ⇡0nn
n!
1X
i=n
⇢i
=nn⇡0
n!(1� ⇢)
⇡0 =
"n�1X
i=0
(n⇢)i
i!+
(n⇢)n
n!(1� ⇢)
#�1
⇢ =�
nµ
Used in call centers to determine number of agents required
M/M/nQueue
28
E[Nw] =1X
i=n
⇡i(i� n)
= ⇡0
1X
i=n
⇢inn
n!(i� n)
= PQ⇢
1� ⇢
E[W ] =E[Nw]
�= PQ
⇢
�(1� ⇢)
E[T ] = PQ⇢
�(1� ⇢)+
1
µ
Quiz:Comparisonof3systems
29
μ
μ
μ
Choose first empty server
λnμλ
μ
μ
μ
λ/n
λ/n
λ/n
M/M/1 M/M/n
FDM
Quiz:Comparisonof3systems
30
μ
μ
μ
Choose first empty server
λnμλ
μ
μ
μ
λ/n
λ/n
λ/n
M/M/1 M/M/n
FDM
E[T ]M/M/1 =1
nµ� �
E[T ]FDM =n
nµ� �FDMisn)messlowerthanM/M/1
Quiz:Comparisonof3systems
31
μ
μ
μ
Choose first empty server
λnμλ
M/M/1 M/M/n
E[T ]M/M/n = PQ⇢
�(1� ⇢)+
1
µE[T ]M/M/1 =
⇢
�(1� ⇢)
E[T ]M/M/n
E[T ]M/M/1= PQ + n(1� ⇢)
M/M/nisn)messlowerwhenρà0
M/M/nandM/M/1arealmostequalwhenρà1
M/G/1QueuePollaczek-KhinchineFormula
32
ServicedistFX Arrival Rate λ
CannotuseMarkovchain
analysis
E[T ] = E[X] +E[X2]
2(1� �E[X])
ProofofPKformula
33
Resid
ualservice)me
Time
X1 X2
E[Tw] = E[Nw] · E[X] + E[R]
= �E[Tw] · E[X] +E[X2]
2
=E[X2]
2(1� �E[X])
M/G/nQueue
34
ServicedistFX
CannotuseMarkovchain
analysis
Choose first empty queue
λ
ServicedistFX
ServicedistFX
E[T ] ⇡ E[X] +E[X2]
2E[X]· E[WM/M/n]
1990’s:SchedulinginParallelComputing
o Bin-Packingo Needjobsizeestimates
35
J2
J3
J1
J4Processors
Time
Forreferencesseesurvey[Weinberg2008]
1990’s:SchedulinginParallelComputing
o Bin-Packingo Needjobsizeestimates
o ProcessorSharing,i.e.switchingb/wthreadsfordifferentjobso Needprocessorspeedestimates
o Load-balancing:Workstealing,Power-of-choiceo Needqueuelengthestimates
36
1990’s:InternetandPCs
37
o PCsbecomeaffordableo Internetconnectivitywentonimproving
o VirtualPrivateNetworks(VPNs)o GridComputing:ConnectcheapPCsviatheInternet
o ManyInternetCompaniesboughttheirownserversandmanagedthemprivately
o ButthentheDotcombubbleburst..
2000’s:TheCloudComputingEra
38
o Theideaofaflexible,low-cost,scalable,sharedcomputingenvironmentdeveloped
o Computingbecomeautility,likeelectricity
KEYISSUE:Jobsizes,serverspeeds&queuelengthsareunpredictableREASON:Large-scaleresourcesharingàVariabilityinservice
• Virtualization,serveroutagesetc.• Normandnotanexception[Dean-Barroso2013]
2000’s:TheCloudComputingEra
39
TheTaleofTails
40
TailatScale:99%ilelatencycanbemuchhigherthanaverage
TheTaleofTails
41
TailatScale:99%ilelatencymuchhigherthanaverage
TaleofTails:Quiz
42
Aserverfinishesataskin1secwithprobability0.9,and10secwithprobability0.1o Whatistheexpectedtaskexecutiontime?
o If100tasksareruninparallelof100servers,whatistheexpectedtimetocompleteallofthem.
TaleofTails:Quiz
43
Aserverfinishesataskin1secwithprobability0.9,and10secwithprobability0.1o Whatistheexpectedtaskexecutiontime?
1*0.9+10*0.1=1.9o If100tasksareruninparallelof100servers,whatis
theexpectedtimetocompleteallofthem.
TaleofTails:Quiz
44
Aserverfinishesataskin1secwithprobability0.9,and10secwithprobability0.1o Whatistheexpectedtaskexecutiontime?
1*0.9+10*0.1=1.9o If100tasksareruninparallelof100servers,whatis
theexpectedtimetocompleteallofthem.1*0.9100+10*(1-0.9100)~10
StragglerReplication
45
Task1
Task2
Task3
Task4
PROBLEM:SlowesttasksbecomeabottleneckSOLUTION:Replicatethestragglersandwaitforonecopy
Task4
Eg.MapReduce,ApacheSparklaunch1replica,keeporiginal
copy
PARAMETERSp:Frac.oftasksreplicatedr:#additionalreplicasc:kill/keeporiginaltask
StragglerReplicationAnalysis[Wang-GJ-WornellSIGMETRICS2014,15]
46
METRICSE[T]=TimetofinishalltasksE[C]=Totalserverruntimepertask Yistheresidual
service)mealeraddingreplicas
PARAMETERSp:Frac.oftasksreplicatedr:#additionalreplicasc:kill/keeporiginaltask
E[X(1�p)n:n] = x1�p = F
�1X (1� p)
CentralValueTheorem
ExtremeValueTheorem
DifferentbehaviorforExponen)al,LightorHeavytailedY
n -> ∞ n -> ∞
SimulationsusingGoogleClusterDataLatency-CostTrade-off
47
Increasingfrac)onoftasksreplicated
Expe
cted
Laten
cyE[T]
ExpectedCostE[C]
Carefulchoiceofreplica)onstrategycan
bebeoerthanthedefaultinMapReduce
TaskReplicationinQueueingSystems
48
TaskReplicationinCloudComputing
IDEA:Assigntasktomultipleserversandwaitforearliestcopy
COST
o Additionalcomputingtimeatservers
49
Waitfortheearliestcopyto
finish,andcanceltherest
Task
TaskReplicationinCloudComputing
IDEA:Assigntasktomultipleserversandwaitforearliestcopy
COST
o Additionalcomputingtimeatservers
o Increasedqueuingdelayforothertasks
50
Waitfortheearliestcopyto
finish,andcanceltherest
Analogy:SupermarketQueues
51
SupermarketQueues
52
SupermarketQueues
Getafriendtojointheotherqueue!
53Whatifeveryoneinthesupermarketusesthisstrategy?
DesignQuestions
o Howmanyreplicastolaunch?
o Whichqueuestojoin?
o Whentoissueandcancelthereplicas?
54
1
2
n
SurprisingInsight
Incertainregimes,replicationcouldmakethewholesystemfaster,andcheaper!
55
VS
Effectiveservicerate>Sumofindividualservers
CloudSpotMarkets
56
o Sparecapacityincloudcomputing
Morning Alernoon Evening Night
Usage
Needthismuchcapacity
Whattodowithsparecapacity?
Capacity
CloudSpotMarkets
57
o Sellitonthespotmarketforalowerprice!
Morning Alernoon Evening Night
SpotInstancePrice
OnDemandprice
Price
BiddingforSpotInstances
58
o Sellitonthespotmarketforalowerprice
Morning Alernoon Evening Night
SpotInstancePrice
OnDemandprice
Bid
Pre-emptedatthis)me
Sept27GuestLecture:Prof.CarleeJoe-Wong
59
o Biddingandpricingstrategiesforspotmarkets
Morning Alernoon Evening Night
SpotInstancePrice
OnDemandprice
Bid
Pre-emptedatthis)me
HistoryandOverview
60
CloudCompu)ng DistributedStorage
a b a+b
HistoryandOverview
61
DistributedStorageo RAIDsystems
o Codingforlocality/repair
o Systemsimplementationofcodes
o Reducinglatencyincontent
download
a b a+b
o Contentisreplicatedonthecloudforreliability
o Cansupportmoreuserssimultaneouslyo Replicatedusedfor“hot”data,i.e.morefrequentaccessed
ReplicatedStorage
62
Any1outof3copiesissufficient
o Withan(n,k)MDScode,anykoutofnchunksaresufficiento Facebook,Google,Microsoftuse(14,10)or(7,4)codeso Currentlyusedforcolddata,increasingforhotdata
ErasureCodedStorage
63
Anyk=2outofn=3aresufficient
RAID:RedundantArrayofIndependentDisks(1987)
64
o LevelsRAID0,RAID1,…:designfordifferentgoalssuch
asreliability,availability,capacityetc.
o Oneoftheinventors,GarthGibsonishereatCMU!
CodingTheory
65
o Forreliablecommunicationinpresenceofnoise
o BellLabswasoneoftheleadersin1950’s
o Keyfigures:ClaudeShannonandRichardHamming
SimplestCodes
66
o RepetitionCodeo 0à000:Rate:1/3
o Ifreceive0??wecanrecoverfrom2erasures
o (3,2)code:Databits:a,bParitybit:(aXORb)
o Example:011,110:Rate2/3
o Ifwereceive0?1or?10wecancorrectthefailedbit
o 2bitsymbols:(01)?(11)
HammingCodes
67
o (7,4)HammingCode:4databits,3paritybits
o Parity
p1 = d1 � d2 � d4
HammingCodes:Quiz
68
o Whatistherateofthecode?
o Correctthe2erasureso (d1,d2,d3,d4,p1,p2,p3)=(0,?,1,?,1,0,0)
HammingCodes:Answer
69
o Whatistherateofthecode?R=4/7
o Correctthe2erasureso (d1,d2,d3,d4,p1,p2,p3)=(0,0,1,1,1,0,0)
(n,k)Reed-SolomonCodes:1960
70
o Data:d1,d2,d3,…dk
o Polynomial:d1+d2x+d3x2+…dkxk-1
o Paritybits:Evaluateatn-kpoints:
x=1: d1+d2+d3+d4
x=2: d1+2d2+4d3+8d4
x=3: …. x=4: …. x=n: …
o Cansolveforthecoefficientsfromanykcodedsymbols
Example:(4,2)Reed-SolomonCode
71
o Data:d1,d2àPolynomial:d1+d2x+d3x2+…dkxk-1
o Cansolveforthecoefficientsfromanykcodedsymbolso Microsoftuses(7,4)codeo Facebookuses(14,10)code
d1 d2 d1+d2 d1+2d2
o RepairingfailednodesishardwithReed-SolomonCodes..
o Ifwelose1node:
o Needtocontactkothernodes
o Needtodownloadktimesthelostdata
LocalityandRepairIssues
72
Solution:LocallyRepairableCodes
73
o Codesdesignedtominimize:o RepairBandwidtho Numberofnodescontacted
GuestLecture:Prof.RashmiVinayak
74
o SystemsImplementationof‘piggybacking’erasurecodesinApacheHadoop
o Withan(n,k)MDScode,anykoutofnchunksaresufficiento Facebook,Google,Microsoftuse(14,10)or(7,4)codeso Currentlyusedforcolddata,increasingforhotdata
Q:Howmanyuserscanweserve,andhowfast?
ErasureCodedStorage
75
Anyk=2outofn=3aresufficient
The(n,k)fork-joinmodel[GJ-Liu-Soljanin2012,14]
o Requestallnchunks,waitforanyktobedownloadedo EachchunktakesservicetimeX~FX
λ
k = 1: Replicated Case k = n: Fork-join system actively studied in 90’s 76
Wait for any 2 out of 3 chunks
Downloadrequests
(3,2) fork-join
CodedComputingandML
77
Ax
o Sofar:Codingforstorageo Codescanalsospeedupcomputingandmachinelearning!
o Example:Matrix-VectorMultiplication
CodedComputingandML
78
x
o Sofar:codingforstorageo Codescanalsospeedupcomputingandmachinelearning!
o Example:Matrix-VectorMultiplication
A1
A2
CodedComputingandML
79
o Sofar:codingforstorageo Codescanalsospeedupcomputingandmachinelearning!
o Example:Matrix-VectorMultiplication
xA1
xA1
x
A1+A2
Needonly2outof3tofinish
GuestLecture:SanghamitraDutta
80
Short-dotcodes
Second-halfoftheClass:MachineLearning
81
CloudCompu)ng DistributedStorage
MachineLearning
Modelreplica
PARAMETERSERVERw’=w–αΔw
Modelreplica
Modelreplica
w Δw
a b a+b
