9/24/16

1

ConcurrentsystemsLecture4:Deadlock,Livelock,andPriorityInversion

Dr RobertN.M.Watson

1

TheDeadlock Lecture

Reminderfromlasttime

• Multi-ReaderSingle-Writer(MRSW)locks• Alternativestosemaphores/locks:– Conditionalcriticalregions(CCRs)–Monitors– Conditionvariables– Signal-and-waitvs.signal-and-continuesemantics

• Concurrencyprimitivesinpractice• Concurrencyprimitiveswrap-up

2

9/24/16

2

Fromlasttime:primitivessummary

• Concurrentsystemsrequiremeanstoensure:– Safety (mutualexclusionincriticalsections),and– Progress (conditionsynchronization)

• Spinlocks(busywait);semaphores;CCRsandmonitors– Hardwareprimitivesforsynchronisation– Signal-and-Waitvs.Signal-and-Continue

• Manyofthesearestillusedinpractice– subtleminordifferencescanbedangerous– requirecaretoavoidbugs– E.g.,“lostwakeups”

• Moredetailonimplementationinourcasestudy

3

Progress isparticularlydifficult,inlargepartbecauseofprimitivesthemselves,andisthetopicofthislecture

Thistime

• Liveness properties• Deadlock– Requirements– Resourceallocationgraphsanddetection– Prevention– theDiningPhilosophersProblem– andrecovery

• Threadpriority andtheschedulingproblem• Priorityinversion• Priorityinheritance

4

9/24/16

3

Liveness properties

• Fromatheoreticalviewpointmustensurethatweeventuallymakeprogress,i.e.wanttoavoid– Deadlock (threadssleepwaitingforeachother),and– Livelock (threadsexecutebutmakenoprogress)

• Practicallyspeaking,alsowantgoodperformance– Nostarvation(singlethreadmustmakeprogress)– (moregenerallymayaimforfairness)– Minimality (nounnecessarywaitingorsignaling)

• Thepropertiesareoftenatoddswithsafety:-(

5

Deadlock• Setofkthreadsgoasleepandcannotwakeup– eachcanonlybewokenbyanotherwho’sasleep!

• Real-lifeexample(Kansas,1920s):– “Whentwotrainsapproacheachotheratacrossing,bothshallcometoafullstopandneithershallstartupagainuntiltheotherhasgone.”

• Inconcurrentprograms,tendstoinvolvethetakingofmutualexclusionlocks,e.g.:

6

// thread 2lock(Y);

... if(<cond>) {

lock(X); ...

// thread 1lock(X);...lock(Y); // critical section unlock(Y);

Riskofdeadlockifboththreadsgethere

simultaneously

9/24/16

4

Requirementsfordeadlock

• Likeallconcurrencybugs,deadlockmayberare(e.g.imagine<cond>ismostlyfalse)

• Inpracticetherearefournecessaryconditions1. MutualExclusion:resourceshavebounded#owners2. Hold-and-Wait:cangetRxandwaitforRy3. NoPreemption:keepRxuntilyoureleaseit4. CircularWait:cyclicdependency

• Requireallfourtobetruetogetdeadlock– Butmostmodernsystemsalwayssatisfy1,2,3

7

Resourceallocationgraphs• Graphicalwayofthinkingaboutdeadlock– Circles arethreads(orprocesses)– Boxes aresingle-ownerresources(e.g.mutexes)– Edgesshowlockholdandwait conditions– Acyclemeanswe(will)havedeadlock

8

T1 T3T2

Ra Rb Rc Rd

ThicklineR->TmeansTholds resourceR

DashedlineT->RTwants resourceR

9/24/16

5

Resourceallocationgraphs

• CangeneralizetoresourceswhichcanhaveKdistinctusers(c/fsemaphores)

• Absenceofacyclemeansnodeadlock…– butpresenceonlymeansmayhave deadlock,e.g.

9

T1 T3T2

Ra(1) Rb(2) Rc(2) Rd(1)

T4

Dealingwithdeadlock

1. Ensureitneverhappens– Deadlockprevention– Deadlockavoidance(Banker’sAlgorithm)

2. Letithappen,butrecover– Deadlockdetection&recovery

3. Ignoreit!– Theso-called“OstrichAlgorithm”;-)– “Haveyoutriedturningitoffandbackonagain?”– Verywidelyusedinpractice!

10

9/24/16

6

Deadlockprevention1. MutualExclusion:resourceshavebounded#owners– Couldalwaysallowaccess…butprobablyunsafe;-(– Howevercanhelpe.g.byusingMRSWlocks

2. Hold-and-Wait:cangetRxandwaitforRy– Requirethatwerequestallresourcessimultaneously;deny

therequestifanyresourceisnotavailablenow– Butmustknowmaximalresourcesetinadvance=hard?

3. NoPreemption:keepRxuntilyoureleaseit– Stealingaresourcegenerallyunsafe(butseelater)

4. CircularWait:cyclicdependency– Imposeapartialorderonresourceacquisition– Canwork:butrequiresprogrammerdiscipline– Lockorderenforcementrulesusedinmanysystemse.g.,

FreeBSDWITNESS– staticanddynamicorderschecked11

Example:DiningPhilosophers

• 5philosophers,5forks,roundtable…

12

while(true) { // philosopher ithink();wait(fork[i]);wait(fork[(i+1) % 5];eat(); signal(fork[i]);signal(fork[(i+1) % 5];

}

Semaphore forks[] = new Semaphore[5];

• Possibleforeveryonetoacquire ‘left’fork(i)• Q:whathappensifweswaporderofsignal()s?

9/24/16

7

Example:DiningPhilosophers

• (one)Solution:alwaystakelowerforkfirst

13

while(true) { // philosopher ithink();first = MIN(i, (i+1) % 5);second = MAX(i, (i+1) % 5); wait(fork[first]);wait(fork[second];eat(); signal(fork[second]);signal(fork[first]);

}

Semaphore forks[] = new Semaphore[5];

• Nowevenif0,12,3areheld,4willnotacquirefinalfork

Deadlockavoidance

• Preventionaimsfordeadlock-free“bydesign”• Deadlockavoidance isadynamicscheme:– Assumeweknowmaximumpossibleresourceallocationforeveryprocess/thread

– Trackactualallocationsinreal-time– Whenarequestismade,onlygrantifguaranteednodeadlockevenifallotherstakemaxresources

• e.g.Banker’sAlgorithm– seetextbooks– Notreallyusefulingeneralasneedaprioriknowledgeof#processes/threads,andtheirmaxresourceneeds

14

9/24/16

8

Deadlockdetection• Deadlockdetectionisadynamicschemethatdeterminesif

deadlockexists– Principle:atasomemomentinexecution,examineresource

allocationsandgraph– determineifthereisatleastoneplausiblesequenceofeventsbywhichprogresscouldbemade

• Whenonlyasingleinstanceofeachresource,canexplicitlycheckforacycle:– Keeptrackwhichobjecteachthreadiswaitingfor– Fromtimetotime,iterateoverallthreadsandbuildthe

resourceallocationgraph– RunacycledetectionalgorithmongraphO(n2)

• Moredifficultifhavemulti-instanceresources

15

Deadlockdetection

• Havemdistinctresourcesandnthreads• V[0:m-1],vectorofcurrently availableresources• A,themxn resourceallocationmatrix,andR,themxn (outstanding)requestmatrix– Ai,j isthenumberofobjectsoftypej ownedbyi– Ri,j isthenumberofobjectsoftypej neededbyi

• ProceedbysuccessivelymarkingrowsinA forthreadsthatarenotpartofadeadlockedset– IfwecannotmarkallrowsofA wehavedeadlock

16Optimisticassumption:ifwecanfulfillthreadi’srequestRi,thenitwillruntocompletionandreleaseheldresourcesforotherthreadstoallocate.

9/24/16

9

Deadlockdetectionalgorithm• MarkallzerorowsofA (sinceathreadholdingzeroresourcescan’tbepartofdeadlockset)

• InitializeaworkingvectorW[0:m-1]toV– W[]describesanyfreeresourcesatstart,plus anyresourcesreleasedbyahypothesizedsequenceofsatisfiedthreadsfreeingandterminating

• Selectanunmarkedrowi ofA s.t.R[i]<=W– (i.e.findathreadwho’srequestcanbesatisfied)– SetW =W +A[i];markrowi,andrepeat

• Terminatewhennosuchrowcanbefound– Unmarkedrows(ifany)areinthedeadlockset

17

Deadlockdetectionexample1

• Fivethreadsandthreeresources(nonefree)

18

X Y Z X Y Z X Y Z T0 0 1 0 0 0 0 0 0 0 T1 2 0 0 2 0 2 T2 3 0 3 0 0 0 T3 2 1 1 1 0 0 T4 0 0 1 0 0 2

A R V

• Findanunmarkedrow,markit,andupdateW• T0,T2,T3,T4,T1

W

X Y Z0 0 0 X Y Z0 1 0 X Y Z3 1 3 X Y Z5 2 4 X Y Z5 2 5 X Y Z7 2 5

Attheendofthealgorithm,allrowsaremarked:thedeadlocksetisempty.

9/24/16

10

Deadlockdetectionexample2

• Fivethreadsandthreeresources(nonefree)

19

X Y Z X Y Z X Y Z T0 0 1 0 0 0 0 0 0 0 T1 2 0 0 2 0 2 T2 3 0 3 0 0 1 T3 2 1 1 1 0 0 T4 0 0 1 0 0 2

A R V

• OneminortweaktoT2’srequestvector…

W

X Y Z0 0 0 X Y Z0 1 0

CannotfindarowinR<=W!!

NowwantsoneunitofresourceZ

ThreadsT1,T2,T3&T4indeadlockset

Deadlockrecovery• Whatcanwedowhenwedetectdeadlock?• Simplestsolution:killsomething!– Ideallysomeoneinthedeadlockset;-)

• Brutal,andnotguaranteedtowork– Butsometimesthebest(only)wecando– E.g.LinuxOOMkiller(betterthansystemreboot?)– …Ornot– oftenkillstheXserver!

• Couldalsoresumefromcheckpoint– Assumingwehaveone

• Inpracticecomputersystemsseldomdetectorrecoverfromdeadlock:relyonprogrammer

20

Note:“killsomeone”breaksthenopreemption requirementfordeadlocks.

9/24/16

11

Livelock• Deadlockisatleast‘easy’todetectbyhumans– Systembasicallyblocks&stopsmakinganyprogress

• Livelockislesseasytodetectasthreadscontinuetorun…butdonothinguseful

• Oftenoccursfromtryingtobeclever,e.g.:

21

// thread 2lock(Y);

... while(!trylock(X)) {

unlock(Y); yield();lock(Y);

}...

// thread 1lock(X);...while (!trylock(Y)) {

unlock(X); yield();lock(X);

} ...

Schedulingandthreadpriorities• Whichthreadshouldrunwhen>1arerunnable?E.g.,if:

– Athreadreleasesacontendedlockandcontinuestorun– CVbroadcastwakesupseveralwaitingthreads

• Manypossibleschedulingpolicies;e.g.,– Roundrobin– rotatebetweenthreadstoensureprogress– Fixedpriorities– assignprioritiestothreads,schedulehighest

priority– e.g.,real-time>interactive >bulk >idle-time– Dynamicpriorities– adjustprioritiestobalancegoals– e.g.,

boostpriorityafterI/Otoimproveinteractivity– Gangscheduling– scheduleforpatternssuchasP-C– Affinity– scheduletoefficientlyutilise resources(e.g.,caches)

• Goals:latencyvs.throughput,utilisation,energy,fairness

22

9/24/16

12

Priorityinversion• Anotherlivenessproblem…– Duetointeractionbetweenlockingandscheduler

• Considerthreethreads:T1,T2,T3– T1 ishighpriority,T2mediumpriority,T3 islow– T3 getsluckyandacquireslockL…– …T1 preemptsT3 andsleepswaitingforL…– …thenT2 runs,preventingT3 fromreleasingL!– Priorityinversion:despitehavinghigherpriorityandnosharedlock,T1 waitsforlowerpriorityT2

• Thisisnotdeadlockorlivelock– Butnotdesirable(particularlyinreal-timesystems)!– DisabledMarsPathfinderrobotforseveralmonths

23

Priorityinheritance• Typicalsolutionispriorityinheritance:– Temporarilyboostpriorityoflockholdertothatofthehighestwaitingthread

– T3 wouldhaverunwithT1’sprioritywhileholdingalockT1 waswaitingfor– preventingT2 frompreemptingT3

– Concretebenefitstosysteminteractivity– (someRTsystems(likeVxWorks)allowyouspecifyonaper-mutex basis[toRover’sdetriment;-])

• Windows“solution”– Checkifanyreadythreadhasn’trunfor300ticks– Ifso,doubleitsquantumandboostitspriorityto15– J

24

9/24/16

13

Problemswithpriorityinheritance• Hardtoreasonaboutresultingbehaviour:heuristic• Worksforlocks

– Morecomplexthanitappearsatfirst:propagationmightneedtobeextendedacrosschainscontainingmultiplelocks

– Howmightwehandlereader-writerlocks?• Butwhataboutconditionsynchronisation,resource

allocation?– Withlocks,weknowwhatthreadholdsthelock– Semaphoresdonotrecordwhichthreadmightissueasignalor

releaseanallocatedresource– Mustcomposeacrossmultiplewaitingtypes:e.g.,“waitingfora

signalwhileholdingalock”• Wherepossible,avoidtheneedforpriorityinheritance

– Avoidresourcesharingbetweenthreadsofdifferingpriorities

25

Summary+nexttime• Liveness properties• Deadlock

– Requirements– Resourceallocationgraphsanddetection– Prevention– theDiningPhilosophersProblem– andrecovery

• Threadpriority andtheschedulingproblem• Priorityinversion• Priorityinheritance

• Nexttime:– Concurrencywithoutshareddata– Activeobjects;messagepassing– Compositeoperations;transactions– ACIDproperties;isolation;serialisability 26