MarkovChainsJ. R. NorrisUniversity of Cambridge~ u ~ ~ u CAMBRIDGE::: UNIVERSITY PRESSPUBLISHED BY THE PRESS SYNDICATE OF THE UNIVERSITY OF CAMBRIDGEThe Pitt Building, Trumpington Street, Cambridge CB2lRP, United KingdomCAMBRIDGE UNIVERSITY PRESSThe Edinburgh Building, Cambridge CB2 2RU, United Kingdom40 West 20th Street, New York, NY 10011-4211, USA10 Stamford Road, Oakleigh, Melbourne 3166, AustraliaCambridge University Press1997This book is in copyright. Subject to statutory exceptionand to the provisions of relevant collective licensing agreements,no reproduction of any part may take place withoutthe written permission of Cambridge University Press.First published1997Reprinted1998First paperback edition 1998Printed in the United States of AmericaTYPeset in Computer ModemA catalogue recordfor this book is available from the British LibraryLibrary of Congress Cataloguing-in-Publication Data is availableISBN 0-521-48181-3 hardbackISBN 0-521-63396-6 paperbackFor my parentsContentsPrefaceIntroduction1. Discrete-time- Markov chains1.1Definition andbasicproperties1.2 Class structure1.3 Hitting timesandabsorptionprobabilities1.4 Strong Markov property1.5 Recurrence andtransience1.6 Recurrence andtransienceof random walks1.7 Invariantdistributions1.8 Convergence to equilibrium1.9 Time reversal1.10 Ergodic theorem1.11Appendix: recurrencerelations1.12 Appendix: asymptoticsforn!2. Continuous-timeMarkov chainsI2.1Q-matrices andtheir exponentials2.2Continuous-time random processes2.3Some properties of the exponential distributionixxiii11101219242933404752575860606770viii Contents2.4Poisson processes 732.5 Birth processes 812.6 Jump chain andholding times 872.7 Explosion 902.8Forward andbackward equations 932.9Non-minimalchains 1032.10 Appendix: matrix exponentials 1053. Continuous-time Markov chains II 1083.1Basic properties 1083.2Class structure 1113.3 Hitting times andabsorption probabilities 1123.4 Recurrence and transience 1143.5 Invariantdistributions 1173.6Convergence toequilibrium 1213.7 Timereversal 1233.8 Ergodic theorem 1254. Further theory 1284.1Martingales 1284.2Potential theory 1344.3Electrical networks 1514.4 Brownian motion 1595. Applications 1705.1Markov chains in biology 1705.2Queues andqueueing networks 1795.3Markov chains in resourcemanagement 1925.4 Markov decision processes 1975.5Markov chain MonteCarlo 2066. Appendix: probability andmeasure 2176.1Countable sets andcountable sums 2176.2Basic factsof measure theory 2206.3Probability spaces and expectation 2226.4Monotoneconvergence andFubini's theorem 2236.5Stopping timesand the strong Markov property 2246.6Uniqueness of probabilities andindependence of a-algebras 228Further reading 232Index 234PrefaceMarkov chainsare thesimplestmathematical modelsforrandomphenom-ena evolving in time. Their simple structure makes it possible to say a greatdeal about theirbehaviour. At thesametime, theclassof Markovchainsisrichenoughtoserveinmanyapplications. This makes Markovchainsthefirst andmost important examplesof randomprocesses. Indeed, thewhole of the mathematical study of random processes can be regarded as ageneralization in one way or another of the theory of Markov chains.This bookis anaccount ofthe elementarytheoryof Markovchains,withapplications. Itwasconceived asa text foradvancedundergraduatesor master's level students, andis developedfromacoursetaught toun-dergraduatesfor several years. Therearenostrict prerequisitesbut it isenvisaged that the reader will have taken a course in elementary probability.In particular, measure theoryisnot aprerequisite.Thefirst half of thebookisbasedonlecturenotesfortheundergradu-atecourse. Illustrative examplesintroducemany of thekeyideas. Carefulproofsaregiven throughout. There isaselection of exercises, which formsthe basis ofclassworkdone bythestudents, andwhichhas beentestedover several years. Chapter 1 deals with the theory of discrete-time Markovchains, andis the basis ofall that follows. Youmust beginhere. Thematerial is quite straightforwardandtheideas introducedpermeatethewholebook. Thebasicpatternof Chapter 1 isrepeatedinChapter3forcontinuous-time chains, making iteasy tofollow thedevelopmentbyanal-ogy. In between, Chapter 2 explains how to set up the theory of continuous-x Prefacetimechains, beginningwithsimpleexamplessuchasthePoissonprocessandchains with finite state space.Thesecondhalf of thebookcomprisesthreeindependent chaptersin-tendedtocomplement thefirst half. Insomesections thestyleis a lit-tlemoredemanding. Chapter 4introduces, inthecontext of elementaryMarkov chains, some of theideascrucial to theadvanced study of Markovprocesses, such as martingales, potentials, electrical networks and Brownianmotion. Chapter 5isdevotedtoapplications, for exampletopopulationgrowth,mathematical genetics, queues and networks of queues, Markov de-cision processesandMonteCarlosimulation. Chapter6 is anappendix tothemaintext, whereweexplainsomeof thebasicnotionsof probabilityand measure used in the rest of the book and give careful proofs of the fewpoints where measure theoryisreally needed.Thefollowingparagraphisdirectedprimarilyat aninstructorandas-sumes some familiarity with the subject. Overall, the book is more focusedon theMarkovian context thanmost otherbooksdealing with the elemen-tarytheoryof stochasticprocesses. Ibelieve that thisrestrictioninscopeisdesirableforthegreatercoherenceanddepthit allows. Thetreatmentof discrete-timechainsinChapter 1includesthecalculationof transitionprobabilities, hitting probabilities, expected hitting times and invariant dis-tributions. Also treated are recurrence and transience, convergence to equi-librium, reversibility, andtheergodictheoremfor long-runaverages. Alltheresults areproved, exploitingtothefull theprobabilisticviewpoint.For example, weuseexcursionsand the strong Markov property to obtainconditions forrecurrence and transience,andconvergence to equilibrium isprovedbythecouplingmethod. InChapters2and3weproceedviathejumpchain/holdingtimeconstructiontotreat all right-continuous, mini-mal continuous-time chains, and establish analogues of all the main resultsobtainedfor discretetime. Noconditionsof uniformlyboundedratesareneeded. Thestudent hastheoptiontotakeChapter3first, tostudytheproperties of continuous-timechainsbeforethetechnicallymoredemand-ing construction. We have left measure theoryinthe background, buttheproofsareintended tobe rigorous, or veryeasilymaderigorous, whenconsideredinmeasure-theoreticterms. Somefurther details aregiveninChapter 6.It isapleasure toacknowledge the work of colleagues from which Ihavebenefittedinpreparingthis book. Thecourseonwhichit is basedhasevolvedover manyyearsandunder manyhands- I inheritedpartsof itfromMartinBarlowandChrisRogers. Inrecent yearsit hasbeengivenbyDougKennedyandColinSparrow. RichardGibbens, GeoffreyGrim-Preface ximett, Frank Kelly and Gareth Roberts gave expert advice at various stages.Meena Lakshmanan, Violet Lo and David Rose pointed out many typos ,andambiguities. BrianRipleyandDavidWilliamsmadeconstructivesugges-tionsforimprovement of an early version.I am especially grateful to David Thanah atCambridge University Pressforhissuggestion to write the book and forhiscontinuing support, and toSarahShea-Simondswhotypesetthewholebook with efficiency, precisionandgoodhumour.Cambridge, 1996 JamesNorrisIntroductionThisbookis about acertain sort of randomprocess. Thecharacteristicproperty of this sort of process is that it retainsnomemory of where ithasbeen in the past. This means that only the current state of the process caninfluence where it goes next. Such a process is called aMarkov process. Weshall beconcerned exclusively with thecase where theprocesscan assumeonlya finite or countableset ofstates, whenit is usual torefer it as aMarkovchain.Examples of Markov chains abound,as you will see throughout the book.What makes themimportant is that not onlydo Markovchains modelmanyphenomena of interest, but also thelack of memory propertymakesit possibletopredict howa Markovchainmaybehave, andtocomputeprobabilitiesandexpectedvalues whichquantifythat behaviour. Inthisbook we shall present general techniques for the analysis of Markov chains,together withmanyexamples andapplications. Inthis introductionweshall discuss afew very simple examples and preview some of the questionswhich thegeneral theory willanswer.We shallconsider chains both indiscretetimenE Z+= {O, 1, 2, ... }andcontinuoustimet EjR+ =[0, (0).Thelettersn, m, kwill alwaysdenoteintegers, whereast andswill refertoreal numbers. Thuswewrite ( X n ) n ~ O for adiscrete-timeprocessand( X t ) t ~ O foracontinuous-timeprocess.XIV IntroductionMarkovchainsareoftenbest describedbydiagrams, of whichwenowgive some simple examples:(i) (Discretetime)13132Youmovefromstate 1tostate2withprobability1. Fromstate3youmoveeither to1 or to2 withequalprobability1/2, andfrom2 you jumpto 3 with probability 1/3, otherwise stay at2. We might have drawn a loopfrom2 toitself withlabel 2/3. Butsince the totalprobability on jumpingfrom2must equal 1, thisdoes not conveyanymoreinformationandweprefer toleave theloopsout.(ii) (Continuoustime)Ao- - - ~ . ~ - - ...... 1Whenin state0 you wait forarandom time with exponentialdistributionof parameterA E(0, 00), then jump to1. Thus thedensity functionof thewaiting time Tisgiven byfort ~ o.We write Trv E(A) forshort.(iii) (Continuous time)A o 1A2 3 4Here, whenyouget to1youdonot stopbut after another independentexponential time of parameter Ajump to 2, and so on. The resulting processiscalled thePoissonprocessof rateA.3Introduction142xv(iv) (Continuoustime)Instate 3youtake twoindependent exponential times T1rv E(2) andT2 rvE (4); if T1isthesmalleryougo to1 aftertimeT1 , andif T2 is thesmaller you go to 2 after time T2 . The rules forstates1 and2 are asgivenin examples(ii)and(iii). It is asimple matter to show that the time spentin3is exponential ofparameter 2 + 4=6, andthat theprobabilityofjumping from3 to1 is2/(2 +4)= 1/3. Thedetails are given later.(v) (Discretetime)3 62o5We usethis exampletoanticipatesomeoftheideas discussedindetailinChapter 1. Thestatesmaybepartitionedintocommunicatingclasses,namely {O}, {I, 2, 3} and {4, 5, 6}. Two of these classes are closed,meaningthat youcannot escape. Theclosedclasses hereare recurrent, meaningthatyou returnagain andagain toevery state. The class{O}istransient.Theclass{4, 5, 6}is periodic, but {I, 2, 3}isnot. Weshall showhowtoestablish thefollowingfactsbysolvingsomesimplelinear/ equations. Youmight like to try fromfirst principles.(a) Starting from0, the probability of hitting6 is1/4.(b) Starting from1, the probability of hitting 3 is1.(c) Starting from1, ittakes on average three steps to hit3.(d) Starting from1, the long-run proportion of time spentin2 is3/8.xvi IntroductionLet uswrite pij for the probability starting fromi of being in state j afternsteps. Then we have:(e) limPOI = 9/32;n---+oo(f) P04doesnot convergeas n~ 00;(g) limpg4= 1/124.n---+oo1Discrete-timeMarkov chainsThischapter isthefoundationfor all that follows. Discrete-timeMarkovchains aredefinedandtheir behaviour is investigated. For better orien-tationwenowlist thekeytheorems: theseareTheorems 1.3.2and1.3.5onhittingtimes, Theorem1.4.2on thestrongMarkovproperty, Theorem1.5.3characterizing recurrenceandtransience, Theorem1.7.7 oninvariantdistributions andpositive recurrence. Theorem1.8.3onconvergence toequilibrium, Theorem1.9.3onreversibility, andTheorem1.10.2onlong-runaverages. Onceyouunderstandtheseyouwill understandthebasictheory. Part of that understandingwill comefromfamiliaritywithexam-ples, soalarge numberare worked outin the text. Exercises atthe end ofeach section are an important partof the exposition.1.1Definition andbasic propertiesLet I beacountableset. Eachi EI iscalleda stateandI iscalledthestate-space. Wesay that A = (Ai: i EI) isameasure onI if 0 ~ Ai o.We say i communicateswithj andwrite i ~ j if both i ~ j andj ~ i.Theorem1.2.1. For distinctstates i and j thefollowing are equivalent:(i) i ~ j;(ii) Pioil Pili2 ... Pin-lin>0forsome states io,il, ... ,inwithio= i andin = j;(iii) p ~ j ) > 0 for some n ~ O.Proof. Observe that00p ~ j ) ::; lPi(Xn= j for some n~ 0)::; Lp ~ j )n=Owhich proves theequivalence of (i) and(iii). Alsop ~ j ) = L Pii1P i li2 .. Pin-dil , ... ,in-lso that (ii) and(iii) are equivalent. 01.3 Hittingtimesandabsorptionprobabilities 11It isclearfrom(ii) that i ~ j andj ~ kimplyi ~ k. Alsoi ~ i forany statei. So~ satisfies theconditionsforan equivalencerelation on I,and thuspartitionsI intocommunicatingclasses. We say thataclassCisclosedifi EC, i ~ j imply j EC.Thus a closedclass is one fromwhichthere is noescape. Astatei isabsorbingif {i}isaclosedclass. Thesmallerpiecesreferred toabovearethesecommunicatingclasses. Achainor transitionmatrixPwhereI isasingle classiscalledirreducible.As thefollowingexamplemakesclear, when one can draw thediagram,theclass structure of achainisvery easy to find.Example1.2.2Find the communicating classesassociated to the stochasticmatrix1 10 0 0 02 20 0 1 0 0 010 01 10P=3 3 30 0 01 102 20 0 0 0 0 10 0 0 0 1 0The solution isobviousfrom thediagram1 42 6theclassesbeing {1,2,3}, {4}and{5,6}, with only {5,6}being closed.Exercises1.2.1 Identify the communicating classes of the following transition matrix:10 0 012 2010102 2P= 0 0 1 0 001 1 1 1:4 :4 :4 :410 0 012 2Which classes are closed?12 1. Discrete-timeMarkovchains1.2.2 Show thatevery transition matrix on a finite state-space has atleastoneclosedcommunicatingclass. Findanexampleof atransitionmatrixwith no closed communicating class.1.3 Hitting times and absorptionprobabilitiesLet(Xn)n>Obe a Markov chain with transition matrix P. The hitting timeof asubsetAof I istherandomvariableH A:n~ {O,1,2, ... } U {oo}given byHA(w) = inf{n~ 0 : Xn(w) E A}where weagree thatthe infimum of the empty set 0 is00. The probabilitystarting fromi that ( X n ) n ~ O ever hitsAis thenWhen Ais a closed class, hfis called theabsorption probability. The meantime takenfor ( X n ) n ~ O to reachAisgiven bykt = lEi(HA) = 2:nJP>(HA= n) + ooJP>(HA= (0).ni(HA n). SoXiJP>i(HA n) forall nand thenXilimJP>i(HA n) =JP>i(HA. en(r+n+s) > (r) (n) (s) > 0Pjk - Pji Pii Pikforall sufficiently large n. 0Here is the main result of this section. The method of proof, by couplingtwoMarkov chains, isingenious.Theorem1.8.3(Convergence to equilibrium). Let Pbe irreducibleandaperiodic, andsupposethat Phas aninvariant distribution7r. Let Abe any distribution. Supposethatis Markov(A, P). ThenP(Xn= j)7rj as n 00forall j.In particular,---t 'lrj as n ---t 00forall i, j.Proof. We use a couplingargument. Letbe Markov(7r, P) andindependentof Fix areference state b andsetT= inf{n2:: 1 : X n= Yn=b}.Step1. We show P(T for all sufficientlylargen; soPis irreducible. Also, Phas aninvariantdistribution given by7r(i,k) =7ri7rkso, byTheorem1.7.7,P ispositiverecurrent. But T isthefirst passagetime of Wnto(b, b) so P(T 0only if i ECrand j ECr+nfor some r;(ii)> 0 for all sufficiently large n, for all i,j ECr, for all r.Proof. Fix a state k and consider S= {n 0 : > O}. Choose n1, n2ESwithn1 0 or some n.ThenCoU ...U Cd-1=I, byirreducibility. Moreover, if >0 and >0forsomer,S E{O, 1, ... ,d - I}, then, choosingm 0so that > 0, we have >0 and > 0 so r =s by minimalityof d. Hencewe haveapartition.To prove (i) suppose> 0 and i ECr. Choose mso that> 0,then >0 soj ECr+nas required. By takingi =j =kwenowsee that dmust divide every elementof S, in particularn1.Nowfor nd wecanwritend =qn1 + r for integers qn1 ando rn1- 1. Sinceddividesn1we thenhaver = mdfor someintegermand then nd = (q - m)n1 +mn2. HenceandhencendES. Toprove(ii) for i, j ECrchoosem1 andm2sothat > 0 and > 0, then44 1. Discrete-timeMarkovchainswheneverndSince ml +m2isthennecessarilyamultipleof d, weare done. DWe call d the period of P. The theorem just proved shows in particular forall i EI thatd is the greatest common divisor of the set {n 0 : > O}.Thisis sometimesuseful in identifying d.Finally, here is a complete descriptionof limitingbehaviour for irre-ducible chains. This generalizes Theorem1.8.3intworespects sincewerequireneitheraperiodicitynortheexistenceof aninvariant distribution.The argumentwe use for thenull recurrent case was discovered recently byB. FristedtandL. Gray.Theorem 1.8.5. Let Pbe irreducible of period d and let Co, C1 , ...,Cd-lbethepartitionobtainedinTheorem1.8.4. Let AbeadistributionwithLiECo Ai =1. Suppose thatis Markov(A, P). Then for r =0,1, ... ,d - 1 and j ECrwe haveP(Xnd+r= j) dlmj as n 00where mj istheexpectedreturntimeto j. Inparticular, fori ECoandj ECrwe haveProof(nd+r) dlPijmjas n 00.Step1. Wereduce to theaperiodiccase. Set v =Apr, thenbyTheorem1.8.4 wehaveSet Yn = Xnd+r, thenis Markov(v, pd)and, by Theorem 1.8.4, pdisirreducible andaperiodicon Cr. For j ECrtheexpected return time of toj is mjld. Soif the theoremholdsin theaperiodiccase, thenP(Xnd+r = j) = P(Yn = j) dlmj as n 00so the theoremholdsin general.Step 2. Assume that Pis aperiodic. If Pis positive recurrent then 1/mj =1rj, where1r is theuniqueinvariant distribution, so theresult followsfromTheorem1.8.3. Otherwise mj =00 andwehave toshow thatP(Xn = j) 0 as n 00.1.8Convergencetoequilibrium 45If P is transient this is easy and we are left with the null recurrentcase.Step3. Assume thatPisaperiodicandnull recurrent. Then00:EPj(Tj > k) =lEj(Tj)= 00.k=OGiven > 0 chooseKso thatThen, fornK- 1n1 :E P(Xk=j andXm1= j form= k + 1, ...,n)k=n-K+lnL P(Xk = j)Pj(Tj > n - k)k=n-K+lK-l= :E P(Xn-k = j)Pj(Tj >k)k=Oso wemust have P(Xn -k = j)/2forsomekE{O, 1, ... ,K - I}.Return now to the coupling argument used in Theorem 1.8.3, only now let be Markov(j.t, P), wherej.t is to be chosen later. Set Wn = (Xn, Yn).As before, aperiodicityofensures irreducibilityof If istransientthen, on takingj.t =A, we obtainP(Xn= j)2 = P(Wn= (j, j))0as required. Assume then thatis recurrent. Then, in the notationof Theorem 1.8.3, we have P(T 0wasarbitrary, thisshowsthat lP(Xn= j) ~ 0asn ~ 00, asrequired. DExercises1.8.1Provetheclaims (e), (f) and(g) madeinexample(v) of theIntro-duction.1.8.2 Find the invariant distributions of the transition matrices in Exercise1.1.7, parts(a), (b) and(c), andcompare them with youranswersthere.1.8.3 A fairdieis thrown repeatedly. LetXndenote the sum of the first nthrows. FindlimP(Xnisamultiple of 13)n--+-ooquoting carefully anygeneral theorems thatyou use.1.8.4Eachmorningastudent takesoneof thethreebooksheownsfromhis shelf. The probability that he chooses book i isQi, where 0 (Vi ---t 7rias n---t 00forall i)= 1.Given c > 0, chooseJfiniteso thatand thenN= N(w) so that, fornN(w)"I Vi(n) I-n- - 1ri < c/4.iEJThen, fornN(w), wehave< c,which establishes thedesiredconvergence. DWe consider now the statistical problem of estimating an unknown tran-sitionmatrixPon thebasisof observationsof thecorrespondingMarkovchain. Consider, tobegin, the case where we have N+ 1observationsThelog-likelihood functionisgiven byI(P)= log(..\xoPxoxl ,PXN-1XN)= LNij logpiji,jEl56 1. Discrete-timeMarkovchainsup toaconstantindependent of P, whereNijis thenumber of transitionsfrom i to j. A standard statistical procedure is to find themaximumlikeli-hoodestimate P, whichisthechoiceof Pmaximizing l(P). SincePmustsatisfy thelinear constraint EjPij=1 foreach i, we first try tomaximizel (P) + 2:/-LiPiji,jEIandthenchoose (/-li : i EI) tofit theconstraints. ThisisthemethodofLagrangemultipliers. Thuswe findN-l N-lPij = L1{Xn =i,Xn+1=j}/ Ll{xn =i}n=O n=Owhich is theproportion of jumps fromi which go to j.Wenow turn to consider theconsistency of this sortof estimate, thatisto say whether Pij ~ Pij with probability 1 asN ~ 00. Since this is clearlyfalsewhen i istransient, weshall slightlymodifyourapproach. Note thatto findPij wesimply have tomaximize2: Nij logPijjEIsubject to EjPij =1: the other terms and constraints are irrelevant. Sup-pose then thatinstead of N + 1 observations we make enough observationstoensure thechainleaves statei atotal of Ntimes. In thetransient casethismayinvolverestartingthechainseveral times. DenoteagainbyNijthenumberof transitionsfromi to j.Tomaximize thelikelihood for (Pij : j EI) we still maximize2: Nij logPijjEIsubject to EjPij = 1, which leads to themaximumlikelihood estimatePij= Nij/N.But Nij =Y1 + ... +YN, whereYn=1 if the nthtransitionfromi istoj, andYn=0otherwise. BythestrongMarkovpropertyYl, . .. ,YN areindependent andidenticallydistributedrandomvariables withmeanPij.So, by the strong law of largenumbersP(Pij ~ Pij asN ~ 00) =1,which shows thatPij isconsistent.Exercises1.11Appendix: recurrencerelations 571.10.1Prove theclaim(d) madein example(v)of theIntroduction.1.10.2Aprofessor hasNumbrellas. He walks to the office in themorningandwalks homeintheevening. If it is raininghelikestocarryanum-brellaandif it isfinehedoesnot. Supposethat it rainsoneach journeywithprobabilityp, independentlyof past weather. What isthelong-runproportion of journeys on which theprofessor gets wet?1.10.3 Let ( X n ) n ~ O be an irreducible Markov chain on I having an invariantdistributionJr. ForJ ~ I let ( Y m ) m ~ O be theMarkov chain onJobtainedby observing( X n ) n ~ O whilst in J. (See Example 1.4.4.) Show that ( Y m ) m ~ Oispositive recurrent andfinditsinvariant distribution.1.10.4Anoperasinger isduetoperformalongseriesof concerts. Hav-inga fine artistictemperament, sheis liabletopullout eachnight withprobability1/2. Oncethishashappenedshewill not singagainuntil thepromoter convinces her of hishigh regard. Thishedoes by sending flowersevery day until she returns. Flowers costing xthousand pounds, 0~ x ~ 1,bring about areconciliation with probabilityyIX. Thepromoter stands tomake 750fromeachsuccessful concert. Howmuchshouldhespendonflowers?1.11Appendix: recurrencerelationsRecurrence relations often arise in the linear equations associated to Markovchains. Hereisanaccount of thesimplest cases. Amorespecializedcasewasdealt withinExample1.3.4. InExample1.1.4wefoundarecurrencerelation of theformXn+l = aXn +b.Welookfirst for aconstant solutionX n= x; thenx = ax +b, soprovideda =I 1wemust have x =b/(l - a). NowYn =Xn- b/(l - a) satisfiesYn+l =aYn, soYn=anyo. Thusthegeneralsolutionwhena=I 1 isgivenbyX n= Aan +b/ (1 - a)whereAisaconstant. Whena =1 thegeneral solution isobviouslyXn= Xo +nb.In Example1.3.3 wefoundarecurrencerelation of theformaXn+l +bXn +CXn-l= 058 1. Discrete-timeMarkovchainswhere a and c were both non-zero. Let us try a solution of the formX n=-Xn;thena-X2+ b-X + c = O. Denote by0: and{3 the roots of this quadratic. ThenYn= Ao:n+B{3nisasolution. If0: =1=(3 then we can solve the equationsxo=A+B, xl=Ao:+B{3so that Yo= XoandYl = Xl; butforall n, so by inductionYn= Xn forall n. If0: = {3 # 0, thenYn= (A +nB)o:nisasolution andwe can solveso that Yo=XoandYl =Xl; then, by thesameargument, Yn=Xnforalln. Thecase0: =(3 =0doesnot arise. Hencethegeneral solutionisgivenby{Ao:n + B{3nXn= (A + nB)anif0: #{3if0: ={3.1.12Appendix: asymptoticsfor n!Ouranalysisof recurrenceandtransienceforrandomwalksinSection1.6restedheavily on theuse of theasymptoticrelationn! rv Ayri(nje)n as n~ 00forsome AE[1, 00). Hereisaderivation.Wemakeuse of thepower series expansionsfor It I 0),starting fromo.00 1(i)IfI: - 0jEIthe chain is foundat 00 with probability Pi 00 (t). The semigroup P(t)is re-ferredtoas thetransitionmatrix of thechain andits entries Pij (t) are thetransitionprobabilities. Thisdescriptionimplies that forall h >0 thedis-crete skeleton is Markov(.x, P(h)). Strictly, in the explosive case,that is, when P(t) is strictly sub-stochastic, we should say P(h)),where and P(h)are defined on IU{oo}, extending.x and P(h)by = 0and Pooj(h) = O. Butthereisnodangerof confusioninusing thesimplernotation.The information coming from these two descriptions is sufficient for mostof theanalysisof continuous-timechainsdoneinthischapter. Notethatwe have notyetsaid how the semigroup P(t)is associated to theQ-matrixQ, except viatheprocess! Thisextrainformationwill berequiredwhenwediscussreversibilityinSection3.7. Sowerecall fromSection2.8thatthesemigroupischaracterizedas theminimal non-negativesolution of thebackwardequationP'(t) = QP(t), P(O) =Iwhich reads in components= LqikPkj(t), Pij(O) = 8ij .kEIThe semigroupis alsothe minimal non-negative solutionofthe forwardequationP'(t) = P(t)Q, P(O) = I.In the case where I is finite, P(t)is simply the matrix exponential etQ,andis theunique solution of thebackwardand forwardequations.3.2Classstructure3.2Class structure111A firststep in theanalysis of a continuous-time Markov chainis toidentifyitsclassstructure. Weemphasise that wedeal only withminimalchains, thosethat dieafterexplosion. Thentheclassstructureissimplythe discrete-time class structure of the jump chain as discussed inSection1.2.We say thati leadstoj andwrite ij iflPi(Xt= j forsome t0) > o.Wesayi communicates withj andwriteij if bothij andj i. Thenotions of communicatingclass, closedclass, absorbingstateandirreducibility are inherited from the jump chain.Theorem3.2.1. For distinctstates i and j thefollowing are equivalent:(i) i (ii) ij forthe jump chain;(iii) Qioilqili2 ... Qin-lin>0for somestatesio,iI, ... ,in withio=i,in = j;(iv) Pij(t) > 0forall t > 0;(v) Pij(t)>0forsome t > o.Proof. Implications (iv) => (v) => (i) => (ii) areclear. If (ii) holds, thenbyTheorem1.2.1, therearestatesio, iI, ... ,inwithio=i, in=j and'1rioil '1rili2 ... '1rin -lin> 0, which implies(iii). If qij > 0, thenforall t > 0, so if (iii) holds, thenPij (t)Pioil (tin) ... Pin-lin (tin)>0forall t > 0, and(iv) holds. DCondition(iv)of Theorem 3.2.1 shows that the situation is simpler thanin discrete-time, where itmay be possible to reach a state, butonly after acertain lengthof time, and thenonlyperiodically.3.3 Hitting times and absorption probabilitiesLet (Xt)t>obeaMarkov chain with generator matrix Q. Thehittingtimeof asubset Aof I is therandom variableDAdefinedby112 3. Continuous-timeMarkovchains IIwiththeusual convention that inf 0 =00. Weemphasisethat (Xt)t>oisminimal. So if H A is thehitting timeof Afor the jump chain, then -{HA0for all i,j soP(h) isirreducibleandaperiodic. ByTheorem 3.5.5, A is invariantfor P(h). So, by discrete-time convergence toequilibrium, forall i, jThus wehavealattice of points along which the desired limit holds; we fillinthegaps usinguniformcontinuity. Fixastatei. Given >0wecanfindh >0 so thatfor0 shand then findN, so thatforn2: N.For t 2: Nhwehave nh t 0ifCj = 0,i =lEi(L c(Yn) +!(YN)lN 0JP>( {t ~ 0 : IBt I o be a Brownian motion in jRd and letf : jRd~ jRbe acontinuous periodicfunction, so thatf(x +z) =f(x) forallzE Zd.Thenforall xEjRd, as t ~ 00, we havelEx[f(Bt )] -t 1=[ f(z)dzJ[O,l]dand, moreoverJIx ( ~it f(Bs)ds -t 1ast -t 00) = 1.Thegenerator ! ~ of BrownianmotioninjRdreappearsasit shouldinthefollowingmartingale characterization of Brownianmotion.4.4Brownianmotion 169Theorem4.4.6. Letbea continuous ]Rd-valuedrandompro-cess. Write(Ft)t2::o for thefiltrationof Thenthefollowingareequivalent:(i) (Xt)t2::ois a Brownian motion;(ii) for all bounded functions f which are twice differentiable withbounded second derivative, thefollowing process isa martingale:This result obviously corresponds to Theorem 4.1.2. In case you are unsure,acontinuous timeprocess (Mt)t2::o is amartingaleif it is adaptedtothegiven filtration (Ft)t2::o, if JEIMtl (Xn -1 = k) = lE((t)xn-1).k==OHence, by induction,we find that E(tXn) = (n)(t), where (n) is the n-foldcomposition0 ... 0 . Inprinciple, this gives theentiredistributionofXn, though(n) maybearather complicatedfunction. Somequantitiesare easilydeduced: wehaveE(Xn)= limdd lE(tXn) = limdd (n)(t) =(lim'(t)r =p,n,til t til t tilwhere J-l= E(N); alsoso, since0isabsorbing, wehaveq = P(Xn = 0forsomen) = lim(n)(o).n--+-ooNow (t) isaconvexfunctionwith (1) =1. Let ussetr= inf{t E[0,1]:(t) = t}, then (r)=r bycontinuity. Sinceisincreasing and0~ r, wehave (O) ~ r and, by induction, (n)(o) ~ r forall n, henceq~ r. On theotherhandq = lim(n+l) (0) = lim((n) (0)) = (q)n --+- 00 n --+- 00soalsoq2:: r. Henceq =r. If '(1) >1 thenwemust haveq..t, sincethe customers inthe queue at time 0are preciselythose whoarrivedduringthequeueingandservicetimesof thedepartingcustomer.HenceG(z)= E(e-A(Q+T)(l-Z)) =M(A(l- z))L(A(l- z))whereMis theLaplace transformOn substituting for G(z)we obtain the formulaM(w)= (1 - p)w/(w - .x(1 - L(w))).Differentiation and l'Hopital's rule, as above, lead to a formula for the meanqueueingtimeE(Q)= -M'(O+)= .xL"(0+)2 (1 +AL' (0+)) 2Wenow turn to thebusyperiod 8. Consider theLaplace transformLet Tdenote the service time of the first customer in the busy period. Thenconditional on T= t, wehave8=t+81 + ... +8N,whereN isthenumber ofcustomers arrivingwhilethefirst customer isserved, whichis Poissonofparameter At, andwhere81,82 , ... areinde-pendent, with the same distribution as 8. HenceB(w)= 100E(e-WSIT =t)dF(t)= 100e-wte->.t(l-B(wdF(t) = L(w + .x(1- B(w))).5.2Queuesandqueueingnetworks 191Although thisisanimplicit relation for B(w), wecan obtainmomentsbydifferentiation:E(S) = -B'(O+) = -'(0+)(1 - AB'(O+)) = Jl(l +AE(S))so themeanlengthof thebusy period isgiven byE(S) = Jl/(l - p).Example5.2.8(M/G/ooqueue)Arrivals atthisqueueformaPoisson process, of rateA, say. Service timesareindependent, witha commondistributionfunctionF(t) = lP(T ~ t).Thereareinfinitelymanyservers, soall customersinfact receiveserviceat once. Theanalysis here is simpler thaninthe last example becausecustomers do not interact. Suppose there are no customers at time O.What, then, isthedistribution of thenumberX tbeing served attime t?Thenumber Nt ofarrivals bytimet is a Poissonrandomvariableofparameter At. We condition on Nt = nand label the times of the narrivalsrandomlybyAI, ... ,An. Then, byTheorem2.4.6, AI, ... ,Anareinde-pendent anduniformlydistributedon theinterval [0, t]. Foreachof thesecustomers, serviceisincomplete attime t with probability1 it 1 itp=- JP>(T>s)ds=- (l-F(s))ds.tot 0Hence, conditional on Nt = n, X tis binomial of parameters nand p. Then00P(Xt= k)= LP(Xt= k I Nt =n)P(Nt=n)n==O00= e->.t(>.pt)k /k! L('\(1 - p)tt-k/(n - k)!n==k= e-APt(Apt)k /k!So wehave shown thatX tisPoisson of parameter,\ it(1 - F(s))ds.192Recall that5. ApplicationsHenceif E(T)