A note on circular Markov chains

Statistics & Probability Letters 47 (2000) 301–306

A note on circular Markov chains

Jos�e Luis Palacios ∗

Departamento de Matem�aticas, Centro de Estadist. y Software Mat., Universidad Sim�on Bol ��var,Apartado 89,000, Caracas, 1080-A, Venezuela

Received January 1999; received in revised form September 1999

Abstract

Using the electric network approach, we give closed-form formulas for the stationary probabilities and expected hittingtimes in balanced circular Markov chains. As an application, we give a closed-form formula for the duration of play in thegeneral ruin problem, where the probabilities of winning a particular game depend on the amount of the current fortune.c© 2000 Elsevier Science B.V. All rights reserved

MSC: primary 60J15; secondary 60C05

Keywords: E�ective resistance; Circular Markov chains; Duration of play

1. Introduction

Since the appearance of the book of Doyle and Snell (1984), a great deal of attention has been devoted tothe relation between electric networks and random walks on graphs. In particular, the computation of stationarydistributions and expected hitting times sometimes is greatly simpli�ed by this electric approach, which consistsof thinking of the edge between vertices i and j as a resistor with resistance rij (or conductance Cij = 1=rij);then we can de�ne the random walk on the connected undirected graph G = (V; E), as the Markov chainXn; n¿0, that from its current vertex v jumps to the neighboring vertex w with probability pvw = Cvw=C(v),where C(v)=

∑w:w∼v Cvw, and w ∼ v means that w is a neighbor of v. We can assign a (�ctitious) conductance

Czz from a vertex z to itself, giving rise to a transition probability from z to itself. Some notation: EaTb isthe expected value, starting from the vertex a, of the hitting time Tb of the vertex b; �z is the stationaryprobability of the chain at the vertex z; Rab is the e�ective resistance, as computed by means of Ohm’s law,between vertices a and b.

∗ Correspondence address: Departamento de C�omputo Cient��co y Estad��stica, Universidad Sim�on Bol��var, Apartado 89,000, Caracas,Venezuela. Tel.: +58-2-906-3233; fax:+58-2-906-3232.E-mail address: [email protected] (J.L. Palacios)

0167-7152/00/$ - see front matter c© 2000 Elsevier Science B.V. All rights reservedPII: S0167 -7152(99)00170 -4

302 J.L. Palacios / Statistics & Probability Letters 47 (2000) 301–306

In this context we have:

Theorem 1.1. For a random walk on a �nite graph

�z =C(z)∑z C(z)

(1)

and

EaTb =12

∑z

C(z)[Rab + Rbz − Raz]; (2)

where C(z) = Czz +∑

x:z∼x Czx and where the Czz’s do not intervene in the computation of the e�ectiveresistances.

Derivations of (1) and (2) can be read, respectively, in Doyle and Snell (1984) and Palacios and Tetali(1996).A circular Markov chain has as state space the set of integers {1; 2; : : : ; N} and transitions from a state

i are only allowed into states i − 1; i or i + 1; 16i6N ; here and in what follows, addition is supposedto be modulo N . In a recent note, Adan and Resing (1997) gave a closed-form formula for the stationarydistribution of circular Markov chains, derived with a clever method of reduction of states. In this note weuse the electric approach spelled out in Theorem 1.1 in order to give an alternative closed-form formula forthat stationary distribution of balanced circular Markov chains. Though this method has the drawback of notcovering all cases of circular chains, it has the advantage of providing formulas for the hitting times.

2. The stationary distribution

The �rst question that arises is: what circular Markov chains can be described as random walks on thecircle? If in the previous question we replace “circular” with “birth and death” and “circle” with “line”, weshowed in Palacios and Tetali (1996) that the answer is “all of them”. The answer in the current context isless general: “the balanced ones” which satisfy condition (3) below. More precisely, let pi; qi and si be thetransition probabilities from i to, respectively, i+ 1; i− 1 and i, 16i6N , with pi + qi + si = 1 for all i, and∏i pi

∏i qi 6= 0, then we �nd conductances Ci = Ci; i+1 and Cii; 16i6N , as follows:

Proposition 2.1. A circular Markov chain can be represented as a random walk on the circle graph if andonly if∏

i

pi =∏i

qi: (3)

In that case; one may de�ne the conductances thus: C1 is arbitrary;

Cj =p2p3 · · ·pjq2q3 · · · qj C1; 26j6N; (4)

and

Cjj =sjpjCj; 16j6N: (5)

Proof. The relationship between probabilities and conductances is given by the set of equations

pk =Ck

Ck + Ckk + Ck−1; 26k6N; (6)

J.L. Palacios / Statistics & Probability Letters 47 (2000) 301–306 303

qk =Ck−1

Ck + Ckk + Ck−1; 26k6N (7)

and the pair of equations that “close the loop”:

p1 =C1

C1 + C11 + CN; (8)

q1 =CN

C1 + C11 + CN: (9)

Now, solving for Ckk in (7) we get

Ckk =1− qkqk

Ck−1 − Ck; 26k6N (10)

and replacing (10) in (6) we get

Ck =pkqkCk−1; 26k6N; (11)

from which we get recurrence (4). Replacing (11) back in (10) we get

Ckk =skpkCk ; 26k6N:

Finally, we take C11 = (s1=p1)C1. We need condition (3) in order to have CN = (p2 · · ·pN =q2 · · · qN )C1 =(q1=p1)C1, thus satisfying simultaneously Eqs. (8) and (9).

Using (4) and (5) of the above proposition and some algebra, we get C(k) = (p2 · · ·pk=q2 · · · qk)(1=pk).Using this fact and Eq. (1) we obtain the following.

Proposition 2.2. The stationary distribution of a balanced circular Markov chain is

�k =(p2 · · ·pk=q2 · · · qk)(1=pk)∑k(p2 · · ·pk=q2 · · · qk)(1=pk)

; 16k6N; (12)

where for k = 1 the quotient p2 · · ·pk=q2 · · · qk is to be taken as 1.

3. The hitting times

Without loss of generality, we shall consider the expected hitting times E1Tj for 26j6N . We recall thatthe e�ective resistance of resistors connected in series equals the sum of the individual resistances, whereasthe e�ective resistance of resistors connected in parallel equals the inverse of the sum of the inverses ofthe individual resistances. On account of these facts, and because of the circular nature of the circuit underconsideration, it is not di�cult to see that we have

R1j + Rjz − R1z =

2∑i ri(rz + · · ·+ rj−1)(rj + · · ·+ rN ); 16z¡ j;

0 z = j;2∑i ri(r1 + · · ·+ rj−1)(rj + · · ·+ rz−1); j ¡ z6N:

(13)


When (13) is inserted into (2) we get the following.

Proposition 3.1. The expected hitting times between states 1 and j of a balanced circular Markov chainare given by

E1Tj =1∑i ri

[(rj + · · ·+ rN )

j−1∑z=1

C(z)(rz + · · ·+ rj−1)

+(r1 + · · ·+ rj−1)N∑

z=j+1

C(z)(rj + · · ·+ rz−1) ; (14)

where C(z) = (p2 · · ·pz=q2 · · · qz)(1=pz); and where the second summation equals 0 in case j = N .

Example 1. On the square with vertices labelled 1 through 4, consider the transition probability matrix

0 0:4 0 0:6

0:2 0:2 0:6 0

0 0:5 0 0:5

0:2 0 0:4 0:4

;

for which it is easy to see that condition (3) holds. Formulas (4) and (5) show that choosing C1 = 1 weobtain C2 = C3 = C44 = 3; C22 = 1 and C4 = 3

2 . Also, r1 = 1; r2 = r3 =13 ; r4 =

23 .

By formula (12), the stationary distribution is � = [ 542 ;521 ;

27 ;

514 ] and by formula (14), the expected hitting

time E1T3 equals

1∑i ri[(r3 + r4)(C(1)(r1 + r2) + C(2)r2) + (r1 + r2)r3C(4)] =

257:

4. The duration of play in the ruin problem

The ruin problem is a classic problem of probability theory: player I with an initial capital of x dollarsplays a series of games against player II with a capital of N − x dollars (for convenience, x and N arenonnegative integers); if the current capital of player I is k, he then wins from his opponent one dollar, orloses one dollar to his opponent, or stays with the same capital with probabilities, respectively, pk; qk andsk = 1−pk − qk . With this general setup (the classical textbook cases are pk = qk = 1

2 and pk =p= 1− qk)we gave in Palacios (1999) a closed-form formula, with the electric approach, for the ruin probability thatplayer I eventually wins all his opponent’s money. Now, as an application of the previous section, we givea closed-form formula for the duration of play in this general ruin problem. The duration of play, a problemwith a long and illustrious history (see Thatcher, 1970), refers to the expected number of games played untileither player I or player II wins all their opponent’s money.In Palacios (1999) we showed how the general ruin problem described above can be represented as a

random walk on the linear graph with vertices 0; 1; : : : ; N and conductances Ck; 16k6N between verticesk − 1 and k given by

Ck =p1 · · ·pk−1q1 · · · qk−1 C1; 26k6N; (15)

J.L. Palacios / Statistics & Probability Letters 47 (2000) 301–306 305

where C1 is arbitrary; we also need conductances from any vertex to itself given by

Ckk =skqkCk ; 16k6N: (16)

Note that (15) and (16) are similar to (4) and (5).Now it should be clear that the duration of play D(x), starting with a capital x, is the same as the hitting

time, starting from the vertex x, of the vertex that results from the fusing together (or shorting) of vertices0 and N . This shorting produces a circular balanced Markov chain. All we have to do now is to translateformula (14) in terms of the resistances and conductances on the linear graph. What we �nally obtain is

D(x) = ExT0;N =1∑i ri

[(r1 + · · ·+ rx)

N−1∑z=x

C(z)(rz+1 + · · ·+ rN )

+(rx+1 + · · ·+ rN )x−1∑z=1

C(z)(r1 + · · ·+ rz)]; (17)

where

rk =1Ck=q1 · · · qk−1p1 · · ·pk−1 ; 26k6N;

we may take r1 = 1 for convenience,

C(j) =p1 · · ·pj−1q1 · · · qj−1

1qj; 26j6N;

and C(1) = 1=q1.

Example 2. When (17) is applied to the particular classical case where ri = 1 for all i, we obtain

D(x) =1N

[2x

N−1∑z=x

(N − z) + 2(N − x)x−1∑z=1

z

]= x(N − x)

as is well known.

Example 3. Consider the ruin problem with N=4 and probabilities p1= 15 ; q1=

25 ; p2=

25 ; q2=

35 ; p3=

35 ; q3=

15 .

Then choosing C1 = 1 we obtain C4 =C11 = 1; C3 =C33 = 13 ; C2 =

12 . One notices that these conductances are

the same as those in Example 1 except for a relabelling of the vertices and a factor of 3. Thus, �nding D(2)is equivalent to �nding E1T3 in Example 1. Formula (17) can alternatively be used to check that, indeed,D(2) = 25

7 .

5. Final comments

A question worth answering is: which Markov chains can be described as random walks on connectedweighted graphs, to which we can apply the electric approach as we did in this note? In the context of �nitechains the answer is: all those irreducible chains which are reversible, that is, those satisfying the detailedbalance equation

�ipij = �jpji; (18)

where � is the stationary distribution, for all i; j. Then either side of Eq. (18) can be taken as the conductancebetween i and j. The in�nite case is far more complicated. Details of the latter case can be found in Aldousand Fill (1999, Chapter 15).


Of course, given that a Markov chain is of the type described in the previous paragraph, whetherEqs. (1) and (2) lead to closed-form formulas like (12) and (14) or not, depends on the particular formof the underlying electric circuit, which may or may not lend itself to easy computations of conductances ande�ective resistances.

References

Adan, I., Resing, J., 1997. Steady state probabilities for circular Markov chains. Math. Scientist 22, 27–31.Aldous, D.J., Fill, J., 1999. Reversible Markov chains and random walks on graphs. Book draft available at www.stat.berkeley.edu/users/aldous/book.html

Doyle, P.G., Snell, J.L., 1984. Random Walks and Electrical Networks. The Mathematical Association of America, Washington, DC.Palacios, J.L., Tetali, P., 1996. A note on expected hitting times for birth and death chains. Statist. Probab. Lett. 30, 119–125.Palacios, J.L., 1999. The ruin problem via electric networks. Amer. Statist. 53, 67–70.Thatcher, A.R., 1970. A note on the early solutions of the problem of the duration of play. In: Pearson, E.S., Kendall, M.G. (Eds.),Studies in the Theory of Statistics and Probability. Gri�n, London.

Documents

A note on circular Markov chains