Discrete Time Markov Chains

EECS 126 Fall 2019

October 15, 2019

1 / 24

Agenda

Announcements

IntroductionRecap of Discrete Time Markov Chainsn-step Transition Probabilities

Classification of StatesRecurrent and Transient StatesDecomposition of StatesGeneral Decomposition of StatesPeriodicity

Stationary DistributionsDefinitionsBalance Equations

2 / 24

Announcements

I Homework 7 due Tomorrow night (10/16)!

I Lab self-grades due on Monday night (10/21).

3 / 24

Recap of Discrete Time Markov Chains

Figure: Example of a Markov chain

I State changes at discrete times

I State Xn belongs to a finite set S (for now)

I Satisfies the Markov property for transitions from state i ∈ Sto state j ∈ S

P(Xn+1 = j | Xn = i ,Xn−1 = xn−1 . . .X1 = x1)

= P(Xn+1 = j | Xn = i) = pij

Where, pij ≥ 0,∑

j pij = 1

I Time homogeneous: the evolution of the system or transitionprobabilities are time independent

4 / 24

Recap of Discrete Time Markov Chains

The probability transition matrix P contains all the informationabout transitions between different states

P =

p11 p12 . . . p1n

p21 p22 . . . p2n...

......

...pn1 pn2 . . . pnn

Let π(n) =

[P(Xn = 1) . . . P(Xn = m)

]then

π(n+1) = π(n)P

⇒ π(n) = π(0)Pn

5 / 24

Example

q1 q2 q3

a

1-a

b

1-b

c

1-c

π0 =[0.3 0.3 0.4

]I Write the probability transition matrix PI What is P(X0 = q1,X1 = q2,X3 = q1)?

I What is P(X0 = q1,X1 = q1,X2 = q2,X3 = q3,X4 = q3)?

6 / 24

Answers

q1 q2 q3

a

1-a

b

1-b

c

1-c

I P =

a 1− a 00 b 1− b0 c 1− c

I P(X0 = q1,X1 = q2,X3 = q1) = 0. You cannot go to q1 from

q2.

I Use the Markov Property.

P(X0 = q1,X1 = q1,X2 = q2,X3 = q3,X4 = q3)

= P(X0 = q1) · P(X1 = q1 | X0 = q1) . . .P(X4 = q3 | X3 = q3)

= 0.3 · a · (1− a) · (1− b) · (1− c)

7 / 24

n-step Transition Probabilities

Let rij(n) = P(Xn = j | X0 = i) represent the probability that youare in state j exactly n steps after reaching state i . The value ofrij(n) can be calculated recursively as

rij(n) =∑k∈S

rik(n − 1)pkj

Observe that rij(1) = pij .

⇒ rij(n) = Pni ,j

the i , j entry of Pn.

8 / 24

Recurrent and Transient States

I Accessible: State j is accessible or reachable from state i if∃n ∈ N such that rij(n) > 0.

I Recurrence: A state i is recurrent if ∀j reachable from i , i isreachable from j . That is if A(i) is the set of reachable statesfrom i , then i is recurrent if ∀j ∈ A(i)⇒ i ∈ A(j).

I Transient: A state i is transient if it is not recurrent.

I Classify the states in the below Markov chain as recurrent ortransient.

q1 q2 q3

a

1-a

0.5

0.5

0.3

0.7

9 / 24

Answer

q1 q2 q3

a

1-a

0.5

0.5

0.3

0.7

I If a = 1, q1, q2, q3 are recurrent.

I If a < 1, q1 is transient and q2, q3 are recurrent.

10 / 24

Decomposition of States

I Recurrent Class: For any recurrent state i , all states A(i)(the set of states reachable from i) form a recurrent class.Any Markov chain can be decomposed into one ore morerecurrent classes.

I A state in a recurrent class is not reachable from states in anyother recurrent class (try to prove this).

I Transient states are not reachable from a recurrent state.Moreover, from every transient state atleast one recurrentstate is reachable.

I Find the recurrent classes in the following MC:

q1 q2 q3

a

1-a

0.5

0.5

0.3

0.7

11 / 24

Answers

q1 q2 q3

a

1-a

0.5

0.5

0.3

0.7

I If a = 1, {q1}, {q2, q3} form two recurrent classes.

I If a < 1, q1 is transient, and {q2, q3} form a recurrent class.

12 / 24

General Decomposition of States

A Markov chain is called irreducible if it only has one recurrentclass. For any non-irreducible Markov chain, we can identify therecurrent classes using the following process

I Create directed edges between any two nodes that have anon-zero transition probability between them.

I Find the strongly connected components of the graph.

I Use transitions between different strongly connectedcomponents to further topologically sort the graph.

I Each strongly connected component at the bottom of thetopologically sorted structure forms a recurrent class. Allother nodes in this final structure are transient.

13 / 24

Example

q1 q2 q3

q4 q5

0.1

0.9

0.4

0.5

0.3

0.6

0.1 0.1

0.5

0.3

0.5 0.7

14 / 24

Periodicity

Consider an irreducible Markov chain. Define

d(i) := g.c.d.{n ≥ 1|rii (n) > 0}

I Remember: rij(n) = P(Xt+n = j |Xt = i)

I ”All paths back to i take a multiple of d(i) steps”

I Fact: ∀i d(i) is the same.

I Fact: for Markov chains with more than one recurrent class,each class has a separate value for d

I We define a Markov chain as aperiodic if d(i) = 1∀i .I Otherwise, we say it’s periodic with period d

15 / 24

Periodicity Examples

Are the following Markov chains aperiodic?

1 2 3

1

0.5

0.5

1

I d = 2, so this is periodic.

1 2 3

1

0.5

0.5

0.99

0.01

I d = 1, so this is aperiodic.

I Adding a self loop will make an irreducible Markov chainaperiodic!

16 / 24

Periodicity Examples (continued)

We won’t particularly worry about periodicity/aperiodicity forMarkov chains with more than 1 recurrent class.

1 2 3

1

0.5

0.5

1

17 / 24

Stationary DistributionIf we choose the initial state of the Markov chain according to thedistribution

P(X0 = j) = π0(j) ∀j

and this impliesP(Xn = j) = π0(j) ∀j , n

then we say that π0 is stationary. The balance equations aresufficient for stationarity:

π0(j) =m∑

k=1

π0(k)pkj ∀j

I The balance equations can be written as π0 = π0P. In linearalgebra terms, π0 is a left eigenvector of P that hascorresponding eigenvalue λ = 1

I In general, there can be multiple unique stationarydistributions.

18 / 24

Stationary Distribution Example

1 20.25

0.75

0.25

0.75

Let’s try π0 = [1, 0].

π1(1) = P(X1 = 1|X0 = 1)π0(1) + P(X1 = 1|X0 = 2)π0(2) (1)

= (0.25)(1) + (0.25)(0) (2)

= 0.25 (3)

Similarly,

π1(2) = P(X1 = 2|X0 = 1)π0(1) + P(X1 = 2|X0 = 2)π0(2) (4)

= (0.75)(1) + (0.75)(0) (5)

= 0.75 (6)

π1 = [0.25, 0.75] 6= π0, so π0 = [1, 0] is not stationary.19 / 24

Stationary Distribution Example (continued)

1 20.25

0.75

0.25

0.75

Let’s solve for the stationary distribution. Let π0 = [x , 1− x ].

x = P(X1 = 1|X0 = 1)x + P(X1 = 1|X0 = 2)(1− x) (7)

= 0.25x + 0.25(1− x) (8)

1− x = P(X1 = 2|X0 = 1)x + P(X1 = 2|X0 = 2)(1− x) (9)

= 0.75x + 0.75(1− x) (10)

We see that 1− x = 3x , so x = 0.25. Our stationary distribution isπ0 = [0.25, 0.75]

20 / 24

Probability Flow Interpretation

i. . .

π(1)p1i

π(m)pmi

. . .

π(i)pi1

π(i)pim

π(i)pii

I For any distribution, probability mass flows in and out of everystate at each step.

I By subtracting π(i)pii from both sides of the balanceequation, we have:∑

j 6=i

π(j)pji︸ ︷︷ ︸flow in

= π(i)∑j 6=i

pij︸ ︷︷ ︸flow out

∀i

21 / 24

Example Revisited

1 21-a

a

b

1-b

Let π0 = [x , 1− x ]. ∑j 6=i

π(j)pji = π(i)∑j 6=i

pij (11)

Using this at state 2,

xa = (1− x)b (12)

x =b

a + b(13)

1− x =a

a + b(14)

22 / 24

The Big Theorem

I If a Markov chain is finite and irreducible, it has a uniqueinvariant distribution π and π(i) is the long term fraction oftime that X (n) is equal to i , almost surely.

I If the Markov chain is also aperiodic, then the distribution ofX (n) converges to π.

23 / 24

References

Introduction to probability. DP Bertsekas, JN Tsitsiklis - 2002

24 / 24