MARKOV CHAIN chap4

  • View
    218

  • Download
    0

Embed Size (px)

DESCRIPTION

MARKOV CHAIN

Text of MARKOV CHAIN chap4

  • Markov Chain

    Stochastic process (discrete time):{X1, X2, ..., }

    Markov chain Consider a discrete time stochastic process with

    discrete space. Xn {0, 1, 2, ...}. Markovian property

    P{Xn+1 = j | Xn = i,Xn1 = in1, ..., X0 = i0}= P{Xn+1 = j | Xn = i} = Pi,j

    Pi,j is the transition probability: the probability ofmaking a transition from i to j.

    Transition probability matrix

    P =

    P0,0 P0,1 P0,2 P1,0 P1,1 P1,2

    .

    .

    .

    .

    .

    .

    .

    .

    .

    .

    .

    .

    Pi,0 Pi,1 Pi,2 ...

    .

    .

    .

    .

    .

    .

    .

    .

    .

    1

  • Example

    Suppose whether it will rain tomorrow depends onpast weather condition ony through whether it rainstoday. Consider the stochastic process {Xn, n = 1, 2, ...}

    Xn =

    {0 rain on day n1 not rain on day n

    P (Xn+1|Xn, Xn1, ..., X1) = P (Xn+1 | Xn) State space {0, 1}. Transition matrix:(

    P0,0 P0,1P1,0 P1,1

    )

    P0,0 = P (tomorrow rain|today rain) = . Then P0,1 =1 .

    P1,0 = P (tomorrow rain|today not rain) = . ThenP1,1 = 1 .

    Transition matrix: ( 1 1

    )

    2

  • Transforming into a Markov Chain

    Suppose whether it will rain tomorrow depends onwhether it rained today and yesterday.

    P (Xn+1|Xn, Xn1, ..., X1) = P (Xn+1|Xn, Xn1). Theprocess is not a first order Markov chain.

    Define Yn:

    Yn =

    0 Xn = 0, Xn1 = 0 RR1 Xn = 0, Xn1 = 1 NR2 Xn = 1, Xn1 = 0 RN3 Xn = 1, Xn1 = 1 NN

    P (Yn+1|Yn, Yn1, ...)

    = P (Xn+1, Xn|Xn, Xn1, ...)= P (Xn+1, Xn|Xn, Xn1)= P (Yn+1|Yn)

    {Yn, n = 1, 2, ...} is a Markov chain.

    P0,0 P0,1 P0,2 P0,3P1,0 P1,1 P1,2 P1,3P2,0 P2,1 P2,2 P2,3P3,0 P3,1 P3,2 P3,3

    3

  • P0,1 = P (Yn+1 = 1|Yn = 0) = P (Xn+1 = 0, Xn =1|Xn = 0, Xn1 = 0) = 0.

    P0,3 = P (Yn+1 = 3|Yn = 0) = P (Xn+1 = 1, Xn =1|Xn = 0, Xn1 = 0) = 0.

    Similarly, P1,1 = P1,3 = 0, P2,0 = P2,2 = 0, P3,0 =P3,2 = 0.

    Transition matrix

    P0,0 0 P0,2 0P1,0 0 P1,2 00 P2,1 0 P2,30 P3,1 0 P3,3

    =

    P0,0 0 1 P0,0 0P1,0 0 1 P1,0 00 P2,1 0 1 P2,10 P3,1 0 1 P3,1

    The Markov chain is specified by P0,0, P1,0, P2,1, P3,1.1. P0,0 = P (tomorrow will rain|today rain, yesterday rain).2. P1,0 = P (tomorrow will rain|today rain, yesterday not rain).3. P2,1 = P (tomorrow will rain|today not rain, yesterday rain).4. P3,1 = P (tomorrow will rain|today not rain, yesterday not rain).

    4

  • Chapman-Kolmogorov Equations

    Transition after nth steps:P ni,j = P (Xn+m = j | Xm = i).

    Chapman-Kolmogorov Equations:

    P n+mi,j =k=0

    P ni,kPmk,j, n,m 0 for all i, j.

    Proof (by Total probability formula):P n+mi,j = P (Xn+m = j|X0 = i)

    =k=0

    P (Xn+m = j,Xn = k|X0 = i)

    =k=0

    P (Xn = k|X0 = i)

    P (Xn+m = j|Xn = k,X0 = i)=

    k=0

    P ni,kPmk,j

    5

  • n-step transition matrix:

    P(n) =

    P n0,0 Pn0,1 P

    n0,2

    P n1,0 Pn1,1 P

    n1,2

    .

    .

    .

    .

    .

    .

    .

    .

    .

    .

    .

    .

    P ni,0 Pni,1 P

    ni,2

    .

    .

    .

    .

    .

    .

    .

    .

    .

    .

    .

    .

    Chapman-Kolmogorov Equations:P

    (n+m) = P(n) P(m), P(n) = Pn. Weather example:

    P =

    ( 1 1

    )=

    (0.7 0.30.4 0.6

    )Find P (rain on Tuesday | rain on Sunday) andP (rain on Tuesday and rain on Wednesday | rain on Sunday).Solution:

    P (rain on Tuesday | rain on Sunday) = P 20,0P

    (2) = P P =(

    0.7 0.30.4 0.6

    )(

    0.7 0.30.4 0.6

    )

    =

    (0.61 0.390.52 0.48

    )P (rain on Tuesday | rain on Sunday) = 0.61

    6

  • P (rain on Tuesday and rain on Wednesday | rain on Sunday)= P (Xn = 0, Xn+1 = 0 | Xn2 = 0)= P (Xn = 0|Xn2 = 0)P (Xn+1 = 0|Xn = 0, Xn2 = 0)= P (Xn = 0|Xn2 = 0)P (Xn+1 = 0|Xn = 0)= P 20,0P0,0= 0.61 0.7 = 0.427

    7

  • Classification of States

    Accessible: State j is accessible from state i if P ni,j >0 for some n 0. i j. Equivalent to: P (ever enter j|start in i) > 0.

    P (ever enter j|start in i)= P (n=0{Xn = j}|X0 = i)

    n=0

    P (Xn = j | X0 = i)

    =n=0

    P ni,j

    Hence if P ni,j = 0 for all n, P (ever enter j|start in i) =0. On the other hand,

    P (ever enter j|start in i)= P (n=0{Xn = j}|X0 = i) P ({Xn = j}|X0 = i) for any n= P ni,j .

    If P ni,j > 0 for some n, P (ever enter j|start in i) P ni,j > 0.

    Examples

    8

  • Communicate: State i and j communicate if they areaccessible from each other. i j. Properties:

    1. P 0i,i = P (X0 = i|X0 = i) = 1. Any state icommunicates with itself.

    2. If i j, then j i.3. If i j and j k, then i k.

    Proof:

    i j = P ni,j > 0 and P n

    j,i > 0

    j k = Pmj,k > 0 and Pm

    k,j > 0

    P n+mi,k =l=0

    P ni,lPml,k Chapman-Kolmogorov Eq.

    > P ni,j Pmj,k> 0

    Similarly, we can show P n+mk,i > 0. Hence ik.

    9

  • Class: Two states that communciate are said to bein the same class. A class is a subset of states thatcommunicate with each other. Different classes do NOT overlap. Classes form a partition of states.

    Irreducible: A Markov chain is irreducible if there isonly one class. Consider the Markov chain with transition proba-

    bility matrix:

    P =

    12 12 01

    214

    14

    0 1323

    The MC is irreducible. MC with transition probability matrix:

    P =

    12

    12

    0 012

    12 0 0

    14

    14

    14

    14

    0 0 0 1

    Three classes: {0, 1}, {2}, {3}.

    10

  • Recurrent and Transient States

    fi: probability that starting in state i, the MC will everreenter state i.

    Recurrent: If fi = 1, state i is recurrent. A recurrent states will be visited infinitely many

    times by the process starting from i. Transient: If fi < 1, state i is transient.

    Starting from i, the MC will be in state i for ex-actly n times (including the starting state) is

    fn1i (1 fi) , n = 1, 2, ...This is a geometric distribution with parameter 1fi. The expected number of times spent in state iis 1/(1 fi).

    11

  • A state is recurrent if and only if the expected numberof time periods that the process is in state i, startingfrom state i, is infinite.

    Recurrent E(number of visits to i|X0 = i) = Transient E(number of visits to i|X0 = i)

  • Proposition 4.1: State i is recurrent if n=0P ni,i =, and transient ifn=0P ni,i
  • Consider a MC with

    P =

    0 0 1212

    1 0 0 00 1 0 00 1 0 0

    The MC is irreducible and finite state, hence all statesare recurrent.

    Consider a MC with

    P =

    12

    12 0 0 0

    12

    12

    0 0 00 0 12

    12 0

    0 0 12

    12

    014

    14 0 0

    12

    Three classes: {0, 1}, {2, 3}, {4}. State 0, 1, 2, 3 arerecurrent and state 4 is transient.

    14

  • Random Walk

    A Markov chain with state space i = 0,1,2, .... Transition probability: Pi,i+1 = p = 1 Pi,i1.

    At every step, move either 1 step forward or 1 stepbackward.

    Example: a gambler either wins a dollar or loses adollar at every game. Xn is the number of dollars hehas when starting the nth game.

    For any i < j, P jii,j = pji > 0, P jij,i = (1 p)ji >0. The MC is irreducible.

    Hence, either all the states are transient or all thestates are recurrent.

    15

  • Under which condition are the states transient or re-current? Consider State 0.

    n=1

    P n0,0 =

    { recurrentfinite transient

    Only for even m, Pm0,0 > 0.

    P 2n0,0 =

    (2nn

    )pn(1 p)n = (2n)!

    n!n!(p(1 p))n

    n = 1, 2, 3, ...

    By Stirlings approximation

    n! nn+1/2en

    2pi

    P 2n0,0 (4p(1p))n

    pin

    .

    When p = 1/2, 4p(1 p) = 1.n=0

    (4p(1 p))npin

    =n=0

    1pin

    ,

    The summation diverges. Hence, all the states arerecurrent.

    When p 6= 1/2, 4p(1 p) < 1. n=0 (4p(1p))npinconverges. All the states are transient.

    16

  • Limiting Probabilities

    Weather example

    P =

    (0.7 0.30.4 0.6

    )

    P(4) = P4 =

    (0.5749 0.42510.5668 0.4332

    )

    P(8) = P(4)P(4) =

    (0.572 0.4280.570 0.430

    )

    P(4) and P(8) are close. The rows in P(8) are close. Limiting probabilities?

    17

  • Period d: For state i, if P ni,i = 0 whenever n is notdivisible by d and d is the largest integer with thisproperty, d is the period of state i. Period d is the greatest common divisor of all them such that Pmi,i > 0.

    Aperiodic: State i is aperiodic if its period is 1. Positive recurrent: If a state i is recurrent and the ex-

    pected time until the process returns to state i is finite.

    If i j and i is positive recurrent, then j is posi-tive recurrent.

    For a finite-state MC, a recurrent state is also pos-itive recurrent.

    A finite-state irreducible MC contains all positiverecurrent states.

    Ergodic: A positive recurrent and aperiodic state isan ergodic state.

    A Markov chain is ergodic if all its states are ergodic.

    18

  • Theorem 4.1: For an irreducible ergodic Markov chain,limnP ni,j exists and is independent of i. Let pij =limnP ni,j, j 0, then pij is the unique nonnegativesolution of

    pij =i=0

    piiPi,j j 0j=0

    pij = 1 .

    The Weather Example:

    P =

    ( 1 1

    )The MC is irreducible and ergodic.

    pi0 + pi1 = 1

    pi0 = pi0P0,0 + pi1P1,0 = pi0 + pi1

    pi1 = pi0P0,1 + pi1P1,1 = pi0(1 ) + pi1(1 )Solve the linear equations, pi0 = 1+, pi1 =

    11+.

    19

  • Gamblers Ruin Problem

    At each play, a gambler either wi