Upload
evangeline-shepard
View
65
Download
4
Embed Size (px)
DESCRIPTION
Chapter 4 Discrete time Markov Chain. Learning objectives : Introduce discrete time Markov Chain Model manufacturing systems using Markov Chain Able to evaluate the steady-state performances Textbook : - PowerPoint PPT Presentation
Citation preview
1
Chapter 4Discrete time Markov Chain
Learning objectives :• Introduce discrete time Markov Chain• Model manufacturing systems using Markov Chain• Able to evaluate the steady-state performances
Textbook :C. Cassandras and S. Lafortune, Introduction to Discrete
Event Systems, Springer, 2007
2
Plan
• Basic definitions of discrete time Markov Chains • Classification of Discrete Time Markov Chains• Analysis of Discrete Time Markov Chains
3
Basic definitions
of discrete time Markov chains
4
Discrete Time Markov Chain (DTMC)
Definition : a stochastic process with discrete state space and discrete time {Xn, n > 0} is a discrete time Markov Chain (DTMC) iff
P[Xn+1 = j Xn = in, ..., X0 = i0] = P[Xn+1 = j Xn = in] = pij(n)
In a DTMC, the past history impacts on the future evolution of the system via the current state of the system
pij(n) is called transition probability from state i to state j at time n.
5
Discrete Time Markov Chain (DTMC)
Stochastic process
Discrete events
Continuous event
Discrete time
Continuous time
Memoryless
A DTMC is a discrete time and memoriless discrete event stochastic process.
6
Example: a mouse in a maze (老鼠在迷宫 )
Which stochastic process can be used to represent the position of the mouse at time t?Under which assumptions, the system can be represented by a discrete time Markov chain?
1 2
3 45
0
start
exit
7
Example: a mouse in a maze
• Let {Xn}n=0, 1, 2, ... the position of the mouse after n rooms visited
• Assume that the mouse does not have any memory of rooms visited previously and that she chooses any corridor equi-probably.
1 2
3 45
0
start
exit
1
1/2
1/2
1/2
11/4
1/4
1/2
1/41/4
1
8
Homogenuous DTMC
• A DTMC is said homogenuous iff its transitions probabilities do not depend on the time n, i.e.
P[Xn+1 = j Xn = i] = P[X1 = j X0 = i] = pij
• A homogenuous DTMC is then defined by its transition matrix P =[pij]i,jE
9
What is the transition matrix of the process?
• Let {Xn}n=0, 1, 2, ... the position of the mouse after n rooms visited
• Assume that the mouse does not have any memory of rooms visited previously and that she chooses any corridor equi-probably.
1 2
3 45
0
start
exit
1
1/2
1/2
1/2
11/4
1/4
1/2
1/41/4
1
10
Stochastic Matrix
A square matrix is said stochastic iff
• all entries are non negative
• each line sums to 1
Properties:
• A transition matrix is a stochastic matrix
• If P is stochastic, then Pn is stochastic
• The eigenvalues of P are all smaller than 1, i.e. || ≤1
11
Assumptions
• In the remaining of the chapter, we limit ourselves to Markov chain
• of discrete time
• defined on a finite state space E
• homogeneous in time.
• Note that most results extend to countable state space.
12
Graphic representation of a DTMC
13
Classification of Discrete Time Markov Chains
14
Classification of states
• Let fjj be the probability of returning to state j after leaving j.
• A state j is said transient if fjj < 1
• A state j is said recurrent if fjj = 1
• A state j is said absorbing if pjj = 1.
• Let Tjj be the average reccurn time, i.e. time of returning to j
• A recurrent state j is positive recurrent if E[Tjj] is finite.
• A recurrent state j is null recurrent if E[Tjj] = .
15
Classify the states of the example
1 2
3 45
0
start
exit
1
1/2
1/2
1/2
11/4
1/4
1/2
1/41/4
1
16
Irreducible Markov chain
• A DTMC is said irreducible iff a state j can be reached in a finite number of steps from any other state i.
• An irreducible DTMC is a strongly connected graph.
17
Irreducble Markov chain
18
Periodic Markov chain
• A state j is said periodic if it is visited only in a number of steps which is multiple of an integer d > 1, called period.
• A state j is said aperiodic otherwise
• A state with a self-loop transition (i.e. pii > 0) is always aperiodic.
• All states of an irreducible Markov chain have the same period.
19
Partitionning a DTMC into irreducible sub-chains
• A DTMC can be partitionned into strongly connected components, each corresponding to an irreducible sub-chain.
20
Classification of irreducible sub-chains
• A sub-chain is said absorbing if there is no arc going out of it.
• Otherwise, the sub-chain is transient.
transcient sub-chain
absorbing sub-chain
absorbing sub-chain
21
Canonic form of transition matrix
• Q : transitions of transient sub-chains
• Pi : transititions between states of aborbing sub-chain i
• Ri: Transitions toward absorbing sub-chain i
22
Formal definitions
• A state j is said reachable from a state i if there is a path from i to j in the state transition diagram.
• A subset S of states is said closed if there is no transition leaving S.
• A closed set S is said irreducible if all states in S are mutually reachable.
• A Markov chain is said irreducible if its state space is irreducible.
23
Theorems
Th1. If a Markov chain has a finite state space, then at least one state is recurrent.
Th2. If i is a recurrent state and j is reachable from i, then state j is recurrent.
Th3. If S is a finite closed irreducible set of states, then every state in S is recurrent.
Th4. If i is a positive recurrent state and j is reachable from i, then state j is positive recurrent.
Th5. If S is a closed irreducible set of states, then every state in S is positive recurrent or every state in S is null recurrent or every state in S is transient.
Th6. If S is a finite closed irreducible set of states, then every state in S is positive recurrent.
24
Analysis of DTMC
25
Sojourn time in a state
• Let Ti be the temps spent in state i before jumping to other states.
• Ti is a random variable of geometric distribution.
1 1ni ii iiP T n p p
26
Properties of geometric distribution
• Let X be a random variable of geometric distribution with parameter p, i.e. P{X = n} = (1-p)n-1p.
• E[X] = 1/p
• Var(X) = 1/p2
• X = 1/p
• Coefficient of variation = X / E[X] = 1
• Memoryless (only discrete distribution of this property):
11
1
¨
11
1
n mm
n
P X n mP X n m X n
P X n
p pp p P X m
p
27
m-step transition probabilities
• The probability of going from i to j in m steps is
pij(m) = P{Xn+m = j|Xn=i} = P{Xm = j|X0=i}.
• Let P(m) = [pij(m)] be the m-step transition matrix
Properties (to prove):
• P(m) = Pm
• Chapman-Kolmogorov equation:
P(l+m) = P(l)P(m)
or l m l mij ik kj
k E
p p p
28
Example
• What is the probability that the mouse is still in room 2 at time 4? (p22
(4))
1 2
3 45
0
start
exit
1
1/2
1/2
1/2
11/4
1/4
1/2
1/41/4
1
29
Probability of going from i to j in exactly n steps
• fij(n) : probability of going from i to j in exactly n steps (without
passing j before)
• fij: probability of going from i to j in a finite number of steps
• Similar approach can be used to determine the average time Tij it takes for going from i to j
1
nij ij
n
ij ij ik kjk j
f f
f p p f
30
Probability distribution of states
• i(n) : probability of being in state i at time n
i(n) = P{Xn = i}
• (n) = (1(n), 2(n), ...) : vector of probability distribution over the state space at time n
• The probability distribution (n) depends on
─ the transition matrix P
─ the initial distribution (0)
• Remark: if the system is at state i for certainty, then i(0) = 1 and j(n) = 0, for j ≠i
• What is the relation between (n), (0), and P?
31
Transient state equations
• By conditioning on the state at time n,
Property:
Let P be the transition matrix of a markov chain and (0) the initial distribution, then over the state space at time n
(n+1) = (n)P
(n)= (0)Pn
32
Steady-state distribution
Key questions :
• Is the distribution (n) converges when n goes to infinity?
• If the distribution converges, does its limit = (1, 2, ...) depend on the initial distribution (0)?
• If a state is recurrent, what is the percentage of time spent in this state and what is the number of transitions between two successive visits to the state?
• If a state is absorbing, what is the probability of ending at this state? What is the average time to this state?
33
Steady state distribution
Theorem : For a irreducible and aperiodic DTMC with positive recurrent states, the distribution (n) converges to a limit vector which is independent of (0) and is the unique solution of the system:
• i are also called stationary probabilities (also called steady state or equilibrium distribution).
• For an irreducible and periodic DTMC, i are the percentage of time spent in state i
1ii E
P
,
1
j i iji E
ii E
p j E
Normalization equation
balance equationequilibrium equation
34
Flow balance equation
• Equation can be interpretated as balance equation of probability flow.
• A probability flow ipij is associated to each transition (i, j).
• is the sum of probability flow into node j
• is the sum of flow out of node j
• The flow balance equation : Outgoing flow = Incoming flow
j i iji E
p
i iji E
p
or j j jii E
p
35
A manufaturing system
• Consider a machine which can be either UP or DOWN.
• The state of the machine is checked every day.
• The average time to failure of an UP machine is 10 days.
• The average time for repair of a DOWN machine is 1.5 days.
• Determine the conditions for the state of the machine {Xn} at the begining of each day to be a Markov chain.
• Draw the Markov chain model.
• Find the transient distribution by starting from state UP and DOWN.
• Check whether the Markov chain is recurrent and aperiodic.
• Determine the steady state distribution.
• Determine the availability of the machine.
36
A telephone call process
• Discrete time model with time slots indexed by k = 0, 1, 2, ...
• At most one telephone call can occur in a single time slot, and there is a probability that a call occurs in any slot
• If the phone is busy, the call is lost; otherwise, the call is processed.
• There is a probability that a call in process completes in any time slot
• If both a call arrival and a call completion occur in the same time slot, the new call will be processed.
Issues to solve:
• Markov chain model
• Loss probability