1

Introduction to Stochastic ModelsIntroduction to Stochastic ModelsGSLM 54100GSLM 54100

2

OutlineOutline

continuous-time Markov chain

3

Continuous-Time Markov ChainContinuous-Time Markov Chain continuous-time analog of discrete-time Markov chain objective: to deduce the long-term average cost per unit

time

Discrete-time Markov chain {Xn}

Continuous-time Markov chain {X(t)}

State

Markov property

Stationary transition

Discrete Discrete

P(Xn+1 = j|Xn = i, Xn-1, …, X0) = P(Xn+1 = j|Xn = i) i, j, n

P(X(t+s) = j|X(t) = i, X(r) = x(r), 0 r < t) = P(X(t+s) = j|X(t) = i), i, j, t, s (> 0)

P(Xn+1 = j|Xn = i) = pij , i, j, n

P(X(t+s) = j|X(t) = i) = pij(s) i, j, t, s (> 0)

4

Differences Between Differences Between CTMCCTMC & & DTMCDTMC

duration of a state DTMC: one period

the consecutive duration staying in a state ~ Geo

CTMC: any positive real number the consecutive duration staying in a state ~ exp

cost calculation DTMC: cost per visit = cost per unit time at state

CTMC: cost per visit cost per unit time at state

5

Cost Calculation in Cost Calculation in CTMCCTMC

ci = cost per unit at state i

long-term average cost per unit time

{ ( ) }01lim ,

ti X s i

i

t

c ds

t

{ ( ) }

1, if ( ) , where 1 =

0, . .X s iX s i

o w

ci = cost per visit of state i

long-term average cost per unit time (total number of visits to state in [0, t])

limi

i

t

c i

t

6

Sojourn Times of Sojourn Times of CTMCCTMC

sojourn time: the time that a CTMC stays at a state in a visit

fact:

result of the Markov property

time

excellent

good

fair

bad

1eX

1gX

2eX

2gX

( ) ~ i.i.d. exp( )iX

7

Markov Property of Markov Property of CTMCCTMC

P(X(t+s) = j |X(t) = i, X(r) = x(r) for 0 r < t) = P(X(t+s) = j | X(t) = i) for all i, j and for all t, s > 0  given present (i.e., X(t)), the future (i.e., X(t+s))

and the past (i.e., X(r) = x(r) for 0 r < t) are independent

0 t t+s

X(t) = i X(t+s) = ?

known states, X(r) = x(r), 0 r < t

present future

past

8

Intuitive Justification Intuitive Justification of i.i.d. Exp Sojourn Timesof i.i.d. Exp Sojourn Times

Ti: the sojourn time of state i

just entering state i at t = 0

P(Ti > t) = P(Ti > t|X(0) = i) being a function of i

suppose X(r) = i for 0 r t P(Ti > t+s|X(r) = i, 0 r t) = P(Ti > t+s|Ti > t)

Markov property P(Ti > t+s|X(r) = i, 0 r s) = P(Ti > t|X(0) = i) = P(Ti > t)

P(Ti > t+s|Ti > t) = P(Ti > t) Ti ~ exp(qi)

9

Definition of a Definition of a CTMCCTMC

{X(t)}, a continuous-time, discrete-state stochastic process is a CTMC if  

(a). sojourn times at state i, Ti ~ exp(qi), independent of everything else, 0 < qi <

(b). whenever leaving state i, next visiting state j with probability pij > 0, j pij = 1

for the time being, assume pii = 0 for all i, because of the way that

we define sojourn times

10

Procedure to Define a Procedure to Define a CTMCCTMC

defining three kinds of quantities the states (and the state space)

qi for all i (where Ti ~ exp(qi)) and

pij for all i, j (where next visiting j with probability pij upon leaving i )

11

Example 8.4.2Example 8.4.2

one machine, one repairman

working times of machine ~ i.i.d. exp()

repairing times of machine ~ i.i.d. exp()

working

repairing

12

Example 8.4.3Example 8.4.3

two machines, one repairman

working times of all machines ~ i.i.d. exp()

repairing times of all machine ~ i.i.d. exp()

state N(t) = number of working machines

1

0

2

N(t)

t

exp(2)

exp() exp()

exp(+) exp(+)

state 2: two exp() competing

state 1: one exp() and one exp() competing

state 0: one exp() competing

13

Another WayAnother Wayto Define a to Define a CTMCCTMC

last example revealing that in a CTMC, events with exponential times to occur competing to be the first one

defining three kinds of quantities the states (and the state space)

qij for all i, j (where Tij ~ exp(qij) is the duration of the activity that leads to state change from i to j)

qi, where Ti = min(Tij, j i) ~ exp(qi) and qi = j qij

14

Two Ways to Define a Two Ways to Define a CTMCCTMC

{qi, {pij}} { Tij ~ exp(qij), Ti = min(Tij, j i)}

Tij ~ exp(qij)Ti = min(Tij, j

i)

qi = j qij,pij = P(Ti = min(Tij, j i)) = qij/qi

Ti ~ exp(qi)

qi, {pij}

occurrence of events following Poisson of rate qi

random partitioning

by pij

time to change to state j,

Ti1 ~ exp(qi pi1),…

Tij ~ exp(qi pij)…

15

Transition DiagramTransition Diagram

Example 8.4.2: Example 8.4.2: NN((tt) = # of working machine ) = # of working machine at at tt

Example 8.4.3: Example 8.4.3: NN((tt) = # of working machine ) = # of working machine at at tt

0

1

0

12

2

16

Birth and Death ProcessesBirth and Death Processes

a CTMC such that qi,j = 0 for |i-j| 1

birth rate at state i: i

death rate at state i: i

2102

1

1

0… n

2 nn1

3 n+1n

17

Standard Standard MM//MM/1 Queue/1 Queue

Poisson arrivals of rate , exponential service of rate , single server, infinite buffer

210

… n

18

MM//MM/1 Queue with Finite Buffer/1 Queue with Finite Buffer

Poisson arrivals of rate , exponential service of rate , single server, finite buffer of at most N customers in system

210

…

NN1

19

Standard Standard MM//MM//cc Queue Queue

Poisson arrivals of rate , exponential service of rate , c servers, infinite buffer

210

…

3

c+1c

2

c1

(c-1) c

c

c

…

20

MM//MM//cc Queue with Finite Buffer Queue with Finite Buffer

Poisson arrivals of rate , exponential service of rate , c servers, finite buffer of size N, N c

210

… c1

(c-1)2

3

c

c

c+1

c…

cN1

cN

c

21

MM//MM// Queue Queue

Poisson arrivals of rate , exponential service of rate , number of servers, infinite buffer

210

…

2

3

3

4

4

5

22

Finite-Source Queue Finite-Source Queue with Exponential Serverswith Exponential Servers

K machines, i.i.d. working times ~ exp()

R repairmen, i.i.d. repairing times ~ exp()

210

K… R1

(R-1)2 3R

R

R+1

R…

RN1

2

RN

R

(K-1) (K-2) (K-R+2) (K-R+1) (K-R) (K-R-1)

23

Limiting Behavior Limiting Behavior of a Positive Irreducible of a Positive Irreducible DTMCDTMC

an irreducible DTMC {Xn} is positive there exists a unique nonnegative solution to

 

j: stationary (steady-state) distribution of {Xn}

0

0

1 (normalization eqt)

, for all , (balance eqts)

jj

j i iji

p j

24

Limiting Behavior Limiting Behavior of a Positive Irreducible of a Positive Irreducible DTMCDTMC

j = fraction of time at state j

j = fraction of expected time at state j

average cost cj for each visit at state j

random i.i.d. Cj for each visit at state j

for aperiodic chain:

1

0lim

k

n

Xk

j jn j

E cc

n

1

0lim ( )

k

n

Xk

j jn j

E CE C

n

0lim ( | )n jn

P X j X

25

Limiting Behavior Limiting Behavior of a Positive Irreducible of a Positive Irreducible CTMCCTMC

if {pj} satisfy

cj = cost/time at j

dj = cost/visit at j

for all

1

j j i iji j

ii

p q p q j

p

( )01lim ,

the fraction of time at state

tX s j

jt

dsp

tj

lim ( ( ) | (0) )jt

p P X t j X i

( )01

lim

tj X s j

jj j

t j

c ds

c pt

(# of visits to state in [0, ])

lim j

jj j j

t j

d j t

d q pt

26

Intuitive Derivation Intuitive Derivation of Expressions for Average Costof Expressions for Average Cost

for cj, cost per unit time

T: a large positive number

time at state j pjT

cost induced by state j cjpjT

total cost j cjpjT

cost per unit time = j cjpj

27

Intuitive Derivation Intuitive Derivation of Expressions for Average Costof Expressions for Average Cost

for dj, cost per visit T: a large positive number

time at state j pjT

mean time in state j per visit = 1/qj

approximate number of visits in pjT qjpjT

cost induced by state j cjqjpjT

total cost j cjqjpj

cost per unit time = j cjqjpj

28

ExamplesExamples

Example 8.6.1 Example 8.6.1

stationary distributions of the queueing stationary distributions of the queueing systemssystems

M/M/1

M/M/c

M/M/

M/M/1 with finite buffer

M/M/c with finite buffer

finite-source queue with exponential servers

29

Expected Time Taken Expected Time Taken to Visit a State to Visit a State

XX(0) = 0(0) = 0

00 = = EE(time taken to visit state 2 for the first (time taken to visit state 2 for the first

time|time|XX(0) = 0) (0) = 0)

define define 11 = = EE(time taken to visit state 2 for the (time taken to visit state 2 for the

first time|first time|XX(0) = 0) (0) = 0)

210

2

0 11

1 01

30

Exercise 8.4.8.

The arrivals of potential customers (cars) to a two-mechanic garage follow a Poisson process of rate . The garage has space for two cars; cars that find the garage full go away and never return. At any moment, a car service requires only one mechanic.

31

Exercise 8.4.8.

(a). Suppose that service times are i.i.d. exponential random duration of rate , and that the service and the arrival processes are independent.

(i). Model the problem as a CTMC. You may draw out the transition diagram of the chain after you have defined the state and state space.

(ii). The chain should be positive. Find its stationary distribution.

(iii). Each served car pays a lump sum of $c for its service, and the garage pays each mechanic $r per unit time. Find the long-term net revenue rate ($ per unit time) of the garage.

(iv). In addition to (iii), suppose that the garage pays back a served car $h for each unit time that the car stays in the garage. (Woo! A car can actually earn some money if its service is longer than c/h.) Find the long-term net revenue rate of the garage in this case.

32

Exercise 8.4.8.

Model the following parts as CTMCs. Each part should be modeled independently from others and from part (a). For each part, first define the state and state space and then draw its transition diagram.

(b). There are two. Any service completed by the first mechanic is of exp(1) and that by the second mechanic of exp(2). When a car comes, if only one mechanics is idle, the mechanic picks up the job; if both mechanics are idle, the car is assigned to the mechanic who has taken longer rest since last service completion.

33

Exercise 8.4.8.

(c). There are two types of service, A and B, provided by the garage. For any of the two mechanics, a type A service ~ exp(A) while type B ~ exp(B). Each car needs only one type of service: type A with probability p, 0 < p < 1, independent of everything else, and type B otherwise. The service times are independent from each other and from the arrival process.