STOCHASTIC MODELS LECTURE 3 PART II

CONTINUOUS-TIME MARKOV PROCESSES

Nan ChenMSc Program in Financial EngineeringThe Chinese University of Hong Kong

(ShenZhen)Oct 21, 2015

Outline1. Birth-Death Processes

3.3 BIRTH-DEATH PROCESSES

Birth-Death Processes

• Birth-death processes are one of the most important examples for continuous-time Markov chains.

• Consider a system whose state at any time is represented by the number of people. Suppose that whenever there are people in the system, then– new arrivals enter it (people born) at an

exponential rate– People leave the system (people die) at a rate

Birth-Death Processes (Continued)

• The above system can be modeled as a continuous-time Markov chain with states

for which the transition parameters are given by – – – –

Example I: The Poisson process

• The Poisson process, as a special example of birth-death processes, has the following parameter specification: – –

Example II: A Birth Process with Linear Birth Rate (Yule Process)• Consider a population whose members can give

birth to new members but cannot die. Suppose that each member acts independently of the others and takes an exponentially distributed amount of time, with mean to give a birth.

• Consider the population number. We have– –

Example III: A Linear Growth Model with Immigration• A model in which– – is called a linear growth process with immigration. In it, each individual is assumed to give birth at an exponential rate and die at rate ; in addition, there is an rate of increase of the population due to an external source such as immigration.

Kolmogorov Equations for B-D Processes • We can easily have the backward equations

for a birth-death process:

Kolmogorov Equations for B-D Processes (Continued)• As for the forward equations, we have

Example IV: Pure Birth Processes

• Calculate for the process in Example II, using the forward equations.

• We have

Example IV: Pure Birth Processes (Continued)• The solution is

Limiting Probabilities

• Let us now determine the limiting probabilities for a birth-death process. From the general equation

we should have– State 0: – State 1:– State 2:– State

Limiting Probabilities (Continued)• We can solve the above equations to obtain

……

Limiting Probabilities (Continued)• Then, the limiting probabilities of a general

birth-death process are given by

Example V: A Machine Repair Model• A job shop consists of machines and one

serviceman. Suppose that the amount of time each machine runs before breaking down is exponentially distributed with mean Suppose that it takes the serviceman an exponential time (with mean ) to fix a machine.

• What is the average number of machine not in use?

Example V: A Machine Repair Model• Establish a birth-death process to model the

system. The process is in state whenever machines are not in use. Then, the process has the following parameters:– –

Example V: A Machine Repair Model (Continued)• By the formula of the limiting probabilities,

we have

Example V: A Machine Repair Model (Continued)• Hence, the average number of machines not

in use is given by

Homework Assignments

• Read Ross Chapter 6.3.• Answer Questions:– Exercises 12 (Page 400, Ross)– Exercises 15 (Page 401, Ross)– Exercises 23 (Page 402, Ross)– (Optional, Extra Bonus) Exercise 25 (Page 402,

Ross).