Introduction to stochastic processIntroduction to stochastic process

Dr. Nur Aini Masruroh

Outlines Outlines

Concept of probability

Random variables

Expected value

Conditional probability

Limit theorem

Stochastic processes

Probability in Industrial EngineeringProbability in Industrial Engineering

Random arrivals of jobsRandom service timeRandom requests for resourcesProbability of queue overflowService delaysScheduling problemsFlow control and routingRevenue managementRisk assessment for decision analysisEtc

Basic probabilityBasic probability

Random experiment: an experiment whose outcome cannot be determined in advanced

Sample space (S): set of all possible outcomes Event (E): a subset of sample space, it occurs if the

outcome of the experiment is an element of that subset A number P(E) is defined and satisfies:

0 ≤ P(E) ≤ 1 P(S) =1 For any sequence of events E1, E2, … that are mutually

exclusive

11

)(i

ii

i EPEp

Random variablesRandom variables

Consider a random experiment having sample space S. A random variable X is a function that assigns a real value to each

outcome in S The distribution function F of the random variable X is defined for

any real number x by

A random variable X is said to be discrete if its set of possible values is countable. For discrete random variables,

A random variable is called continuous if there exists a function f(x), called the probability density function, such that

xXPxF ,)(

xy

yXPxF )(

B

dxxfBXP )(inis For every set B

Random variablesRandom variables

Since it follows that

The join distribution function F of two random variables X and Y is defined by

F(x, y) = P{X≤ x, Y ≤ y} The distribution functions of X and Y,

Fx(x) = P{X≤ x} and FY(y) = P{Y≤y}

It can be obtained from F(x, y) by making use of the continuity property of the probability operator.

Continuity property:

Similarly,

F(x, y) = Fx(x)FY(y) if X and Y are independent

x

dxxfxF )()( )()( xFdx

dxf

xXPyYxXP nn

,lim

),(lim)(or),(lim)( yxFyFyxFxFx

Yy

X

Cumulative Distribution Function (CDF)Cumulative Distribution Function (CDF)

CDF, F, of the random variable X is defined by all real numbers b, -∞ < b < ∞, by F(b) = P(X ≤ b)

F(b) denotes the cumulative probability that the random variable X takes on a value that is less or equal to b (the total probability mass)

CDF is a non-decreasing function: P(X≤ a) ≤ P(X ≤ b) if a < b and thus F(a) ≤ F(b) for a < b

F(∞) = 1 and F(-∞)=0: bounded CDF is a right continuous function: for any b, F(b+), value

of F(b) just to the right of b, equals to F(b) as b+ get closer to b. The CDF is right continuous at all values, but may be left discontinuous at some values

Probabilities from CDFProbabilities from CDF

P(X ≤ b) = F(b) P(X<b) = F(b – ε) = F(b-)

ε is a small number

P(X=b) = P(X ≤ b) – P(X < b) = F(b) – F(b-) P(a < X ≤ b) = F(b) – F(a) P(a ≤ X ≤ b) = F(b) – F(a-) P(a ≤ X < b) = F(b-) – F(a-) P(a < X < b) = F(b-) – F(a)

Probability Density Function (pdf)Probability Density Function (pdf)

The pdf of a continuous random variable X is defined by the derivative of the continuous CDF at differentiable intervals : dF(b)/db = f(b)

The pdf of a continuous random variable tells us the density of the mass at each point on the sample space

The value of pdf is not probability, for example, if f(x) = 2x, 0 ≤ x ≤ 1, and 0 otherwise, f(1) = 2 is obviously not a probability value

Graphically, pdf is the area under f(x) from ain interval a,b.

Note: if pmf has meaning itself, the value itself for pdf has no meaning

b

a

dxxfbXaP )()(

Discrete random variable: summaryDiscrete random variable: summary

A random variable X associates a number with each outcome of an experiment

A discrete random variable takes on a finite number The probabilistic behavior of a discrete random variable

X is described by its probability mass function (pmf), p(u) P(uj) = p(X=uj)

All the probabilistic information about the discrete random variable X is summarized in its pmf

A discrete random variable X has a CDF F(b) which is: Right-continuous Staircase CDF

Continuous random variable: summaryContinuous random variable: summary

No mass to define pmf: every event has zero probability Example: p(height of people= 173.7897654) 0 However P(147< height of people < 187) 1

The set of possible values are uncountable while the set of possible values were finite or countably infinite for a discrete random variable

Sample space is not a discrete set, but a continuous space (or interval)

A continuous random variable X has a CDF F(b) which is: Continuous at all b, -∞ < b < ∞ Differentiable at all b (except possibly at finite set of points)

Expected valueExpected value

Expectation of the random variable X,

The variance of the random variable X is defined by

Var X = E[(X – E[X])2]

= E[X2] – E2[X]

Two jointly distributed random variables X and Y are said to be uncorrelated if their covariance defined by

Cov (X, Y) = E[(X – EX)(Y – EY)]

= E[XY] – E[X]E[Y]

x

XxXxP

Xdxxxf

xxdFXE

discreteisif}{

continueisif)(

)(][

Expected valueExpected value

Expectation of a sum of random variables is equal to the sum of the expectations

Variances:

n

ii

n

ii XEXE

11

[

),(2)(11

ji

n

i jii

n

ii XXCovXVarXVar

Conditional probabilitiesConditional probabilities

In general, given information about the outcome of some events, we may revise our probabilities of other events

We do this through the use of conditional probabilities The probability of an event X given specific outcomes of another

event Y is called the conditional probability X given Y The conditional probability of event X given event Y and other

background information ξ, is denoted by p(X|Y, ξ) and is given by

0)|(for)|(

)|(),|(

Yp

Yp

YXpYXp

Bayes’ TheoremBayes’ Theorem

Given two uncertain events X and Y. Suppose the probabilities p(X|ξ) and p(Y|X, ξ) are known, then

X

XYpXpYp

where

Yp

XYpXpYXp

)||()|()|(

)|(

),|()|(),|(

Factorization rule for joint probabilityFactorization rule for joint probability

Limit theorem Limit theorem Strong Law of Large Numbers

If X1, X2, … are independent and identically distributed with mean μ, then

Central Limit Theorem If X1, X2, … are independent and identically distributed with

mean μ and variance σ2, then

Thus if we let where X1, X2, … are independent and identically distributed, then the Strong of LLN states that, with probability1, Sn/n will converge to E[Xi]; whereas the CLT states that Sn will have an asymptotic normal distribution as n → ∞

1...

lim 1

n

XXP n

n

dxean

nXXP x

an

n

2/12

2

1...lim

n

i in XS1

Stochastic processStochastic process

X(t) is the state of the process (measurable characteristic of interest) at time t

• the state space of the a stochastic process is defined as the set of all possible values that the random variables X(t) can assume

• when the set T is countable, the stochastic process is a discrete time process; denote by {Xn, n=0, 1, 2, …}

• when T is an interval of the real line, the stochastic process is a continuous time process; denote by {X(t), t≥0}

Stochastic processStochastic process

Hence,• a stochastic process is a family of random

variables that describes the evolution through time of some (physical) process.

• usually, the random variables X(t) are dependent and hence the analysis of stochastic processes is very difficult.

• Discrete Time Markov Chains (DTMC) is a special type

of stochastic process that has a very simple dependence among X(t) and renders nice results in the analysis of {X(t), t∈T} under very mild assumptions.

Example of stochastic processesExample of stochastic processes

Refer to X(t) as the state of the process at time t A stochastic process {X(t), t T} is a time indexed ∈

collection of random variables

X(t) might equal the total number of customers that have entered a supermarket by time t

X(t) might equal the number of customers in the supermarket at time t

X(t) might equal the stock price of a company at time t

Counting processCounting process

Definition: A stochastic process {N(t), t≥0} is a counting process if

N(t) represents the total number of “events” that have occurred up to time t

Counting process Counting process

Examples: If N(t) equal the number of persons who have entered a

particular store at or prior to time t, then {N(t), t≥0} is a counting process in which an event corresponds to a person entering the store

• If N(t) equal the number of persons in the store at time t, then {N(t), t≥0} would not be a counting process. Why?

If N(t) equals the total number of people born by time t, then {N(t), t≥0} is a counting process in which an event corresponds to a child is born

If N(t) equals the number of goals that Ronaldo has scored by time t, then {N(t), t≥0} is a counting process in which an event occurs whenever he scores a goal

Counting processCounting process

A counting process N(t) must satisfy N(t)≥0 N(t) is integer valued If s ≤t, then N(s) ≤ N(t) For s<t, N(t)-N(s) equals the number of events that have

occurred in the interval (s,t), or the increments of the counting process in (s,t)

A counting process has Independent increments if the number of events which occur in

disjoint time intervals are independent Stationary increments if the distribution of the number of

events which occur in any interval of time depends only on the length of the time interval

Independent incrementIndependent increment

This property says that numbers of events in disjoint intervals are independent random variables.

Suppose that t1< t2≤ t3< t4. Then N(t2)-N(t1), the number of events occurring in (t1,t2], is independent of N(t4)-N(t3), the number of events occurring in (t3, t4].