CH1 Introduction - ecourse2.ccu.edu.tw

教育部網路通訊人才培育先導型計畫

CH1

Introduction

Random Processes Chapter 1 1


Text books recommended

Chapter 1 Introduction 2

Henry Stark and John Woods, Probability and Random

Processes with Applications to Signal Processing, 3rd ed.,

Prentice-Hall, 2002.

A. Papoulis and S. U. Pillai, Random Variables, and

Stochastic Processes, 4th ed., McGraw-Hill, 2002.

A. Leon Garcia, Probability and Random Processes for

Electrical Engineers, 2nd ed., Addison Wesley, 1993.

Charles W. Therrien, Discrete Random Signal and

Statistical Signal Processing, 1st ed., Prentice Hall, 1992.


Course Contents (1/4)

Chapter 1 Introduction

• 1.1 Sets and measures

• 1.2 Introduction to probability

• 1.3 Random variables

• 1.4 Probability Distribution Function

• 1.5 Probability Density Function

• 1.6 Functions of random variables

Chapter 2 Random Vectors

• 2.1 Joint distribution and densities

• 2.2 Expectation vectors and covariance matrices

• 2.3 Properties of covariance matrices

• 2.4 Gaussian random vector

• 2.5 Characteristic functions

• 2.6 The Central Limit Theorem

• 2.7 ProjectionChapter 1 Introduction 3



Chapter 3 Estimation

• 3.1 Parameter estimation

• 3.2 Estimation of vector means and covariance matrices

• 3.3 Maximum likelihood functions

• 3.4 Linear estimation of vector parameters

Chapter 4 Random Sequences

• 4.1 Basic concepts

• 4.2 Properties of discrete-time linear systems

• 4.3 Random sequences and linear systems

• 4.4 WSS random sequence

• 4.5 Markov random sequence

• 4.6 ARMA model

• 4.7 Convergence of random sequences

• 4.8 Laws of large numbers




Chapter 5 Random Processes

• 5.1 Definitions

• 5.2 Some important random processes

• 5.3 Continuous-time linear systems with random inputs

• 5.4 White noises

• 5.5 Stationary

• 5.6 Power Spectral Density

• 5.7 Periodic and cyclostationary processes

Chapter 6 Advanced Topics in Random Processes

• 6.1 Mean-squares calculus

• 6.2 Ergodicity

• 6.3 Karhunen-Loève expansion

• 6.4 Bandlimited process

• 6.5 WSS periodic process




Chapter 7 Applications to Statistical Signal Processing

• 7.1 Wiener filters and random sequences

• 7.2 Innovation sequence and Kalman Filter

• 7.3 Expectation-maximization algorithm

• 7.4 Hidden Markov Models (HMM)

• 7.5 Spectral estimation



Outline

1.1 Sets and measures

1.2 Introduction to probability

1.3 Random variables

1.4 Probability distribution function

1.5 Probability density function

1.6 Functions of random variables

1.7 Summary

Reference

Problems



Outline







1.7 Summary

Reference

Problems



Apply Set Theory to probability (1/2)

Random Experiment：

A random experiment is an experiment in which the

outcome varies in an unpredictable fashion when the

experiment is repeated under the same conditions.



Outcome

An outcome of the random experiment cannot be

decomposed into other results.

An outcome is distinguishable from every other

outcome.

The outcomes are mutually exclusive in the sense that

they cannot occur simultaneously.



Properties of Sample Space

1. Finest-grain：All possible distinguishable outcomes

are identified separately.

2. Mutually exclusive：If one outcome occurs, then no

other outcomes occur.

3. Collectively exhaustive：Every outcome of the

experiment must be in the sample space .



Example1.2-1

The experiment consists of choosing a person at random

and counting the hairs on his or her head. Then,

That is, the set of all nonnegative integers up to , it

being assumed that no human head has more than

hairs.

710710

71,2,3, ,10



Set Algebra Probability

Element Outcome

Set Event

Universal set Sample Space

Null set Impossible event


The terminology of set theory and probability


Probability space

PF,,

Sample space - field Probability rule


To define a probability space, we need to specify

Probability Space: It is the triple ( , , )

: sample space set of all experimental outcomes

: sigma algebra collection of events that is closed under

complements, unions, and intersections

F P

F

A F A

: probability measure assigned to that satisfies that

axioms of probability

1. 0 2. 1

3. For , and

P A F

P A P

A B F A B P A B P A P B


Field (In our considered scenario, it is a set of subsets)(1/2)

Consider an universal set and a collection of subsets

of . ( : the set of all experimental outcomes)

Let A and B denote subsets in this collection. This

collection of subsets forms a field (an algebra) F

if 1.

2.

3.

* (Subsets of are called events.)



EX:

0 , 1 , , 0,1 is a field

0 , , 0,1 is not a field


Field (In our considered scenario, it is a set of subsets)(2/2)


Sigma-field ( -field) F (1/3)

A sigma ( ) field is a field that is closed under any

countable set of unions, intersections and combinations.

That is, if belong to sigma-field F,

so do

1.

2.

,...,...,,,, 4321 nEEEEE

1

1

i

i

i

i

FE

FE




When the sample space is finite or countable infinite,

we simply let F consist of all subsets of .

Ex：

,1,0,1,0

1,0

F




1. When the sample space is uncountably infinite (for

example is the real line or an interval of R), we cannot

let F to be all possible subsets of R and still satisfy the

axioms of probability.

2. Fortunately, we can obtain all possible events of

practical interest by letting F be the class of events

obtained as complement and countable unions and

intersections of intervals of a real line. This class of F

is called the Borel Field.



Probability Measures (1/4)

The probability law for a random experiment is a rule

to the event of the experiment that belongs to sigma-

field F .

The rule of probability assignment has to satisfy the

axioms of probability.




Let H be a random experiment with sample

space and sigma field F. The probability lawfor H is a rule that assigns each event A in F a

number P[A] that satisfies the following axioms：

1.

2.

3. If ,

then

0AP

1P

BA

BPAPBAP




1.

2.

3.

4. If are mutually exclusive, then

, .

10 AP

0P

nAAAAA ,...,,,, 4321

n

i

i

n

i

i APAP11

2n

1cP A P A




5.

6.

7.

8.

BAPBPAPBAP

BPAPBA

11

1

1 2( 1)

n n

k k k j

k j kk

n

n

P A P A P A A

P A A A

, where ,cP AB P A P AB A B F



Example1.2-2 (1/2)

We wish to prove .

Proof ?

BAPBPAPBAP



Proof：

First we decompose the event into three disjoint

events as follows：

By Axiom 3

= , by axiom 3 again

=

=

c c

c c

P A B P AB A B P AB

P AB P A B P AB

P A P AB P B P AB P AB

P A P B P AB

c cA B AB A B AB


Example1.2-2 (2/2)


Outline








1.7 Summary

Reference

Problems


Definition of a Random Variable (1/18)

Consider an experiment H with sample description

space Ω. The elements or points of Ω , ζ are the

random outcomes of H .

If to every ζ we assign a real number X(ζ), we establish a

correspondence rule between ζ and R, the real line. Such

a rule, subject to certain constraints, is called a random

variable.



Thus, a random variable X(‧) or simply X is not

really a variable but a function whose domain is Ω

and whose range is some subset of the real line.

Being a function, every ζ generates a specific X(ζ)

although for a particular X(ζ) there may be more

than one outcome ζ that produced it.




Through the mapping X, such an event maps into

points on the real line.



Figure 1.3-1 Symbolic representation of the action of the random variable X.


In particular, the event {ζ : X(ζ)≦ x}, often abbreviated

{X≦ x}, will denote an event of unique importance, and

we should like to assign a probability to it.

The probability is called the

probability distribution function (PDF) of X.

XP X x F x




The function X must satisfy the following:

For every Borel set of numbers B, the set

must correspond to an event , that is, it must

be in the domain of the probability function .

X B

BE F

P




State somewhat more mathematically, this require-

ment demands that X can be a random variable only if

the inverse image under X of all Borel subsets in R,

making up the field are events.




What is an inverse image?

Consider an arbitrary Borel set of real numbers B;

the set of points in Ω for which X(ζ) assumes values

in B is called the inverse image of the set B under the

mapping X.

BE




Finally, all sets of engineering interest can be written

as countable unions or intersections of events of the

form . The event gets mapped under

X into .

Thus, if X is a random variable, the set of points

is an event.

, x X x F

, x B

, x




Definition :

Let H be an experiment with sample description

space Ω. Then the real random variable X is a function

whose domain is Ω that satisfies the following：

(i) For every Borel set of numbers B, the set

is an event.

(ii)

,BE X B

0P X P X




Loosely speaking, when the range of X consists of a

countable set of points, X is said to be a discrete

random variable; and if the range of X is a continuum,

X is said to be continuous.




This is somewhat inadequate definition of discrete

and continuous random variables for the simple reason

that we often like to take for the range of X the whole

real line R.




Example 1.10-1

A person, chosen at random in the street, is asked if

he or she has a younger brother. If the answer is no,

the data is encoded as zero; if the answer is yes, the data is encoded as one.




The underlying experiment H has sample

description space Ω ={no, yes}, F =[φ, Ω ,{no},{yes}],

probabilities

The associtate probabilities of are

3 11, 0 and 1 .

4 4

X

P X P X P X

0, 1,

3 1no an assumption , yes

4 4

P P

P P




Take any and consider, for example, the

probabilities that X lies in sets of the type

Thus,

1 2, x x

1 2 1 2 1 2, , or , .x x x x x x

3 4 0

30 1 no

4

0 2 1

10 1 yes

4

P X P

P X P

P X P

P X P




Thus, every set is related

to an event defined on Ω.

Hence, X is a random variable.

1 2 2, , ,X x x X x X x




Example 1.10-2

A bus arrives at random in [0,T]. Let t denote the

time of arrival. The sample description space Ω is

. A random variable X is defined

by

: 0,t t T

1, ,

4 2

0, otherwise

T Tt

X t




Assume that the arrival time is uniform over [0,T].

Compute what is

1 or 0 or 5 .P X t P X t P X t




Solution

12 41

4

32 40

4

5 1

T T

P X tT

T TT

P X tT

P X t




Outline








1.7 Summary

Reference

Problems


1.4 Probability Distribution Function (1/21)

The Probability Distribution Function (PDF) or

Cumulative Distribution Function (CDF) is a function of

x, which contains all the information necessary to

compute P[B] for any B in the Borel field of events.

The PDF, , is defined by XF x


: ( ) , 1.4 1X XF x P X x P x


Equation 1.4-1 is read as “the set of all outcomes ζ

in the underlying sample description space such that the

function X(ζ) assumes values less than or equal to x ”.




, , , ,X

XF P R P

Under the mapping we have, in effect, generate a new

probability space , , , where is the real line,

is the Borel -algebra of all subsets of generated by

countable unions and intersections of

X

X

R P R

R

sets of the form ,

and is a set function assigning a number 0 to each

set .

X X

x

P P B

B

B

B

B

B




Thus, there is a subset of outcomes

that under the mapping generates the set .

The sets and are equivalent

events. We shall frequently leave out the dependence

on the underlying sample space and write merely

.

: X x

X

or P X x P a X b

: X x

( , ]x

( , ]x




1

: ,X

X x F xX

event event

The algebra of events defined on is denoted by .

The family of Borel subsets of points on is denoted by .

F

R

B

B




Example 1.12-1

The experiment consists of observing the voltage X of

the parity bit in a word in computer memory. If the bit

is on, then X=1; if off then X=0. Assume that the off

state has probability q and the on state has probability

1-q. The sample space has only two points:

Ω={off, on}.




Computation of

1. x < 0: The event .

2. The event is equivalent to the

event {off} and excludes the event {on}.

Hence, .

and 0XX x F x

0 1:x X x

XF x q

XF x

on 1

off 0 .

X x

X x




3. The event the certain event since 1:x X x

on 1

off 0 .

X x

X x

is a countinuous function of , then

X

X X

F x x

F x F x




Outline








1.7 Summary

Reference

Problems


is continuous and differentiable, the pdf is

computed from

X

X

X

If F x

dF xf x

dx


1.5 Probability Density Function (pdf) (1/22)


Interpretation of Xf x

If is continuous in its first derivative then, for

sufficiently small ,

Hence, for small

Observe that if exi

X X

X

x x

X X X X

x

X

X

P x X x x F x x F x

F x

x

F x x F x f d f x x

x

P x X x x f x x

f x

sts, then is continuous

and therefore, from Equation 1.4-3, 0. (p.82)

XF x

P X x


1.5 Probability Density Function (pdf) (3/22)


Conditional and joint distributions and densities (1/4)

A conditional probability measure defined byP B A

, 0P AB

P B A P AP A

, 0P AB

P A B P BP B

143Chapter 1 Introduction


Example 1. 5-8 (1/3)

In a binary communication system (Figure 1.5-11), that is,

one in which a two-symbol alphabet is used to communicate,

two symbols are zero and one .

In this system, let stand for the received sy

“ ” “ ”

mbolY

and stand

for the transmitted symbol.

The sample description space for the experiment is

= , : 0 or 1, 0 or 1

0,0 , 0,1 , 1,0 , 1,1

X

X Y X Y



The event is given by ,0 , ,1 .

The probability function is

, , , 0,1.

X X X X

P X Y i j P X i P Y j X i i j

, , ,P X Y i j P X i Y j X i Y j

X i Y j

0,

10

2

X

P X

1,

11

2

X

P X

0Y

1Y

0.1

0.1

0.9

0.9

Figure 1.5-11


Example 1. 5-8 (2/3)


Because of a noise a transmitted zero sometimes gets

decoded as a received one and vice versa.

From measurements it is known that

1 1 0.9 1 0 0.1

0 1 0.1 0 0 0.9

P Y X P Y X

P Y X P Y X

and by design 0 1 0.5. The various joint

probabilities are then

P X P X

0, 0 0 0 0 0.45

0, 1 1 0 0 0.05

1, 0 0 1 1 0.05

1, 1 1 1 1 0.45

P X Y P Y X P X

P X Y P Y X P X

P X Y P Y X P X

P X Y P Y X P X


Example 1. 5-8 (3/3)


Independent

We will define two events and to be independent if A B

P A B P A P B

, 0P A B P A P B

P A B P A P BP B P B

and

, 0P B A P B P A



Example 1. 5-9 (1/3)

Two numbers and are selected at random between zero and

one. Let the events , , and be defined as follows:

x y

A B C

0.5 , 0.5 ,A x B y C x y

Are the events and independent?

Are the events and independent?

A B

A C



Ans ：

Figure 1.5-12 (a) shows the region of the unit square that

corresponds to the above events. Then, we have

1 4 1

|1 2 2

P A BP A B P A

P B

so events and are independent.A B

Figure 1.5-12

(a) Events and are independent.A B

A

B


Example 1. 5-9 (2/3)


Ans ：

Similarly,

3 8 3 1

|1 2 4 2

P A CP A C P A

P C

so events and are not independent. From Figure 1.5-12 (b),

we can see the knowledge of the fact that is greater than

increases the probability that is greater than 0.5.

A C

x y

x

Figure 1.5-12

(b) Events and are not independent.A C

A

C


Example 1. 5-9 (3/3)

Documents

CH1 Introduction - ecourse2.ccu.edu.tw