Continuous Random Variables: Joint PDFs, Conditioning ...berlin.csie.ntnu.edu.tw/Courses/Probability... · More than Two Random Variables • The joint PDF of three random variables

Berlin ChenDepartment of Computer Science & Information Engineering

National Taiwan Normal University

Continuous Random Variables: Joint PDFs, Conditioning, Expectation and Independence

Reference:- D. P. Bertsekas, J. N. Tsitsiklis, Introduction to Probability , Sections 3.4-3.6

Probability-Berlin Chen 2

Multiple Continuous Random Variables (1/2)

• Two continuous random variables and associated with a common experiment are jointly continuous and can be described in terms of a joint PDF satisfying

– is a nonnegative function

– Normalization Probability

• Similarly, can be viewed as the “probability per unit area” in the vicinity of

– Where is a small positive number

X Y

Byx

YX dxdyyxfBYX,

, ,,P

YXf ,

1,, dxdyyxf YX

YXf ,

caf YX ,, ca,

2

,, ,,

,

cafdxdyyxf

cYcaXa

YXaa

cc YX

P


Multiple Continuous Random Variables (2/2)

• Marginal Probability

– We have already defined that

• We thus have the marginal PDF

Similarly

AX YX dydxyxf

YAXAX

,

, and

,

PP

AX X dxxfAXP

dyyxfxf YXX ,,

dxyxfyf YXY ,,


An Illustrative Example

• Example 3.10. Two-Dimensional Uniform PDF. We are told that the joint PDF of the random variables and is a constant on an area and is zero outside. Find the value of and the marginal PDFs of and .

X YS

cc

X Y

Sx,yx,yf

Sx,yx,yf

S

YX

YX

for 41

otherwise ,0

if ,S area of Size

13.9) Example (cf. be todefined is areaan on PDFjoint uniform ingcorrespond The

,

,

31

31

21

21

21

41

32for 41

41

21for

dx

dxx,yfyf

y

dx

dxx,yfyf

y

X,YY

X,YY

32

32

41

41

41

41

32for 43

41

21

dy

dyx,yfxf

x

dy

dyx,yfxf

xfor

X,YX

X,YX

21

21

41

41

43for

dx

dxx,yfyf

y

X,YY


Joint CDFs• If and are two (either continuous or discrete)

random variables associated with the same experiment , their joint cumulative distribution function (Joint CDF) is defined by

– If and further have a joint PDF ( and are continuous random variables) , then

And

If can be differentiated at the point

X Y

yYxXyxF YX ,,, P

X Y YXf ,

yx

yxFyxf YX

YX

,, ,

2

,

x y

YXYX dsdttsfyxF ,, ,,

YXF , yx,

X Y


An Illustrative Example

• Example 3.12. Verify that if X and Y are described by a uniform PDF on the unit square, then the joint CDF is given by

10for ,,,, x,yxyyYxXyxF YX P

squareunit in the , allfor ,,1,

,,

2yxyxf

yxyxF

YXYX

X

Y 1,1

0,0 0,1

1,0


Expectation of a Function of Random Variables

• If and are jointly continuous random variables, and is some function, then is also a random variable (can be continuous or discrete)– The expectation of can be calculated by

– If is a linear function of and , e.g., , then

• Where and are scalars

X Yg YXgZ ,

Z

dxdyyxfyxgYXgZ YX ,,, ,EE

Z X Y bYaXZ

YbXabYaXZ EEEE

a b

We will see in Section 4.1 methods for computing the PDF of (if it has one).Z

More than Two Random Variables

• The joint PDF of three random variables , and is defined in analogy with the case of two random variables

– The corresponding marginal probabilities

• The expected value rule takes the form

– If is linear (of the form ), then


X Y Z

BZYX

ZYX dxdydzzyxfBZYX,,

,, ,,,,P

dydzzyxfxf

dzzyxfyxf

ZYXX

ZYXYX

,,

,,,

,,

,,,

g

dzdxdyzyxfzyxgZYXg ZYX ,,,,,, ,,E

ZcYbXacZbYaX EEEE

cZbYaX


Conditioning PDF Given an Event (1/3)

• The conditional PDF of a continuous random variable , given an event– If cannot be described in terms of , the conditional PDF

is defined as a nonnegative function satisfying

• Normalization property

XA

XA

dxxfABX B AXP

xf AX

1 dxxf AX



– If can be described in terms of ( is a subset of the real line with ), the conditional PDF is defined as a nonnegative function satisfying

• The conditional PDF is zero outside the conditioning event

and for any subset

– Normalization Property

A xf AX

X A 0 AXP

otherwise ,0

if , AxAX

xfxf

X

AX P

B

BA AX

BA X

dxxfAX

dxxfAX

AXBXAXBX

,

P

PPP

1 A AXAX dxxfdxxf

remains the same shape as except that it is scaled along

the vertical axis

AXf

Xf



• If are disjoint events with for each , that form a partition of the sample space, then

– Verification of the above total probability theorem

nAAA ,,, 21 0iAPi

n

iAXiX xfAxf

i1P

n

iAXiX

n

i

xAXi

xX

n

iii

xfAxf

x

dttfAdttf

AxXAxX

i

i

1

1

1

respect to with sidesboth of derivative theTaking

P

P

PPP

think of as an event , and use the total probability theorem from Chapter 1

xX B


Illustrative Examples (1/2)

• Example 3.13. The exponential random variable is memoryless.– The time T until a new light bulb burns out is exponential

distribution. John turns the light on, leave the room, and when he returns, t time units later, find that the light bulb is still on, which corresponds to the event A={T>t}

– Let X be the additional time until the light bulb burns out. What is the conditional PDF of X given A ?

tetTP

tetf

Tt

T

otherwise ,0

0,lexponentia is

x

t

xt

ee

etTP

xtTPtTP

tTxtTPtTxtTP

xtTxtTPAxXPAX

tTAtTX

and )0 (where by defined is given of CDF lconditiona The

,

.parameter with lexponentia also is event the

given of PDF lconditiona The

A

X



• Example 3.14. The metro train arrives at the station near your home every quarter hour starting at 6:00 AM. You walk into the station every morning between 7:10 and 7:30 AM, with the time in this interval being a uniform random variable. What is the PDF of the time you have to wait for the first train to arrive?

yPBPyPAPyP BYAYY

BYAY

XBB

XAA

Y

X

on dconditione uniform be Let -on dconditione uniform be Let -

train)30:7 theboard(You 30:715:7 event a be Let -

train)15:7 theboard(You 15:710:7 event a be Let -

time waiting themodel variblerandomLet -30:7 to10:7 interval over the variable

random uniform a is ,by denoted time,arrival The -

1/20

201

151

430

41 ,155For

101

151

43

51

41 ,50For

yPy

yPy

Y

Y

Total Probability theorem:


Conditioning one Random Variable on Another

• Two continuous random variables and have a joint PDF. For any with , the conditional PDF of given that is defined by

– Normalization Property

• The marginal, joint and conditional PDFs are related to each other by the following formulas

,,, yxfyfyxf YXYYX

.,, dyyxfxf YXX

X YXy 0yfY

yY

yf

yxfyxf

Y

YXYX

,,

1 dxyxf YX

marginalization



• Notice that the conditional PDF has the same shape as the joint PDF , because the normalizing factor does not depend on

Figure 3.16: Visualization of the conditional PDF . Let , have a joint PDF which is uniform on the set . For each fixed , we consider the joint PDF along the slice and normalize it so that it integrates to 1

yxf YX yxf YX ,,

yfY x

yxf YX

Xy

SYyY

1

4/14/1

5.35.3,

5.3 , Y

YXYX f

xfxf

1

4/14/1

5.15.1,

5.3 , Y

YXYX f

xfxf

2/1

2/14/1

5.25.2,

5.3 , Y

YXYX f

xfxf

cf. example 3.13



• Example 3.15. Circular Uniform PDF. Ben throws a dart at a circular target of radius . We assume that he always hits the target, and that all points of impact are equally likely, so that the joint PDF of the random variables and is uniform– What is the marginal PDF

yxf YX ,, yfY

otherwise ,0

,1

otherwise ,0

circle in the is ,if ,circle theof area

1,

2222

,

ryxπr

yxyxf YX

yx yx ,

r

uniform)not is PDF that here (Notice

2

1111

1

222

22

2

22

22222

222

yf

r y, if yrπr

dxπr

dxπr

dxπr

dxx,yfyf

Y

yryrryx

ryxX,YY

22222

222

2

2

1

2

1

ry if x, yr

yrπr

πr

yfx,yf

yxfY

X,YYX

For each value , is uniformy yxf YX


Conditional Expectation Given an Event

• The conditional expectation of a continuous random variable , given an event ( ), is defined by

– The conditional expectation of a function also has the form

– Total Expectation Theorem

and

• Where are disjoint events with for each , that form a partition of the sample space

X A

dxxxfAX AXE

Xg

dxxfxgAXg AXE

n

iii AXAX

1EPE

0AP

nAAA ,,, 21 0iAP

i

n

iii AXgAXg

1EPE


An Illustrative Example• Example 3.17. Mean and Variance of a Piecewise Constant PDF.

Suppose that the random variable has the piecewise constant PDF

X

otherwise. ,0

,21 if ,3/2,10 if ,3/1

xx

xf X

3/23/2 ,3/13/1

[1,2] interval second in the lies event [0,1] intervalfirst in the lies event Define

212

101

2

1

dxAdxA

XAXA

PP

3 and 2 is ][

intervalan over variablerandom uniform a ofmoment second andmean that theRecall

22 /baba/baa, b

36/116/79/15var

9/153/73/23/13/1 EEE

6/72/33/22/13/1 EEE

2

22

212

12

2211

X

AXAAXAX

AXAAXAX

PP

PP

3/7E,2/3E

3/1E,2/1E

22

2

12

1

AXAX

AXAX

otherwise ,0

2 1 ,1

otherwise ,0

1 0 ,121 21

xAXP

xfxf

xAXP

xfxf

X

AX

X

AX


Conditional Expectation Given a Random Variable

• The properties of unconditional expectation carry though, with the obvious modifications, to conditional expectation

dxyxxfyYX YXE

dxyxfxgyYXg YXE

dxyxfyxgyYYXg YX,,E


Total Probability/Expectation Theorems

• Total Probability Theorem– For any event and a continuous random variable

• Total Expectation Theorem– For any continuous random variables and

A

dyyfyYAA YPP

YX

Y

dyyfyYXX YEE

dyyfyYXgXg YEE

dyyfyYYXgYXg Y,, EE


Independence

• Two continuous random variables and are independent if

– Since that

• We therefore have

• Or

X Y

x,yyfxfyxf YXYX allfor ,,,

0 with all and allfor , yfyxxfyxf YXYX

0 with all and allfor , xfxyyfxyf XYXY

xyfxfyxfyfyxf XYXYXYYX ,,


More Factors about Independence (1/2)

• If two continuous random variables and are independent, then– Any two events of the forms and are

independent

– It also implies that

– The converse statement is also true (See the end-of-chapter problem 28)

X Y

AX BY

BYAX

dyyfdxxf

dydxyfxf

dydxyxfBYAX

By YAx X

Ax By YX

Ax By YX

PP

P

,, ,

xFxFyYxXyYxXyxF YXYX PPP ,,,


More Factors about Independence (2/2)

• If two continuous random variables and are independent, then–

–

– The random variables and are independent for any functions and

• Therefore,

YXXY EEE

Xg Yhg h

YhXgYhXg EEE

YXYX varvarvar

X Y


Recall: the Discrete Bayes’ Rule

• Let be disjoint events that form a partition of the sample space, and assume that , for all . Then, for any event such that we have

nAAA ,,, 21

0iAP iB 0BP

nn

ii

nk kk

ii

iii

ABAABAABA

ABAABA

BABA

BA

PPPPPP

PPPP

PPP

P

11

1

Multiplication ruleTotal probability theorem


Inference and the Continuous Bayes’ Rule

• As we have a model of an underlying but unobserved phenomenon, represented by a random variable with PDF , and we make a noisy measurement , which is modeled in terms of a conditional PDF . Once the experimental value of is measured, what information does this provide on the unknown value of ?

XYXf

XYfY

X

Measurement InferenceX

xf X

Y

xyf XY yxf YX

dttyftf

xyfxf

yfyxf

yxfXYX

XYX

Y

YXYX

,,

YXYYXXYX fffff , thatNote


Inference and the Continuous Bayes’ Rule (2/2)

• If the unobserved phenomenon is inherently discrete– Let is a discrete random variable of the form that

represents the different discrete probabilities for the unobserved phenomenon of interest, and be the PMF of

N nN

Np N

iNYN

NYN

Y

NYN

iyfip

nyfnpyf

nyfnpyYy

nNyYynN

yYynNyYnN

PPP

PP

Total probability theorem

Inference about a Discrete Random Variable


Illustrative Examples (1/2)• Example 3.19. A lightbulb produced by the General Illumination

Company is known to have an exponentially distributed lifetime . However, the company has been experiencing quality control problems. On any given day, the parameter of the PDF of is actually a random variable, uniformly distributed in the interval

. – If we test a lightbulb and record its lifetime ( ), what can

we say about the underlying parameter ?

Y

Y

yY

0,0 , yeyf y

Y

otherwise 0,2/31for ,2

f

231for ,

222/3

12/3

1

/λdtte

edttyftf

yffyf ty

y

Y

YY

2/3 ,1

Conditioned on , has a exponential distribution with parameter

Y



• Example 3.20. Signal Detection. A binary signal is transmitted, and we are given that and . – The received signal is , where is a normal noise

with zero mean and unit variance , independent of .– What is the probability that , as a function of the observed value

of ?

pS 1PS

pS 11PNSY N

S

y--sesyf sySY and ,1 and 1for ,

21 2/2

Conditioned on , has a normal distribution with mean and unit varianceYsS s

1SYy

yy

y

yyyy

yy

yy

y

SYSSYS

SYS

Y

SYS

eppepe

epepee

pee

epep

ep

yfpyfp

yfp

yf

yfpyYS

11

211

21

21

1111

11111

2/12/1

2/1

2/12/1

2/1

22

2

22

2

P

Inference Based on a Discrete Random Variable

• The earlier formula expressing in terms of can be turned around to yield


yYA P yf AY

dttYAtf

yYAyfA

yYAyfyf

Y

Y

YAY

P

PPP

?

)1:propertyion normalizat(

dyyfdyyYAyfA

dyyYAyfdyyfA

yYAyfyfA

AYY

YAY

YAY

PP

PP

PP


Recitation

• SECTION 3.4 Joint PDFs of Multiple Random Variables– Problems 15, 16

• SECTION 3.5 Conditioning– Problems 18, 20, 23, 24

• SECTION 3.6 The Continuous Bayes’ Rule– Problems 34, 35

Documents

Continuous Random Variables: Joint PDFs, Conditioning ...berlin.csie.ntnu.edu.tw/Courses/Probability... · More than Two Random Variables • The joint PDF of three random variables