Upload
lavey
View
59
Download
2
Embed Size (px)
DESCRIPTION
Basics on Probability. Jingrui He 09/11/2007. Coin Flips. You flip a coin Head with probability 0.5 You flip 100 coins How many heads would you expect. Coin Flips cont. You flip a coin Head with probability p Binary random variable Bernoulli trial with success probability p - PowerPoint PPT Presentation
Citation preview
Basics on ProbabilityJingrui He
09/11/2007
Coin Flips You flip a coin
Head with probability 0.5
You flip 100 coins How many heads would you expect
Coin Flips cont. You flip a coin
Head with probability p Binary random variable Bernoulli trial with success probability p
You flip k coins How many heads would you expect Number of heads X: discrete random variable Binomial distribution with parameters k and p
Discrete Random Variables Random variables (RVs) which may take on
only a countable number of distinct values E.g. the total number of heads X you get if you
flip 100 coins
X is a RV with arity k if it can take on exactly one value out of E.g. the possible values that X can take on are 0,
1, 2,…, 100
1, , kx x
Probability of Discrete RV Probability mass function (pmf): Easy facts about pmf
if if
P X X 0i jx x
i j
1 2P X X X 1kx x x P X X P X P Xi j i jx x x x
i j
P X 1iix
P X ix
Common Distributions Uniform
X takes values 1, 2, …, N E.g. picking balls of different colors from a box
Binomial X takes values 0, 1, …, n
E.g. coin flips
X 1, ,U N
P X 1i N
X ,Bin n p
P X 1n iin
i p pi
Coin Flips of Two Persons Your friend and you both flip coins
Head with probability 0.5 You flip 50 times; your friend flip 100 times How many heads will both of you get
Joint Distribution Given two discrete RVs X and Y, their joint
distribution is the distribution of X and Y together E.g. P(You get 21 heads AND you friend get 70
heads)
E.g.
P X Y 1x y
x y
50 100
0 0P You get heads AND your friend get heads 1
i ji j
Conditional Probability is the probability of ,
given the occurrence of E.g. you get 0 heads, given that your friend gets
61 heads
P X Yx y
P X YP X Y
P Y
x yx y
y
X xY y
Law of Total Probability Given two discrete RVs X and Y, which take
values in and , We have 1, , mx x 1, , ny y
P X P X Y
P X Y P Y
i i jj
i j jj
x x y
x y y
Marginalization
Marginal Probability Joint Probability
Conditional Probability
P X P X Y
P X Y P Y
i i jj
i j jj
x x y
x y y
Marginal Probability
Bayes Rule X and Y are discrete RVs…
P Y X P XP X Y
P Y X P X
j i ii j
j k kk
y x xx y
y x x
P X YP X Y
P Y
x yx y
y
Independent RVs Intuition: X and Y are independent means that
neither makes it more or less probable that
Definition: X and Y are independent iff P X Y P X P Yx y x y
X xY y
More on Independence
E.g. no matter how many heads you get, your friend will not be affected, and vice versa
P X Y P X P Yx y x y
P X Y P Xx y x P Y X P Yy x y
Conditionally Independent RVs Intuition: X and Y are conditionally
independent given Z means that once Z is known, the value of X does not add any additional information about Y
Definition: X and Y are conditionally independent given Z iff
P X Y Z P X Z P Y Zx y z x z y z
More on Conditional Independence
P X Y Z P X Z P Y Zx y z x z y z
P X Y , Z P X Zx y z x z
P Y X , Z P Y Zy x z y z
Monty Hall Problem You're given the choice of three doors: Behind one
door is a car; behind the others, goats. You pick a door, say No. 1 The host, who knows what's behind the doors, opens
another door, say No. 3, which has a goat. Do you want to pick door No. 2 instead?
Host mustreveal Goat B
Host mustreveal Goat A
Host revealsGoat A
orHost reveals
Goat B
Monty Hall Problem: Bayes Rule : the car is behind door i, i = 1, 2, 3 : the host opens door j after you pick door i
iC
ijH
1 3iP C
0
0
1 2
1 ,
ij k
i j
j kP H C
i k
i k j k
Monty Hall Problem: Bayes Rule cont. WLOG, i=1, j=3
13 1 11 13
13
P H C P CP C H
P H
13 1 11 1 1
2 3 6P H C P C
Monty Hall Problem: Bayes Rule cont.
13 13 1 13 2 13 3
13 1 1 13 2 2
, , ,
1 11
6 31
2
P H P H C P H C P H C
P H C P C P H C P C
1 131 6 1
1 2 3P C H
Monty Hall Problem: Bayes Rule cont.
1 131 6 1
1 2 3P C H
You should switch!
2 13 1 131 2
13 3
P C H P C H
Continuous Random Variables What if X is continuous? Probability density function (pdf) instead of
probability mass function (pmf) A pdf is any function that describes the
probability density in terms of the input variable x.
f x
PDF Properties of pdf
Actual probability can be obtained by taking the integral of pdf E.g. the probability of X being between 0 and 1 is
0,f x x
1f x
1
0P 0 1X f x dx
1 ???f x
Cumulative Distribution Function Discrete RVs
Continuous RVs
X P XF v v
X P Xi
ivF v v
X
vF v f x dx
X
dF x f x
dx
Common Distributions Normal
E.g. the height of the entire population
2X ,N
22
1exp ,
2 2
xf x x
-5 -4 -3 -2 -1 0 1 2 3 4 50
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
x
f(x)
Common Distributions cont. Beta
: uniform distribution between 0 and 1 E.g. the conjugate prior for the parameter p in
Binomial distribution
X ,Beta
111; , 1 , 0,1
,f x x x x
B
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
x
f(x)
Joint Distribution Given two continuous RVs X and Y, the joint
pdf can be written as
X,Y ,f x y
X,Y , 1x y
f x y dxdy
Multivariate Normal Generalization to higher dimensions of the
one-dimensional normal
1X 1 22
1
1, ,
2
1exp
2
d d
T
f x x
x x
Covariance Matrix
Mean
Moments Mean (Expectation):
Discrete RVs:
Continuous RVs: Variance:
Discrete RVs:
Continuous RVs:
XE X P X
ii iv
E v v XE xf x dx
2X XV E 2X P X
ii iv
V v v 2XV x f x dx
Properties of Moments Mean
If X and Y are independent,
Variance If X and Y are independent,
X Y X YE E E
X XE a aE XY X YE E E
2X XV a b a V
X Y (X) (Y)V V V
Moments of Common Distributions Uniform
Mean ; variance Binomial
Mean ; variance Normal
Mean ; variance Beta
Mean ; variance
X 1, ,U N 1 2N 2 1 12N
X ,Bin n pnp 2np
2X ,N 2
X ,Beta 2
1
Probability of Events X denotes an event that could possibly happen
E.g. X=“you will fail in this course” P(X) denotes the likelihood that X happens,
or X=true What’s the probability that you will fail in this
course? denotes the entire event set
X,X
The Axioms of Probabilities 0 <= P(X) <= 1 , where are
disjoint events Useful rules
P 1
1 2P X X P Xii Xi
1 2 1 2 1 2P X X P X P X P X X
P X 1 P X
Interpreting the Axioms
1X2X