Probability

Raul Queiroz Feitosa

These slides are mostly inspired on slides from Christofer Bishop www.computervisiontalks.com/graphical-models-1-christopher-bishop-mlss-2013-tubingen/

11/6/2019 Probability 2

Objective

Recall some fundamentals of Probability Theory.

Interpretation of Probability

Frequentist:

Limit of an infinite number of trials.

Bayesian

A way to quantify uncertainty.

11/6/2019 Probability 3

Discrete Random Variables

A discrete random variable X can take on any value on a finite

or countable set of values X.

The probability that 𝑋=𝑥 is denoted by 𝑃 𝑋 = 𝑥 , or just, 𝑃 𝑥 ,

whereby and

𝑝() is called the probability mass function (pmf).

Example: for X={1,2,3,4}

11/6/2019 Probability 4

0 ≤ 𝑃 𝑥 ≤ 1 𝑃 𝑥 = 1

𝑥∈𝐗

1

1 2 3 4

.10

.30 .45

.15

𝑥

𝑝𝑥

A murder has been committed. Two suspects:

Butler Cook

There are three possible murder weapons:

Pistol Knife fireplace Poker

Murder mystery

11/6/2019 Probability 5

Prior Distribution

Prior Probability expresses the belief that an event

might occur without taking any evidence into

account.

Butler has served the family for many years.

Cook hired recently, rumors of dodgy history.

𝑃 Culprit = Butler = 20%

𝑃 Culprit = Cook = 80%

Probabilities add up to 100%.

11/6/2019 Probability 6

𝑃 Culprit

Culprit ∈ Butler, Cook

This is called a

factor graph

Conditional Distribution

Conditional distribution expresses the belief that

an event might occur given some observation(s) or

evidence(s).

Butler is an ex-army and keeps a pistol in a locked drawer.

Cook has access to lot of knives.

𝑃 Weapon Culprit)

11/6/2019 Probability 7

Pistol Knife Poker

Cook 5% 65% 30%

Butler 80% 10% 10%

=100%

=100%

Joint Distribution

Joint Probability expresses the belief that multiple

joint events occur.

What is the probability that the Cook committed the murder

with a Pistol?

𝑃 Culprit = Cook = 20%

𝑃 Weapon = Pistol | Culprit = Cook = 80%

𝑃 Weapon = Pistol , Culprit = Cook = 20% × 80% = 16%

Likewise for other combinations of Weapon and Culprit.

11/6/2019 Probability 8

Joint Distribution

𝑃(Weapon, Culprit)=𝑃 Weapon Culprit) 𝑃(Culprit)

product rule

11/6/2019 Probability 9

Pistol Knife Poker

Cook 4% 52% 24%

Butler 16% 2% 2% =100%

𝑃(𝑦, 𝑥)=𝑃 𝑦 𝑥) 𝑃(𝑥)

joint

distribution

conditional

distribution

prior/marginal

distribution

Generative Viewpoint

Murderer Weapon

Cook Knife

Butler Knife

Cook Pistol

Cook Poker

Cook Knife

Butler Pistol

Cook Poker

Cook Knife

Butler Pistol

Cook Knife

… …

11/6/2019 Probability 10

pistol

knife

poker

poker

pistol

knife

Cook

Butler

Marginal Distribution

Marginal Probability is the probability that an

event occurred obtained by summing over the

probabilities of all other events.

Given the joint distribution (weapon, culprit), what is the

probability distribution that murder was committed with a

Pistol?

𝑃 Weapon = Pistol , Culprit = Cook = 4% 𝑃 Weapon = Pistol , Culprit = Butler = 16%

𝑃 Weapon = Pistol = 4%+ 16% = 20%

Likewise for other weapons.

11/6/2019 Probability 11

Marginal Distribution 𝑃(Culprit)=𝑃 Weapon = Pistol, Culprit +

+𝑃 Weapon = Knife, Culprit + 𝑃 Weapon = Poker, Butler

𝑃(Weapon)=𝑃 Weapon, Culprit = Cook +

+𝑃(Weapon, Culprit = Butler)

sum rule

11/6/2019 Probability 12

Pistol Knife Poker Total

Cook 4% 52% 24% 80%

Butler 16% 2% 2% 20%

Total 20% 54% 26% 100%

𝑃 𝑥 = 𝑃(𝑥, 𝑦)

𝑦

marginal

distribution

of culprit

(=prior)!

joint

distributions

marginal distribution

of weapon

Posterior Distribution

Posterior Probability is the revised of prior after

receiving additional information.

A Pistol was found in the scene of the crime.

11/6/2019 Probability 13

Pistol Knife Poker

Cook 4% 52% 24%

Butler 16% 2% 2%

𝑃 Culprit Pistol) =𝑃(Weapon=Pistol,Culprit)

𝑃 Weapon=Pistol,Culprit=Cook +𝑃(Weapon=Pistol,Culprit=Butler)

= 𝑃 Weapon=Pistol Culprit)𝑝(Culprit)

𝑃(Weapon=Pistol)

Generative Viewpoint

Murderer Weapon

Cook Knife

Butler Knife

Cook Pistol

Cook Poker

Cook Knife

Butler Pistol

Cook Poker

Cook Knife

Butler Pistol

Cook Knife

… …

11/6/2019 Probability 14

A Pistol was found in the scene of the crime.

Posterior Distribution

.

Bayes rule

11/6/2019 Probability 15

Pistol Knife Poker

Cook 4% 52% 24%

Butler 16% 2% 2%

𝑃 𝑥|𝑦 =𝑃(𝑥, 𝑦)

𝑃(𝑦)

Joint distribution

Pistol Knife Poker

Cook 20% 96% 92%

Butler 80% 4% 8%

Posterior distribution 𝑝 Culprit Weapon)

𝑃 𝑥|𝑦 =𝑃 𝑦 𝑥 𝑃(𝑥)

𝑃(𝑦)

Bayes Theorem

It follows from the product rule

11/6/2019 Probability 16

𝑃(𝑦, 𝑥)=𝑃 𝑦 𝑥) 𝑃(𝑥) =𝑃 𝑥 𝑦) 𝑃(𝑦)

𝑃 𝑥 𝑦) = 𝑃 𝑦 𝑥) 𝑃(𝑥) 𝑃(𝑦)

prior

posterior

likelihood

𝑃 𝑦 = 𝑃 𝑦 𝑥 𝑃(𝑥)

𝑥

marginal

The Rules of Probability

Sum Rule

Product Rule

Bayes Theorem

Denominator

11/6/2019 Probability 17

𝑃 𝑥 = 𝑃(𝑥, 𝑦)

𝑦

𝑃(𝑥, 𝑦)=𝑃 𝑥 𝑦)𝑃(𝑦)=𝑃 𝑦 𝑥) 𝑃(𝑥)

𝑃 𝑦 𝑥) = 𝑃 𝑥 𝑦) 𝑃(𝑦) 𝑃(𝑥)

𝑃 𝑥 = 𝑃 𝑥 𝑦)𝑃(𝑦)

𝑦

Continuous Random Variables A continuous random variable X can take any real value.

The probability that 𝑋≤𝑞, denoted by 𝐹 𝑞 = 𝑃 𝑋 ≤ 𝑞 is called

cumulative probability density or cdf.

We define the probability density function - pdf as

11/6/2019 Probability 18

𝑝 𝑥 =𝑑𝐹 𝑥

𝑑𝑥

𝑝 𝑥

𝐹 𝑥

𝑥

𝑝𝑥

𝐹𝑥

May take values

greater than 1

The Rules of Probability

Assuming that x is continuous and y is discrete

Sum Rule

Product Rule

Bayes Theorem

Denominator

11/6/2019 Probability 19

𝑝 𝑥 = 𝑝(𝑥, 𝑦)

𝑦

𝑝(𝑥, 𝑦)=𝑝 𝑥 𝑦)𝑃 𝑦 𝑃(𝑦)=𝑃 𝑦 𝑥) 𝑝(𝑥)

𝑃 𝑦 𝑥) = 𝑝 𝑥 𝑦) 𝑃(𝑦)𝑝(𝑥)

𝑝 𝑥 𝑦) = 𝑃 𝑦 𝑥) 𝑝(𝑥) 𝑃(𝑦)

𝑝 𝑥 = 𝑝 𝑥 𝑦)𝑃 𝑦 𝑃 𝑦 = 𝑝 𝑥, 𝑦 𝑑𝑥+∞

−∞𝑦

𝑃 𝑦 = 𝑝 𝑥, 𝑦 𝑑𝑥+∞

−∞

Probability

END

11/6/2019 Graphical Models 20