PROBABILITY AND BAYES THEOREM

1

2

POPULATION SAMPLE

PROBABILITY

STATISTICAL INFERENCE

3

• PROBABILITY: A numerical value

expressing the degree of uncertainty regarding the occurrence of an event. A measure of uncertainty.

• STATISTICAL INFERENCE: The science of drawing inferences about the population based only on a part of the population, sample.

4

PROBABILITY• CLASSICAL INTERPRETATION If a random experiment is repeated an infinite

number of times, the relative frequency for any given outcome is the probability of this outcome.

Probability of an event: Relative frequency of the occurrence of the event in the long run.– Example: Probability of observing a head in a fair coin toss is 0.5 (if coin is

tossed long enough).

• SUBJECTIVE INTERPRETATION The assignment of probabilities to event of interest is

subjective– Example: I am guessing there is 50% chance of rain today.

5

PROBABILITY

• Random experiment– a random experiment is a process or course of action,

whose outcome is uncertain.

• Examples Experiment Outcomes

• Flip a coin Heads and Tails• Record a statistics test marks Numbers between 0

and 100• Measure the time to assemble Numbers from zero

and abovea computer

6

• Performing the same random experiment repeatedly, may result in different outcomes, therefore, the best we can do is consider the probability of occurrence of a certain outcome.

• To determine the probabilities, first we need to define and list the possible outcomes

PROBABILITY

7

• Determining the outcomes.– Build an exhaustive list of all possible

outcomes.– Make sure the listed outcomes are mutually

exclusive.

• The set of all possible outcomes of an experiment is called a sample space and denoted by S.

Sample Space

8

Sample Space

Countable Uncountable(Continuous )

Finite number of elements

Infinite number of elements

9

EXAMPLES• Countable sample space examples:

– Tossing a coin experimentS : {Head, Tail}

– Rolling a dice experimentS : {1, 2, 3, 4, 5, 6}

– Determination of the sex of a newborn childS : {girl, boy}

• Uncountable sample space examples:– Life time of a light bulb

S : [0, ∞)– Closing daily prices of a stock

S : [0, ∞)

Sample Space

• Multiple sample spaces for the same experiment are possible

• E.g. with 5 coin tosses we can take:S={HHHHH, HHHHT, …} or if we are only

interested in the number of heads we can take S*={0,1,2,3,4,5}

10

11

EXAMPLES

• Examine 3 fuses in sequence and note the results of each experiment, then an outcome for the entire experiment is any sequence of N’s (non-defectives) and D’s (defectives) of length 3. Hence, the sample space is

S : { NNN, NND, NDN, DNN, NDD, DND, DDN, DDD}

12

– Given a sample space S ={O1,O2,…,Ok}, the following characteristics for the probability P(Oi) of the simple event Oi must hold:

– Probability of an event: The probability P(A), of event A is the sum of the probabilities assigned to the simple events contained in A.

k

ii

i

OP

ieachforOP

1

1.2

10.1

k

ii

i

OP

ieachforOP

1

1.2

10.1

Assigning Probabilities

13

Assigning Probabilities

• P(A) is the proportion of times the event A is observed.

total outcomes in A( )

total outcomes in SP A

Set theory: Definitions

• Set: a set A is a collection of elements (or outcomes)• Membership: x A (x is in A), or x A (x is not in A)• Complement: • Union: • Intersection:• Difference:• Subset: A is contained in B• Equality: • Symmetric difference:

14

}:{ AxxAC

}:{ BxorAxxBA

}:{ BxandAxxBA

}:{\ BxandAxxABBA C

)( BxAxBA

ABandBAifBA

},:{ bothnotbutBxorAxxBA

Algebraic laws

• commutative: A B = B A∪ ∪  A ∩ B = B ∩ A• associative: (A B) C = A (B C)∪ ∪ ∪ ∪

A ∩ (B ∩ C) = (A ∩ B) ∩ C• distributive: A (B ∩ C) = (A B) ∩ (A C)∪ ∪ ∪

   A ∩ (B C) = (A ∩ B) (A ∩ C)∪ ∪• DeMorgan’s:   (A B)' = A' ∩ B' ∪ (' is complement)

   (A ∩ B)' = A' B'∪

15

16

Intersection

• The intersection of event A and B is the event that occurs when both A and B occur.

• The intersection of events A and B is denoted by (A and B) or AB.

• The joint probability of A and B is the probability of the intersection of A and B, which is denoted by P(A and B) or P(AB).

17

Union

• The union event of A and B is the event that occurs when either A or B or both occur.

• At least one of the events occur.

• It is denoted “A or B” OR AB

18

For any two events A and B

P(A B) = P(A) + P(B) - P(A B)P(A B) = P(A) + P(B) - P(A B)

Addition Rule

19

Complement Rule

• The complement of event A (denoted by AC) is the event that occurs when event A does not occur.

• The probability of the complement event is calculated by

P(AC) = 1 - P(A)P(AC) = 1 - P(A)A and AC consist of all the simple events in the sample space. Therefore,P(A) + P(AC) = 1

20

MUTUALLY EXCLUSIVE EVENTS

• Two events A and B are said to be mutually exclusive or disjoint, if A and B have no common outcomes. That is,

A and B = (empty set)

•The events A1,A2,… are pairwise mutually exclusive (disjoint), if Ai Aj = for all i j.

21

EXAMPLE

• The number of spots turning up when a six-sided dice is tossed is observed. Consider the following events.

A: The number observed is at most 2.

B: The number observed is an even number.

C: The number 4 turns up.

22

VENN DIAGRAM

• A graphical representation of the sample space.

2

S

1

3 5

4 6

A

BC

AB

1

4 6

A

B2

AB

1

4 6

A

B22

AC = A and C are mutually exclusive

23

AXIOMS OF PROBABILTY(KOLMOGOROV AXIOMS)

Given a sample space S, the probability function is a function P that satisfies

1) For any event A, 0 P(A) 1.

2) P(S) = 1.

3) If A1, A2,… are pairwise disjoint, then

11

)()(i

ii

i APAP

Probability

P : S [0,1]

Probability domain range

function

24

25

THE CALCULUS OF PROBABILITIES

• If P is a probability function and A is any

set, then

a. P()=0

b. P(A) 1

c. P(AC)=1 P(A)

26

THE CALCULUS OF PROBABILITIES

• If P is a probability function and A and B any sets, then

a. P(B AC) = P(B)P(A B)

b. If A B, then P(A) P(B)c. P(A B) P(A)+P(B) 1 (Bonferroni Inequality)

d. 1 2

11

for any sets A , ,i i

ii

P A P A A

(Boole’s Inequality)

Principle of Inclusion-Exclusion

• A generalization of addition rule

• Proof by induction

27

ji

nn

ji

n

ii

n

ii AAAPAAPAPAP )...()1(...)()()( 21

1

11

28

EQUALLY LIKELY OUTCOMES• The same probability is assigned to each simple

event in the sample space, S.

• Suppose that S={s1,…,sN} is a finite sample

space. If all the outcomes are equally likely, then

P({si})=1/N for every outcome si.

29

ODDS• The odds of an event A is defined by

( ) ( )( ) 1 ( )C

P A P AP A P A

•It tells us how much more likely to see the occurrence of event A.

•P(A)=3/4P(AC)=1/4 P(A)/P(AC) = 3. That is, the odds is 3. It is 3 times more likely that A occurs as it is that it does not.

ODDS RATIO

• OR is the ratio of two odds.

• Useful for comparing the odds under two different conditions or for two different groups, e.g. odds for males versus females.

• If odds of event A is 4.2 for males and 2 for females, then odds ratio is 2.1. The odds of observing event A is 2.1 times higher for males compared to females.

30

CONDITIONAL PROBABILITY

• (Marginal) Probability: P(A): How likely is it that an event A will occur when an experiment is performed?

• Conditional Probability: P(A|B): How will the probability of event A be affected by the knowledge of the occurrence or nonoccurrence of event B?

• If two events are independent, then P(A|B)=P(A)

31

CONDITIONAL PROBABILITY

32

)B|AA(P)B|A(P)B|A(P)B|AA(P

1)A|A(P

)B|A(P1)B|A(P

1)B|A(P0

0)B(Pif)B(P

)BA(P

212121

C

B)|P(A

Example• Roll two dice

• S=all possible pairs ={(1,1),(1,2),…,(6,6)}

• Let A=first roll is 1; B=sum is 7; C=sum is 8

• P(A|B)=?; P(A|C)=?• Solution:

• P(A|B)=P(A and B)/P(B)P(B)=P({1,6} or {2,5} or {3,4} or {4,3} or {5,2} or {6,1})

= 6/36=1/6

P(A|B)= P({1,6})/(1/6)=1/6 =P(A) A and B are independent

33

Example

• P(A|C)=P(A and C)/P(C)=P(Ø)/P(C)=0

A and C are disjoint

Out of curiosity:

P(C)=P({2,6} or {3,5} or {4,4} or {5,3} or {6,2})

= 5/36

CONDITIONAL PROBABILITY

35

)|()()|()()( BAPBPABPAPABP

),...,|()...,|()|()()...( 1121312121 nnn AAAPAAAPAAPAPAAAP

Example

• Suppose we pick 4 cards at random from a deck of 52 cards containing 4 aces.

• A=event that we pick 4 aces

• Ai=event that ith pick is an ace (i=1,2,3,4)

36

725,270/149

1

50

2

51

3

52

4)...|()()()(

725,270/14

52/1)(

1214321

AAPAPAAAAPAP

AP

BAYES THEOREM

• Suppose you have P(B|A), but need P(A|B).

37

0)B(Pfor)B(P

)A(P)A|B(P

)B(P

)BA(P)B|A(P

Example

• Let:

– D: Event that person has the disease;

– T: Event that medical test results positive

• Given:

– Previous research shows that 0.3 % of all Turkish population carries this disease; i.e., P(D)= 0.3 % = 0.003

– Probability of observing a positive test result for someone with the disease is 95%; i.e., P(T|D)=0.95

– Probability of observing a positive test result for someone without the disease is 4%; i.e. P(T| )= 0.04

• Find: probability of a randomly chosen person having the disease given that the test result is positive.

38

CD

Example

• Solution: Need P(D|T). Use Bayes Thm.

P(D|T)=P(T|D)*P(D)/P(T)

P(T)=P(D and T)+P( and T)

= 0.95*0.003+0.04*0.997 = 0.04273

P(D|T) =0.95*0.003 / 0.04273 = 6.67 %

Test is not very reliable!39

CD

BAYES THEOREM• Can be generalized to more than two events. • If Ai is a partition of S, then,

• Can be rewritten in terms of odds– Suppose A1,A2,… are competing hypotheses and B is evidence

or data relevant to choosing the correct hypothesis

Posterior odds = likelihood ratio x prior odds

40

jjj

iiiii APABP

APABP

BP

APABPBAP

)()|(

)()|(

)(

)()|()|(

)(

)(

)|(

)|(

)|(

)|(

j

i

j

i

j

i

AP

AP

ABP

ABP

BAP

BAP

Independence• A and B are independent iff

– P(A|B)=P(A) or P(B|A)=P(B)– P(AB)=P(A)P(B)

• A1, A2, …, An are mutually independent iff

for every subset j of {1,2,…,n}

E.g. for n=3, A1, A2, A3 are mutually independent iff P(A1A2A3)=P(A1)P(A2)P(A3) and P(A1A2)=P(A1)P(A2) and P(A1A3)=P(A1)P(A3) and P(A2A3)=P(A2)P(A3)

41

ji

iji

i APAP )()(

Independence

• If n=4, then the number of conditions for independence is

• Find these conditions.

42

112

4

3

4

4

4

Sequences of events

• A sequence of events A1, A2, … is increasing iff

• A sequence of events A1, A2, … is decreasing iff

• If {An} is increasing, then

• If {An} is decreasing, then

43

...321 AAA

...321 AAA

1

limi

inn

AA

1

limi

inn

AA

Examples

• Let S=(0,1) and An=(1/n,1)

{An} is increasing. What is limit of An as n goes to infinity?

• Let S=(0,1) and Bn=(0,1/n)

{Bn} is decreasing. What is limit of Bn as n goes to infinity?

44

Problems

1. Show that two nonempty events cannot be disjoint and independent at the same time.

Hint: First, prove that if they are disjoint, then they are not independent. Second, prove that if they are independent, then they are not disjoint.

Problems

2. If P(A)=1/3 and P(Bc)=1/4, can A and B be disjoint? Explain.

Problems

3. Either prove the statement is true or disprove it:

If P(B|A)=P(B|AC), then A and B are independent.

Problems

4. An insurance company has three types of customers – high risk, medium risk, and low risk. Twenty percent of its customers are high risk, and 30% are medium risk. Also, the probability that a customer has at least one accident in the current year is 0.25 for high risk, 0.16 for medium risk, and 0.1 for low risk.

a) Find the probability that a customer chosen at random will have at least one accident in the current year.

b) Find the probability that a customer is high risk, given that the person has had at least one accident during the current year.

Problems5. Eleven poker chips are numbered consecutively

1 through 10, with two of them labeled with a 6 and placed in a jar. A chip is drawn at random.

i) Find the probability of drawing a 6.

ii) Find the odds of drawing a 6 from the jar.

iii)Find the odds of not drawing a 6.

49

Problems

6. If the odds in favor of winning a horse race are 3:5, find the probability of winning the race.

50

Problems7. In a hypothetical clinical study, the following results

were obtained.

Find the odds ratio and interpret.

51

Treatment

Total number of patients treated

Number who achieved at least 50% pain relief

Number who did not achieve at least 50% pain relief

Ibuprofen 400 mg

40 22 18

Placebo 40 7 33