ENGG 2040C: Probability Models and Applications

Andrej Bogdanov

Spring 2014

8. Limit theorems

Many times we do not need to calculate probabilities exactly

An approximate or qualitative estimate often sufficesP(magnitude 7+ earthquake within 10 years) = ?

This is often a much easier task

What do you think?

I toss a coin 1000 times. The probability that I get a streak of 14 consecutive heads is

< 10% ≈ 50% > 90%A B C

Consecutive heads

where Ii is an indicator r.v. for the event “14 consecutive heads starting at position i”

Let N be the number of occurrences of 14 consecutive heads in 1000 coin flips.

N = I1 + … + I987

E[Ii ] = P(Ii = 1) = 1/214E[N ] = 987 ⋅ 1/214

= 987/16384

≈ 0.0602

Markov’s inequality

For every non-negative random variable X and every value a:

P(X ≥ a) ≤ E[X] / a.

E[N ] ≈ 0.0602

P[N ≥ 1] ≤ E[N ] / 1 ≤ 6%.

Proof of Markov’s inequality

For every non-negative random variable X: and every value a:

P(X ≥ a) ≤ E[X] / a.

E[X ] = E[X | X ≥ a ] P(X ≥ a) + E[X | X < a ] P(X < a)

≥ 0≥ a ≥ 0

E[X ] ≥ a P(X ≥ a) + 0.

Hats

1000 people throw their hats in the air. What is the probability at least 100 people get their hat back?

N = I1 + … + I1000where Ii is the indicator for the event that person i

gets their hat. Then E[Ii ] = P(Ii = 1) = 1/n

Solution

E[N ] = n 1/n

= 1 P[N ≥ 100] ≤ E[N ] / 100 = 1%.

Patterns

A coin is tossed 1000 times. Give an upper bound on the probability that the pattern HH occurs:

(b) at most 100 times

(a) at least 500 times

Patterns

Let N be the number of occurrences of HH.

P[N ≥ 500] ≤ E[N ] / 500

= 249.75/500

≈ 49.88%so 500+ HHs occur with probability ≤

49.88%.P[N ≤ 100] ≤ ? P[999 – N ≥

899]

(b)P[N ≤ 100] =

≤ E[999 – N ] / 899= (999 – 249.75)/

899≤ 83.34%

Last time we calculated E[N ] = 999/4 = 249.75.

(a)

Computer simulation of patterns# toss n coins and count number of consecutive head pairsdef consheads(n): count = 0 lastone = randint(0, 1) thisone = randint(0, 1) for i in range(n - 1): if lastone == 1 and thisone == 1: count = count + 1 lastone = thisone thisone = randint(0, 1) return count

>>> for i in range(100): print(consheads(1000), end = “ ”)

264 260 256 263 272 224 256 254 275 231 242 232 247 268 229 270 231 272 241 238 257 239 251 252 255 249 267 223 272 254 219 266 271 265 212 262 239 253 265 254 262 231 271 242 258 255 219 281 238 246 242 263 245 239 270 199 251 229 240 253 282 258 237 276 247 221 242 226 232 244 222 258 255 294 239 267 253 259 236 239 236 243 254 240 232 248 270 252 232 282 248 244 251 223 226 222 288 266 268 236

Chebyshev’s inequality

For every random variable X and every t:

P(|X – m| ≥ ts) ≤ 1 / t2.

where m = E[X], s = √Var[X].

Patterns

E[N ] = 999/4 = 249.75Var[N] = (5⋅999 – 7)/16 = 311.75

m = 249.75s ≈ 17.66

(a)

P(X ≥ 500)≤ P(|X – m| ≥ 14.17s) ≤ 1/14.172 ≈ 0.50%

(b)

P(X ≤ 100)≤ P(|X – m| ≥ 8.47s) ≤ 1/8.472 ≈ 1.39%

Proof of Chebyshev’s inequality

For every random variable X and every a:

P(|X – m| ≥ ts) ≤ 1 / t2.

where m = E[X], s = √Var[X].

P(|X – m| ≥ ts) = P((X – m)2 ≥ t2s2) ≤ E[(X – m)2] / t2s2 = 1 / t2.

An illustration

mm – ts m + ts

sP(|X – m| ≥ ts ) ≤ 1 / t2.

m a

P( X ≥ a ) ≤ m / a.

0

Markov’s inequality:

Chebyshev’s inequality:

Polling

1

2

3

45 6

7 8

9

Polling

Xi =1 if i

0 if i

X1,…, Xn are independent Bernoulli(m)

where m is the fraction of blue voters

X = X1 + … + Xn

X/n is the pollster’s estimate of m

Polling

How accurate is the pollster’s estimate X/n?

E[X] =

= mn E[X1] + … + E[Xn]

Var[X]

= Var [X1] + … + Var [Xn] = s2n

m = E[Xi], s = √Var[Xi]

X = X1 + … + Xn

Polling

E[X] = mnVar[X] = s2n

P( |X – mn| ≥ ts √n ) ≤ 1 / t2.

P( |X/n – m| ≥ e) ≤ d.

confidenceerror

samplingerror

X = X1 + … + Xn

den

The weak law of large numbers

For every e, d > 0 and n ≥ s2/(e2d): P(|X/n – m| ≥ e) ≤ d

X1,…, Xn are independent with same p.m.f. (p.d.f.)m = E[Xi], s = √Var[Xi],

X = X1 + … + Xn

Polling

Say we want confidence error d = 10% and sampling error e = 5% . How many people should we poll?

For e, d > 0 and n ≥ s2/(e2d):

P(|X/n – m| ≥ e) ≤ d

n ≥ s2/(e2d) ≥ 4000s2

For Bernoulli(m) samples, s2 = m (1 – m) ≤ 1/4

This suggests we should poll about 1000 people.

A polling simulation

number of people polled n

X 1 +

… +

Xn

nX1, …, Xn independent Bernoulli(1/2)

polls

ter’s

est

imat

e

A polling simulation

number of people polled n

X 1 +

… +

Xn

n

20 simulations

polls

ter’s

est

imat

e

A more precise estimate

Let’s assume n is large.

Weak law of large numbers:X1 + … + Xn ≈ mn with high

probability

X1,…, Xn are independent with same p.m.f. (p.d.f.)

P( |X – mn| ≥ ts √n ) ≤ 1 / t2.

this suggests X1 + … + Xn ≈ mn + Ts √n

Some experiments

X = X1 + … + Xn Xi independent Bernoulli(1/2)

n = 6

n = 40

Some experiments

X = X1 + … + Xn Xi independent Poisson(1)

n = 3

n = 20

Some experiments

X = X1 + … + Xn Xi independent Uniform(0, 1)

n = 2

n = 10

The normal random variable

f(t) = (2p)-½ e-t /22

tp.d.f. of a normal random variable

The central limit theorem

X1,…, Xn are independent with same p.m.f. (p.d.f.)

where T is a normal random variable.

m = E[Xi], s = √Var[Xi], X = X1 + … + Xn

For every t (positive or negative):

lim P(X ≤ mn + ts √n ) = P(T ≤ t)n → ∞

Polling again

Probability model

X = X1 + … + Xn Xi independent Bernoulli(m)

m = fraction that will vote blue

E[Xi] = m, s = √Var[Xi] = √m(1 - m) ≤ ½.

Say we want confidence error d = 10% and sampling error e = 5% . How many people should we poll?

Polling again

lim P(X ≤ mn – ts √n ) = P(T ≤ -t)n → ∞

5% n

lim P(X ≥ mn + ts √n ) = P(T ≥ t)n → ∞

5% n

lim P(X/n is not within 5% of m) = P(T ≤ -t) + P(T ≥ t)n → ∞

= 2 P(T ≤ -t)

ts √n = 5% n t = 5%√n/s

The c.d.f. of a normal random variable

t

F(t)

P(T ≤ -t)t-t

P(T ≥ t)

Polling again

confidence error = 2 P(T ≤ -t)

We want a confidence error of ≤ 10%:

= 2 P(T ≤ -5%√n/s)≤ 2 P(T ≤ -√n/10)

We need to choose n so that P(T ≤ -√n/10) ≤ 5%.

Polling again

t

F(t)

P(T ≤ -√n/10) ≤ 5%

-√n/10 ≈ -1.645

n ≈ 16.452 ≈ 271

http://stattrek.com/online-calculator/normal.aspx

Party

Give an estimate of the probability that the average arrival time of a guest is past 8:40pm.

Ten guests arrive independently at a party between 8pm and 9pm.

Acute triangles

Drop three points at random on a square. What is the probability that they form an acute triangle?

Simulation

# indicate whether the triangle with the given vertices is acutedef is_acute(x1, y1, x2, y2, x3, y3):

def dot(x1, y1, x2, y2, x0, y0): return (x1 - x0) * (x2 - x0) + (y1 - y0) * (y2 - y0)

a1 = dot(x2, y2, x3, y3, x1, y1) a2 = dot(x3, y3, x1, y1, x2, y2) a3 = dot(x1, y1, x2, y2, x3, y3) return a1 > 0 and a2 > 0 and a3 > 0

# count the fraction of acute triangles among n random samplesdef simulate_triangles(n): count = 0 for i in range(n): if is_acute(uniform(0.0, 1.0), uniform(0.0, 1.0), uniform(

count = count + 1 return 1.0 * count / n

Idea: Conduct a poll among random triangles!

Simulation

Want sampling error e = .01, confidence error d = .051. Rigorous estimate: By weak law of large numbers, we can choose n = s2/(e2d) ≤ 50,000 > simulate_triangles(50000) 0.27326> simulate_triangles(50000)0.27392> simulate_triangles(50000)0.27612

Simulation

Want sampling error e = .01, confidence error d = .052. Non-rigorous (but better) estimate: Central limit theorem suggests choosing n such that ts √n ≤ en, P(Normal < -t) = d

> simulate_triangles(5366) 0.28158777487886694> simulate_triangles(5366)0.27003354453969436> simulate_triangles(5366)0.2849422288483041

t = 1.465

n = (t/2e)2 ≈ 5366