Statistical Inference Working backwards

Towards Statistical Inference

02/04/2014 ST2352 Week 9 1

Concepts Given -Prior prob information about …

-Data Y = y1, y2,…yn

what can we say about …., …?

• Seek Posterior prob, given info,

Key concept

• Likelihood= Jt Prob (Y given ) (forward prob)

Post prob Likelihood Prior

Point Estimates: value of … that maximises: (i) Post prob max a posteriori

(ii) Likelihood max likelihood estimator

Interval estimates

ST2352 Week 9 2

Classical Bayesian

02/04/2014

Working backwards

ST2352 Week 9 3

Data ( )process that gave rise to ?

Model realisation of rv

can be simulated; data generating system

system ,paramete

obeserve

rs

;

Se p

d

ek r

Y

y aspects of y

y Y

Y DGS Z

aspects of unobserved Y

Y pdf f y Y transform Z

|ob dist |

Procedure

1 Simulate random from knowledge, absent data

2 Prefer values for which likelihood of these data is high

; Pr | ;

OR equivalent, if algebra/models simple

Y

Y

f Y y

f

f y Y y L y

Prior Likelihood Posterior Mechanics

Data like this

02/04/2014

Partially Observed Stoch Sys: Signal and Noise

AR1 process {Zt} Observed with error as {Yt } Reconstruct {Zt}

02/04/2014 ST2352 Week 9 4

Too complicated Hence undeveloped

Inference for Signal Data Gen System A

• = Z generated by AR process – Z0=0

– =0.9

– σ=1

• Y generated by Z + – σ =0.1

Monte Carlo study Simulate many series Z Resample Prefer Z ‘close’ to Y which make Y ‘likely or As above via prob theory rather than MC

Known simple case

02/04/2014 ST2352 Week 9 5

Sampling: Inference for p

• RedC report Sample Survey

– 1000 chosen randomly

– 53% “agree”

• What can we say

– about true value p?

• eg p ‘close to’ 0.53

• eg p ‘probably’ > 0.5

Data like this

02/04/2014 ST2352 Week 9 6

Bayesian Inference for p Data Gen System

= p chosen randomly

from dist fP(p) prior

A Finite pop N

– Np have value 1,N(1-p) value 0

– Sample indep w’out replacement

– Y = number of 1’s

(B Process

– Z = 1 Prob = p; else 0 indep

– Obs Y )

Monte Carlo study Simulate many p from prior Resample from posterior prefer p which make Y ‘likely’ or via prob theory 02/04/2014 ST2352 Week 9 7

Classical Inference for p Data Gen System

p unknown

A Finite pop N

– Np have value 1,N(1-p) value 0

– Sample indep w’out replacement

– Y = number of 1’s

(B Process

– Z= 1 Prob = p; else 0 indep

– Obs Y )

Estimate p Use sampling dist interval estimate Prob theory as tho’ p known CLT approx Prob theory recognising p estimated

02/04/2014 ST2352 Week 9 8

Classical Inference for Signal Estimate unknown “paras” Plug in ‘as tho known’ Prob theory Prob Theory recognising using estimator

Data Gen System B

• Z generated by AR process – Z0=?

– =?

– σ=?

• X generated by Z + – σ =0.1

02/04/2014 ST2352 Week 9 9

Bayesian Inference for Signal Z “paras”

Study joint posterior f, Y=y(, ; y) Marginalise wrt f|Y=y(; y)

Data Gen System B

• Z generated by AR process – Z0=?

– =?

– σ=?

• Y generated by Z + – σ =0.1

02/04/2014 ST2352 Week 9 10

Classical Inference for μ given sample x1,… xn

• Use sample mean as estimator of μ

• Precision eg 95% Interval 1. Assume (approx) Normality in DGS

Estimate variance by s2

Approx Mean 2 SE

Better Mean t-dist value SE recognising var estimated

2. Avoid Normal assumption Use Monte Carlo technique ‘bootstrap’ later

02/04/2014 ST2352 Week 9 11

Prior – Likelihood - Posterior

ST2352 Week 9 12

Data Generating System for

; ; ;

Bayesian Approach: update prior

Posterior ; ;

Point Estimate eg most probable, max

Int Estimate from Posterior

Classical A

Y Y

Y

Y

Y pdf f y L y f y

f

f y L y f

a posteriori

pproach: use only ;

ˆPoint Estimate eg MLE

ˆInt Estimate from Sampling Dist

L y

Data like this

02/04/2014

The Likelihood Function

1 1 1 1

Pr Observe these data, , given values of everything else

Pr

Pr , Pr

/ ;

sufficient to within multiplicative function of only

Pri

t t t t

ii

t

Y

y

L Y y

Y y

Y y Y y Y y if Markov

jt pmf pdf f y gen

Y y if ind

e

ep

ral

y

02/04/2014 ST2352 Week 9 13

(log)Likelihood: Binomial

,

Pr 1

;

;

n yy

Y B n p

nY y p p

y

L p y

l p y const

02/04/2014 ST2352 Week 9 14

Bayesian: posterior

Prior a p∼U (0,1)

b p∼N 0.3,0.12( )Data (i) n =10, y = 2

(ii) n =100, y = 20

02/04/2014 ST2352 Week 9 15

Binomial: Maximum Likelihood

; 1 ; ln ln 1

(10,2) (100,20)

ˆFind value of based on data

ˆPropose that value of which

renders to be th

best

me value of

ˆMaximised at

;

ost

;

k y

0

li el

n yyL p y p p l p y const y p n y p

p y p y Estimator

p y p

this y Y

p p

dL p y dl p y

dp

1ˆ0 np ydp

02/04/2014 ST2352 Week 9 16

Normal dist: Likelihood

2

2

2

21

1 22

2

22 1 1

22

21 1 1

22

/1

; 1,...

,

, ; exp

exp

i

i

i

y

i

n n

i

yy y

nn

Data y i n

Model Y N indep

L y

y

e e

22

2 2

i i

i

y y y y

y y n y

02/04/2014 ST2352 Week 9 17

Normal dist: posterior

2

21

1 22

2 2

2

21

122

2 2

22

/2 1

2 2

,

/2 2 1

,

2

,

, ;

Prior ,

Post ,

; , ;

i

i

yy y

nn

y y y

n n

Y y

YY

L y e e

f eg f f

f f e f e

f y f y

02/04/2014 ST2352 Week 9 18

Normal dist: Likelihood and MLE

2

2 2

2

2

2

; 1,...

,

ˆ ˆ,

, ;0

, ;0

iData y i n

Model Y N indep

Maximised at

L y

L y

m = 1

nyiå = y

s 2 = 1

nyi- y( )å

2

= n-1

ns2

02/04/2014 ST2352 Week 9 19

Normal dist MLE: properties

2

22 211 1

221 1

21 1

2 2 2 22( 1)1

ˆ ˆ

Consider as functions of random Sampling Dists properties

ˆ ˆ

ˆ ˆ ˆ; ,

ˆ ˆ;

nn n ni i

n ni i

nn

nnn n

Data y y y y y s

Y

Y Y Y Y

E Var and N

Unbiased Consistent

E Var an

2

2

2 1

ˆn

nd

Biased Consistent

02/04/2014 ST2352 Week 9 20

Maximum Likelihood

Model Observed is a realisation of random variable

; known, up to unknown parameter

ˆ ˆ maximises ;

ˆ arg max ;

ˆExplicit formula in special cases

Numerical algorithm in general

Y

Y

Y

y Y

f y

MLE y f y wrt

f y

y

02/04/2014 ST2352 Week 9 21

Maximum Likelihood

Model Observed is a realisation of random variable

; known, up to unknown parameter

ˆ ˆFor many distributions has properties:

ˆasymtotically unbiased E

ˆconsistent 0

asymtotic

Y

n

n

y Y

f y

Y

Y

Var Y

ˆ ˆ ˆally E ,Y N Y Var Y

02/04/2014 ST2352 Week 9 22