Bayesian Estimation

Group H

Phạm Thiều Minh

Bùi Lê Quý Thái

Trần Diệp Huệ Mẫn

Nguyễn Phạm Xuân Quỳnh

Content

• Background• Bayesian estimation• Credible interval• Pros & Cons of Bayesian estimator• References

BACKGROUND

Example

Estimator

• Statistic used to estimate the value of an unknown parameter θ

Estimate

• Observed value of the estimator

EstimatePoint

Maximum Likelihood

BayesianInterval

Likelihood function

• We don’t know the parameters (for example mean μ or variance σ2)

• We have known data

From known data, we can calculate missing parameter

BAYESIAN ESTIMATION

What is Bayesian estimator?

Terminology

Squared error loss

Absolute value loss

Example

What is Bayesian estimator

• Bayesian estimator is an estimator that minimizes the expected loss (Bayes risk) of a given posterior distribution π(θ|D) over parameter θ.

Terminology

• Prior distribution π(θ): initial beliefs about some unknown quantity

• Likelihood function p(x|θ): information in the data

• Given data D, the posterior density

where)(

)()|()|(

data

priorlikelihoodposterior

xp

xpD

Terminology - example

• Prior distribution: uniform distribution on (0,1)

• Likelihood function

• Data

10,1)(

xxxp 1)1()|(

)()()|1

()1

( dpnxxp

nxxp

Terminology

• The mean of discrete random variable:

• The mean of the prior distribution:

• The mean of the posterior distribution:

)]([E(x) xPx

d)(E(x)

dD)|(D)|E(

Terminology

• Bayesian estimator:• True value: θ• Loss function - to find a lower value

that aindicate estimate is better estimate of θ

• Expected loss (Bayes risk):

How to minimize Bayes risk

Bayes risk

Squared error loss

Absolute error loss

Squared error loss (MSE)

• Other name is Minimum Squared Error (MSE)• Loss function:

= (true value – Bayesian estimator)2

• Bayes risk: • Minimize the risk by taking the 1st derivation =

0

The Bayes estimator of a parameter θ P with respect to squared loss is the mean of the posterior density

MSE - Example

MSE - Example

• Secondly, we calculate posterior density

)!1()( xx

Toss a coin 10 times, the number success (coin is head) is 6, then assuming a uniform (0,1) prior distribution on θThe posterior distribution is

MSE - Example

• Finally we evaluate Bayesian estimator

How to minimize Bayes risk

Bayes risk

Squared error loss

Absolute error loss

Absolute value loss

• Loss function: • Bayes risk:

• Minimize the risk by taking the 1st derivation to be 0

The Bayes estimator of a parameter θ P with respect to the absolute value loss is the median of the posterior density

CREDIBLE INTERVAL(HIGHEST DENSITY REGIONS )

What is HDR• Highest Density Regions (HDR’s) are intervals

containing a specified posterior probability. The figure below plots the 95% highest posterior density region.

HDR

PROS & CONS

Pros

• Incorporating prior knowledge into an analysis

• Loss functions allow a range of outcomes rather only 2 (the null & alternative hypothesis)

Past data

Present data

Cons

Posterior

REFERENCE

References

• Wikipedia (http://en.wikipedia.org/wiki/Bayes_estimator)

• FISH 497 course by Tim Esington (http://www.fish.washington.edu/classes/fish497/)

• Sheldon M. Ross – Probability and Statistics for Engineer and Scientists 3rd edition