An introduction to bayesian statistics

An Introduction to

Bayesian Statistics

Paul HerendeenApril 2013

1960 1970 1980 1990 2000 20100

2000

4000

6000

8000WOS "Bayesian"Citations by Year

The rise of Bayesian statistics

So…What are Bayesian Statistics?

So…What are Bayesian Statistics?

1) A fundamentally different approach to probability

So…What are Bayesian Statistics?

1) A fundamentally different approach to probability

2) An associated set of mathematical tools

Frequentists vs. BayesianRound 1

Parameters fixed

Data varies

Data fixed

Parameters Vary

Frequentists vs. BayesianRound 1

Probability

Likelihood

Frequentists vs. BayesianRound 1

Confidence Interval

Credible Interval

Conditional Probability in 2 minutes

Conditional Probability in 2 minutes

All possible outcomes

Conditional Probability in 2 minutes

red blue

𝑃 (𝑅 ,𝐵 )=?

Conditional Probability in 2 minutes

red blue

𝑃 (𝑅 ,𝐵 )=¿𝑃 (𝑅 ) 𝑃 (𝐵 )

Conditional Probability in 2 minutes

red blue

𝑃 (𝐵|𝑅 )=¿𝑃 (𝐵 ,𝑅 )𝑃 (𝑅)  

Conditional Probability in 2 minutes

red blue

𝑃 (𝑅|𝐵 )= 𝑃 (𝐵|𝑅 )𝑃 (𝑅 )𝑃 (𝐵)

Conditional Probability in 2 minutes

red blue

𝑃 (𝑅|𝐵 )= 𝑃 (𝐵|𝑅 )𝑃 (𝑅 )𝑃 (𝐵)

Bayes’ Theorem

𝑃 (𝜃|𝐷 )= 𝑃 (𝐷|𝜃 ) 𝑃 (𝜃 )

∫𝑃 (𝐷|𝜃 ) 𝑃 (𝜃)

Bayes’ Theorem

𝑃 (𝜃|𝐷 )= 𝑃 (𝐷|𝜃 ) 𝑃 (𝜃 )𝑃 (𝐷)

Prior

Bayes’ Theorem

𝑃 (𝜃|𝐷 )= 𝑃 (𝐷|𝜃 ) 𝑃 (𝜃 )𝑃 (𝐷)

Likelihood

Prior

Bayes’ Theorem

𝑃 (𝜃|𝐷 )= 𝑃 (𝐷|𝜃 ) 𝑃 (𝜃 )𝑃 (𝐷)

Likelihood

Prior

Evidence

Bayes’ Theorem

𝑃 (𝜃|𝐷 )= 𝑃 (𝐷|𝜃 ) 𝑃 (𝜃 )𝑃 (𝐷)

Likelihood

Prior

Posterior

Evidence

Frequentist vs. BayesianRound 2

“The Strength of the Prior”

𝑃 (𝜃|𝐷 )= 𝑃 (𝜃 )𝑃 (𝐷|𝜃 )𝑃 (𝐷)

Sparse Data

𝑃 (𝜃|𝐷 )= 𝑃 (𝜃 )𝑃 (𝐷|𝜃 )𝑃 (𝐷)

Abundant Data

𝑃 (𝜃|𝐷 )= 𝑃 (𝜃 )𝑃 (𝐷|𝜃 )𝑃 (𝐷)

Uniform Prior

Where do Priors Come From?

So…What are Bayesian Statistics?

1) A fundamentally different approach to probability

2) An associated set of mathematical tools

𝑃 (𝜃|𝐷 )= 𝑃 (𝐷|𝜃 )𝑃 (𝜃 )

∫𝑃 (𝐷|𝜃 ) 𝑃 (𝐷)

How do you actually do this?

So how do you actually do this?

1.Analytical methods

So how do you actually do this?

1.Analytical methods

2.Grid approximation

So how do you actually do this?

1.Analytical methods

2.Grid approximation

3.Markov Chain Monte Carlo

MCMC• Algorithm for exploring parameter

space

MCMC• Algorithm for exploring parameter

space1.Pick a starting point

MCMC• Algorithm for exploring parameter

space1.Pick a starting point2.Propose a move

MCMC• Algorithm for exploring parameter

space1.Pick a starting point2.Propose a move3.Accept or decline move based on

probability

MCMC• Algorithm for exploring parameter

space1.Pick a starting point2.Propose a move3.Accept or decline move based on

probability• Time spent at each point

approximates parameter distribution

MCMC• Algorithm for exploring parameter space

1.Pick a starting point2.Propose a move3.Accept or decline move based on

probability• Time spent at each point approximates

parameter distribution• E.g. Metropolis-Hastings, Gibbs

sampling

MCMC2D example

MCMC2D example

So what does all this get us?

Bayesian methods really shine in complex (hierarchical) models…

For example,

IndividualFecundity

Group Effect

Population Effect

Foraging success

Environment

or…

Individual Fecundity

Group Effect

Population Effect

Environment

Many benefits to this approach

• Simultaneously estimate parameters• …as well as parameter relationships• “Borrow” strength across studies• Model comparison

So, is it a Bayesian Revolution?

Bayesian stats can do most things

frequentist,

Bayesian stats can do most things

frequentist, but…• Many simple models don’t gain

much• Better do something ‘boring’

well than something exciting poorly

Bayesian stats can do most things

frequentist, but…• Many simple models don’t gain

much• Better do something ‘boring’

well than something exciting poorly

• Don’t be this guy

DO use Bayesian methods if

• You have a complex model with many interacting parameters

• You have ‘messy’ data• You don’t want to make

assumptions about distributions

In Conclusion• Bayesian methods are powerful

tools for ecological research • Like most things statistical, they

are no substitute for thinking• They are here to stay, and you

should at least be familiar with them

Great, I want to learn more!

JAGS(Just Another Gibbs Sampler)