Shanghai tutorial

From MCMC to ABC Methods


Christian P. Robert

Universite Paris-Dauphine, IuF, & CREST

http://www.ceremade.dauphine.fr/~xian

O’Bayes 11, Shanghai, June 10, 2011

http://www.ceremade.dauphine.fr/~xian


Outline

Computational issues in Bayesian statistics

The Metropolis-Hastings Algorithm

The Gibbs Sampler

Approximate Bayesian computation

ABC for model choice



A typology of Bayes computational problems

(i). use of a complex parameter space, as for instance inconstrained parameter sets like those resulting from imposingstationarity constraints in dynamic models;





(ii). use of a complex sampling model with an intractablelikelihood, as for instance in some latent variable or graphicalmodels or in inverse problems;






(iii). use of a huge dataset;







(iv). use of a complex prior distribution (which may be theposterior distribution associated with an earlier sample);







(iv). use of a complex prior distribution (which may be theposterior distribution associated with an earlier sample);

(v). use of a particular inferential procedure as for instance, Bayesfactors

Bπ01(x) =

P (θ ∈ Θ0 | x)P (θ ∈ Θ1 | x)

/π(θ ∈ Θ0)

π(θ ∈ Θ1).





The Metropolis-Hastings AlgorithmMonte Carlo basicsMonte Carlo Methods based on Markov ChainsThe Metropolis–Hastings algorithmRandom-walk Metropolis-Hastings algorithms

The Gibbs Sampler





Monte Carlo basics

General purpose

Given a density π known up to a normalizing constant, and anintegrable function h, compute

Π(h) =

∫h(x)π(x)µ(dx) =

∫h(x)π(x)µ(dx)∫π(x)µ(dx)

when∫h(x)π(x)µ(dx) is intractable.



Monte Carlo basics

Monte Carlo 101

Generate an iid sample x1, . . . , xN from π and estimate Π(h) by

ΠMCN (h) = N−1

N∑

i=1

h(xi).

LLN: ΠMCN (h)

as−→ Π(h)If Π(h2) =

∫h2(x)π(x)µ(dx) < ∞,

CLT:√N(ΠMC

N (h)−Π(h))

L N

(0,Π

{[h−Π(h)]2

}).



Monte Carlo basics

Monte Carlo 101

Generate an iid sample x1, . . . , xN from π and estimate Π(h) by

ΠMCN (h) = N−1

N∑

i=1

h(xi).

LLN: ΠMCN (h)

as−→ Π(h)If Π(h2) =

∫h2(x)π(x)µ(dx) < ∞,

CLT:√N(ΠMC

N (h)−Π(h))

L N

(0,Π

{[h−Π(h)]2

}).

Caveat announcing MCMC

Often impossible or inefficient to simulate directly from Π



Monte Carlo basics

Importance Sampling

For Q proposal distribution such that Q(dx) = q(x)µ(dx),alternative representation

Π(h) =

∫h(x){π/q}(x)q(x)µ(dx).



Monte Carlo basics

Importance Sampling

For Q proposal distribution such that Q(dx) = q(x)µ(dx),alternative representation

Π(h) =

∫h(x){π/q}(x)q(x)µ(dx).

Principle of importance

Generate an iid sample x1, . . . , xN ∼ Q and estimate Π(h) by

ΠISQ,N (h) = N−1

N∑

i=1

h(xi){π/q}(xi).



Monte Carlo basics

Properties of importance

ThenLLN: ΠIS

Q,N (h)as−→ Π(h) and if Q((hπ/q)2) < ∞,

CLT:√N(ΠIS

Q,N (h)−Π(h))L N

(0, Q{(hπ/q −Π(h))2}

).



Monte Carlo basics

Properties of importance

ThenLLN: ΠIS

Q,N (h)as−→ Π(h) and if Q((hπ/q)2) < ∞,

CLT:√N(ΠIS

Q,N (h)−Π(h))L N

(0, Q{(hπ/q −Π(h))2}

).

Caveat

If normalizing constant of π unknown, impossible to use ΠISQ,N

Generic problem in Bayesian Statistics: π(θ|x) ∝ f(x|θ)π(θ).



Monte Carlo basics

Self-Normalised Importance Sampling

Self normalized version

ΠSNISQ,N (h) =

(N∑

i=1

{π/q}(xi))−1 N∑

i=1

h(xi){π/q}(xi).



Monte Carlo basics



ΠSNISQ,N (h) =

(N∑

i=1

{π/q}(xi))−1 N∑

i=1

h(xi){π/q}(xi).

LLN : ΠSNISQ,N (h)

as−→ Π(h)

and if Π((1 + h2)(π/q)) < ∞,

CLT :√N(ΠSNIS

Q,N (h)−Π(h))L N

(0, π {(π/q)(h−Π(h)}2)

).



Monte Carlo basics



ΠSNISQ,N (h) =

(N∑

i=1

{π/q}(xi))−1 N∑

i=1

h(xi){π/q}(xi).

LLN : ΠSNISQ,N (h)

as−→ Π(h)

and if Π((1 + h2)(π/q)) < ∞,

CLT :√N(ΠSNIS

Q,N (h)−Π(h))L N

(0, π {(π/q)(h−Π(h)}2)

).

c© The quality of the SNIS approximation depends on thechoice of Q



Monte Carlo Methods based on Markov Chains

Running Monte Carlo via Markov Chains (MCMC)

It is not necessary to use a sample from the distribution f toapproximate the integral

I =

∫h(x)f(x)dx ,




Running Monte Carlo via Markov Chains (MCMC)

It is not necessary to use a sample from the distribution f toapproximate the integral

I =

∫h(x)f(x)dx ,

We can obtain X1, . . . , Xn ∼ f (approx) without directlysimulating from f , using an ergodic Markov chain withstationary distribution f




Running Monte Carlo via Markov Chains (2)

Idea

For an arbitrary starting value x(0), an ergodic chain (X(t)) isgenerated using a transition kernel with stationary distribution f




Running Monte Carlo via Markov Chains (2)

Idea

For an arbitrary starting value x(0), an ergodic chain (X(t)) isgenerated using a transition kernel with stationary distribution f

◮ Insures the convergence in distribution of (X(t)) to a randomvariable from f .

◮ For a “large enough” T0, X(T0) can be considered as

distributed from f

◮ Produce a dependent sample X(T0), X(T0+1), . . ., which isgenerated from f , sufficient for most approximation purposes.



The Metropolis–Hastings algorithm


BasicsThe algorithm uses the target density

f

and a conditional densityq(y|x)

called the instrumental (or proposal) distribution




The MH algorithm

Algorithm (Metropolis–Hastings)

Given x(t),

1. Generate Yt ∼ q(y|x(t)).2. Take

X(t+1) =

{Yt with prob. ρ(x(t), Yt),

x(t) with prob. 1− ρ(x(t), Yt),

where

ρ(x, y) = min

{f(y)

f(x)

q(x|y)q(y|x) , 1

}.




Features

◮ Independent of normalizing constants for both f and q(·|x)(ie, those constants independent of x)

◮ Never move to values with f(y) = 0

◮ The chain (x(t))t may take the same value several times in arow, even though f is a density wrt Lebesgue measure

◮ The sequence (yt)t is usually not a Markov chain




Convergence properties

Under irreducibility,

1. The M-H Markov chain is reversible, withinvariant/stationary density f since it satisfies the detailedbalance condition

f(y)K(y, x) = f(x)K(x, y)




Convergence properties

Under irreducibility,

1. The M-H Markov chain is reversible, withinvariant/stationary density f since it satisfies the detailedbalance condition

f(y)K(y, x) = f(x)K(x, y)

2. As f is a probability measure, the chain is positive recurrent




Convergence properties (2)

4. Ifq(y|x) > 0 for every (x, y), (2)

the chain is irreducible







5. For M-H, f -irreducibility implies Harris recurrence







5. For M-H, f -irreducibility implies Harris recurrence

6. Thus, for M-H satisfying (1) and (2)(i) For h, with Ef |h(X)| < ∞,

limT→∞

1

T

T∑

t=1

h(X(t)) =

∫h(x)df(x) a.e. f.

(ii) and

limn→∞

∥∥∥∥∫

Kn(x, ·)µ(dx)− f

∥∥∥∥TV

= 0

for every initial distribution µ, where Kn(x, ·) denotes thekernel for n transitions.



Random-walk Metropolis-Hastings algorithms

Random walk Metropolis–Hastings

Use of a local perturbation as proposal

Yt = X(t) + εt,

where εt ∼ g, independent of X(t).The instrumental density is of the form g(y − x) and the Markovchain is a random walk if we take g to be symmetric g(x) = g(−x)




Algorithm (Random walk Metropolis)

Given x(t)

1. Generate Yt ∼ g(y − x(t))

2. Take

X(t+1) =

Yt with prob. min

{1,

f(Yt)

f(x(t))

},

x(t) otherwise.




Optimizing the Acceptance Rate

Problem of choice of the transition kernel from a practical point ofviewMost common alternatives:

1. an instrumental density g which approximates f , such thatf/g is bounded for uniform ergodicity to apply;

2. a random walk

In both cases, the choice of g is critical,




Case of the random walk

Different approach to acceptance ratesA high acceptance rate does not indicate that the algorithm ismoving correctly since it indicates that the random walk is movingtoo slowly on the surface of f .




Case of the random walk

Different approach to acceptance ratesA high acceptance rate does not indicate that the algorithm ismoving correctly since it indicates that the random walk is movingtoo slowly on the surface of f .If x(t) and yt are close, i.e. f(x(t)) ≃ f(yt) y is accepted withprobability

min

(f(yt)

f(x(t)), 1

)≃ 1 .

For multimodal densities with well separated modes, the negativeeffect of limited moves on the surface of f clearly shows.




Case of the random walk (2)

If the average acceptance rate is low, the successive values of f(yt)tend to be small compared with f(x(t)), which means that therandom walk moves quickly on the surface of f since it oftenreaches the “borders” of the support of f




Case of the random walk (2)

If the average acceptance rate is low, the successive values of f(yt)tend to be small compared with f(x(t)), which means that therandom walk moves quickly on the surface of f since it oftenreaches the “borders” of the support of f

In small dimensions, aim at an average acceptance rate of 50%. Inlarge dimensions, at an average acceptance rate of 25%.

[Gelman,Gilks and Roberts, 1995]




Example (Noisy AR(1))

Hidden Markov chain from a regular AR(1) model,

xt+1 = ϕxt + ǫt+1 ǫt ∼ N (0, τ2)

and observablesyt|xt ∼ N (x2t , σ

2)




Example (Noisy AR(1))

Hidden Markov chain from a regular AR(1) model,

xt+1 = ϕxt + ǫt+1 ǫt ∼ N (0, τ2)

and observablesyt|xt ∼ N (x2t , σ

2)

The distribution of xt given xt−1, xt+1 and yt is

exp−1

2τ2

{(xt − ϕxt−1)

2 + (xt+1 − ϕxt)2 +

τ2

σ2(yt − x2t )

2

}.




Example (Noisy AR(1) continued)

For a Gaussian random walk with scale ω small enough, therandom walk never jumps to the other mode. But if the scale ω issufficiently large, the Markov chain explores both modes and give asatisfactory approximation of the target distribution.




Markov chain based on a random walk with scale ω = .1.




Markov chain based on a random walk with scale ω = .5.




MA(2)

xt = ǫt − θ1ǫt−1 − θ2ǫt−2

Since the constraints on (ϑ1, ϑ2) are well-defined, use of a flatprior over the triangle as prior.Simple representation of the likelihood

library(mnormt)

ma2like=function(theta){

n=length(y)

sigma = toeplitz(c(1 +theta[1]^2+theta[2]^2,

theta[1]+theta[1]*theta[2],theta[2],rep(0,n-3)))

dmnorm(y,rep(0,n),sigma,log=TRUE)

}




Basic RWHM for MA(2)

Algorithm 1 RW-HM-MA(2) sampler

set ω and ϑ(1)

for i = 2 to T dogenerate ϑj ∼ U(ϑ(i−1)

j − ω, ϑ(i−1)j + ω)

set p = 0 and ϑ(i) = ϑ(i−1)

if ϑ within the triangle thenp = exp(ma2like(ϑ)−ma2like(ϑ(i−1)))

end ifif U < p thenϑ(i) = ϑ

end ifend for




Outcome

Result with a simulated sample of 100 points and ϑ1 = 0.6,ϑ2 = 0.2 and scale ω = 0.2




Outcome





Outcome



The Gibbs Sampler

The Gibbs Sampler

The Gibbs SamplerGeneral PrinciplesSlice samplingConvergence


The Gibbs Sampler

General Principles

General Principles

A very specific simulation algorithm based on the targetdistribution f :

1. Uses the conditional densities f1, . . . , fp from f


The Gibbs Sampler

General Principles

General Principles



2. Start with the random variable X = (X1, . . . , Xp)


The Gibbs Sampler

General Principles

General Principles



2. Start with the random variable X = (X1, . . . , Xp)

3. Simulate from the conditional densities,

Xi|x1, x2, . . . , xi−1, xi+1, . . . , xp

∼ fi(xi|x1, x2, . . . , xi−1, xi+1, . . . , xp)

for i = 1, 2, . . . , p.


The Gibbs Sampler

General Principles

Algorithm (Gibbs sampler)

Given x(t) = (x(t)1 , . . . , x

(t)p ), generate

1. X(t+1)1 ∼ f1(x1|x(t)2 , . . . , x

(t)p );

2. X(t+1)2 ∼ f2(x2|x(t+1)

1 , x(t)3 , . . . , x

(t)p ),

. . .

p. X(t+1)p ∼ fp(xp|x(t+1)

1 , . . . , x(t+1)p−1 )

X(t+1) → X ∼ f


The Gibbs Sampler

General Principles

Properties

The full conditionals densities f1, . . . , fp are the only densities usedfor simulation. Thus, even in a high dimensional problem, all ofthe simulations may be univariate


The Gibbs Sampler

General Principles

Properties

The full conditionals densities f1, . . . , fp are the only densities usedfor simulation. Thus, even in a high dimensional problem, all ofthe simulations may be univariateThe Gibbs sampler is not reversible with respect to f . However,each of its p components is. Besides, it can be turned into areversible sampler, either using the Random Scan Gibbs sampler

see section or running instead the (double) sequence

f1 · · · fp−1fpfp−1 · · · f1


The Gibbs Sampler

General Principles

Limitations of the Gibbs sampler

Formally, a special case of a sequence of 1-D M-H kernels, all withacceptance rate uniformly equal to 1.The Gibbs sampler

1. limits the choice of instrumental distributions


The Gibbs Sampler

General Principles




2. requires some knowledge of f


The Gibbs Sampler

General Principles





3. is, by construction, multidimensional


The Gibbs Sampler

General Principles





3. is, by construction, multidimensional

4. does not apply to problems where the number of parametersvaries as the resulting chain is not irreducible.


The Gibbs Sampler

General Principles

A wee mixture problem

−1 0 1 2 3 4

−1

01

23

4

µ1

µ2

Gibbs started at random


The Gibbs Sampler

General Principles

A wee mixture problem

−1 0 1 2 3 4

−1

01

23

4

µ1

µ2

Gibbs started at random

Gibbs stuck at the wrong mode

−1 0 1 2 3

−1

01

23

µ1

µ2


The Gibbs Sampler

Slice sampling

Slice sampler as generic Gibbs

If f(θ) can be written as a product

k∏

i=1

fi(θ),


The Gibbs Sampler

Slice sampling

Slice sampler as generic Gibbs

If f(θ) can be written as a product

k∏

i=1

fi(θ),

it can be completed as

k∏

i=1

I0≤ωi≤fi(θ),

leading to the following Gibbs algorithm:


The Gibbs Sampler

Slice sampling

Algorithm (Slice sampler)

Simulate

1. ω(t+1)1 ∼ U[0,f1(θ(t))]

;

. . .

k. ω(t+1)k ∼ U[0,fk(θ(t))]

;

k+1. θ(t+1) ∼ UA(t+1) , with

A(t+1) = {y; fi(y) ≥ ω(t+1)i , i = 1, . . . , k}.


The Gibbs Sampler

Slice sampling

Example of results with a truncated N (−3, 1) distribution

0.0 0.2 0.4 0.6 0.8 1.0

0.00

00.

002

0.00

40.

006

0.00

80.

010

x

y

Number of Iterations 2


The Gibbs Sampler

Slice sampling


0.0 0.2 0.4 0.6 0.8 1.0

0.00

00.

002

0.00

40.

006

0.00

80.

010

x

y

Number of Iterations 2, 3


The Gibbs Sampler

Slice sampling


0.0 0.2 0.4 0.6 0.8 1.0

0.00

00.

002

0.00

40.

006

0.00

80.

010

x

y

Number of Iterations 2, 3, 4


The Gibbs Sampler

Slice sampling


0.0 0.2 0.4 0.6 0.8 1.0

0.00

00.

002

0.00

40.

006

0.00

80.

010

x

y

Number of Iterations 2, 3, 4, 5


The Gibbs Sampler

Slice sampling


0.0 0.2 0.4 0.6 0.8 1.0

0.00

00.

002

0.00

40.

006

0.00

80.

010

x

y

Number of Iterations 2, 3, 4, 5, 10


The Gibbs Sampler

Slice sampling


0.0 0.2 0.4 0.6 0.8 1.0

0.00

00.

002

0.00

40.

006

0.00

80.

010

x

y

Number of Iterations 2, 3, 4, 5, 10, 50


The Gibbs Sampler

Slice sampling


0.0 0.2 0.4 0.6 0.8 1.0

0.00

00.

002

0.00

40.

006

0.00

80.

010

x

y

Number of Iterations 2, 3, 4, 5, 10, 50, 100


The Gibbs Sampler

Slice sampling

Good slices, tough slices

The slice sampler usually enjoys good theoretical properties (likegeometric ergodicity and even uniform ergodicity under bounded fand bounded X ).As k increases, the determination of the set A(t+1) may getincreasingly complex.


The Gibbs Sampler

Convergence

Properties of the Gibbs sampler

Theorem (Convergence)

For(Y1, Y2, · · · , Yp) ∼ g(y1, . . . , yp),

if either[Positivity condition]

(i) g(i)(yi) > 0 for every i = 1, · · · , p, implies thatg(y1, . . . , yp) > 0, where g(i) denotes the marginal distributionof Yi, or

(ii) the transition kernel is absolutely continuous with respect to g,

then the chain is irreducible and positive Harris recurrent.


The Gibbs Sampler

Convergence

Properties of the Gibbs sampler (2)

Consequences

(i) If∫h(y)g(y)dy < ∞, then

limnT→∞

1

T

T∑

t=1

h1(Y(t)) =

∫h(y)g(y)dy a.e. g.

(ii) If, in addition, (Y (t)) is aperiodic, then

limn→∞

∥∥∥∥∫

Kn(y, ·)µ(dx)− f

∥∥∥∥TV

= 0

for every initial distribution µ.


The Gibbs Sampler

Convergence

Hammersley-Clifford theorem

An illustration that conditionals determine the joint distribution

Theorem

If the joint density g(y1, y2) have conditional distributionsg1(y1|y2) and g2(y2|y1), then

g(y1, y2) =g2(y2|y1)∫

g2(v|y1)/g1(y1|v) dv.

[Hammersley & Clifford, circa 1970]


The Gibbs Sampler

Convergence

General HC decomposition

Under the positivity condition, the joint distribution g satisfies

g(y1, . . . , yp) ∝p∏

j=1

gℓj (yℓj |yℓ1 , . . . , yℓj−1 , y′ℓj+1

, . . . , y′ℓp)

gℓj (y′ℓj|yℓ1 , . . . , yℓj−1 , y

′ℓj+1

, . . . , y′ℓp)

for every permutation ℓ on {1, 2, . . . , p} and every y′ ∈ Y .






The Gibbs Sampler

Approximate Bayesian computationABC basicsAlphabet soupCalibration of ABC




ABC basics

Untractable likelihoods

Cases when the likelihood function f(y|θ) is unavailable and whenthe completion step

f(y|θ) =∫

Z

f(y, z|θ) dz

is impossible or too costly because of the dimension of z

c© MCMC cannot be implemented!



ABC basics

Illustrations

Example

Stochastic volatility model: fort = 1, . . . , T,

yt = exp(zt)ǫt , zt = a+bzt−1+σηt ,

T very large makes it difficult toinclude z within the simulatedparameters

0 200 400 600 800 1000

−0.

4−

0.2

0.0

0.2

0.4

t

Highest weight trajectories



ABC basics

Illustrations

Example

Potts model: if y takes values on a grid Y of size kn and

f(y|θ) ∝ exp

{θ∑

l∼i

Iyl=yi

}

where l∼i denotes a neighbourhood relation, n moderately largeprohibits the computation of the normalising constant



ABC basics

Illustrations

Example

Inference on CMB: in cosmology, study of the Cosmic MicrowaveBackground via likelihoods immensely slow to computate (e.gWMAP, Plank), because of numerically costly spectral transforms[Data is a Fortran program]

[Kilbinger et al., 2010, MNRAS]



ABC basics

Illustrations

Example

Phylogenetic tree: in populationgenetics, reconstitution of a commonancestor from a sample of genes viaa phylogenetic tree that is close toimpossible to integrate out[100 processor days with 4parameters]

[Cornuet et al., 2009, Bioinformatics]



ABC basics

The ABC method

Bayesian setting: target is π(θ)f(x|θ)



ABC basics

The ABC method

Bayesian setting: target is π(θ)f(x|θ)When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:



ABC basics

The ABC method

Bayesian setting: target is π(θ)f(x|θ)When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:

ABC algorithm

For an observation y ∼ f(y|θ), under the prior π(θ), keep jointlysimulating

θ′ ∼ π(θ) , z ∼ f(z|θ′) ,until the auxiliary variable z is equal to the observed value, z = y.

[Tavare et al., 1997]



ABC basics

Why does it work?!

The proof is trivial:

f(θi) ∝∑

z∈D

π(θi)f(z|θi)Iy(z)

∝ π(θi)f(y|θi)= π(θi|y) .

[Accept–Reject 101]



ABC basics

A as approximative

When y is a continuous random variable, equality z = y is replacedwith a tolerance condition,

(y, z) ≤ ǫ

where is a distance



ABC basics

A as approximative

When y is a continuous random variable, equality z = y is replacedwith a tolerance condition,

(y, z) ≤ ǫ

where is a distanceOutput distributed from

π(θ)Pθ{(y, z) < ǫ} ∝ π(θ|(y, z) < ǫ)



ABC basics

ABC algorithm

Algorithm 2 Likelihood-free rejection sampler 2

for i = 1 to N dorepeatgenerate θ′ from the prior distribution π(·)generate z from the likelihood f(·|θ′)

until ρ{η(z), η(y)} ≤ ǫset θi = θ′

end for

where η(y) defines a (not necessarily sufficient) statistic



ABC basics

Output

The likelihood-free algorithm samples from the marginal in z of:

πǫ(θ, z|y) =π(θ)f(z|θ)IAǫ,y

(z)∫Aǫ,y×Θ π(θ)f(z|θ)dzdθ ,

where Aǫ,y = {z ∈ D|ρ(η(z), η(y)) < ǫ}.



ABC basics

Output

The likelihood-free algorithm samples from the marginal in z of:

πǫ(θ, z|y) =π(θ)f(z|θ)IAǫ,y

(z)∫Aǫ,y×Θ π(θ)f(z|θ)dzdθ ,

where Aǫ,y = {z ∈ D|ρ(η(z), η(y)) < ǫ}.The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution:

πǫ(θ|y) =∫

πǫ(θ, z|y)dz ≈ π(θ|y) .



ABC basics

MA example

Back to the MA(q) model

xt = ǫt +

q∑

i=1

ϑiǫt−i

Simple prior: uniform over the inverse [real and complex] roots in

Q(u) = 1−q∑

i=1

ϑiui

under the identifiability conditions



ABC basics

MA example

Back to the MA(q) model

xt = ǫt +

q∑

i=1

ϑiǫt−i

Simple prior: uniform prior over the identifiability zone, e.g.triangle for MA(2)



ABC basics

MA example (2)

ABC algorithm thus made of

1. picking a new value (ϑ1, ϑ2) in the triangle

2. generating an iid sequence (ǫt)−q<t≤T

3. producing a simulated series (x′t)1≤t≤T



ABC basics

MA example (2)

ABC algorithm thus made of

1. picking a new value (ϑ1, ϑ2) in the triangle

2. generating an iid sequence (ǫt)−q<t≤T

3. producing a simulated series (x′t)1≤t≤T

Distance: basic distance between the series

ρ((x′t)1≤t≤T , (xt)1≤t≤T ) =T∑

t=1

(xt − x′t)2

or distance between summary statistics like the q autocorrelations

τj =T∑

t=j+1

xtxt−j



ABC basics

Comparison of distance impact

Evaluation of the tolerance on the ABC sample against bothdistances (ǫ = 100%, 10%, 1%, 0.1%) for an MA(2) model



ABC basics





ABC basics





ABC basics

ABC advances

Simulating from the prior is often poor in efficiency



ABC basics

ABC advances

Simulating from the prior is often poor in efficiencyEither modify the proposal distribution on θ to increase the densityof x’s within the vicinity of y...

[Marjoram et al, 2003; Bortot et al., 2007; Sisson et al., 2007]



ABC basics

ABC advances



...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ǫ

[Beaumont et al., 2002]



ABC basics

ABC advances



...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ǫ

[Beaumont et al., 2002]

.....or even by including ǫ in the inferential framework [ABCµ][Ratmann et al., 2009]



Alphabet soup

ABC-NP

Better usage of [prior] simulations byadjustement: instead of throwing awayθ′ such that ρ(η(z), η(y)) > ǫ, replaceθs with locally regressed

θ∗ = θ − {η(z)− η(y)}Tβ[Csillery et al., TEE, 2010]

where β is obtained by [NP] weighted least square regression on(η(z)− η(y)) with weights

Kδ {ρ(η(z), η(y))}

[Beaumont et al., 2002, Genetics]



Alphabet soup

ABC-MCMC

Markov chain (θ(t)) created via the transition function

θ(t+1) =

θ′ ∼ Kω(θ′|θ(t)) if x ∼ f(x|θ′) is such that x = y

and u ∼ U(0, 1) ≤ π(θ′)Kω(θ(t)|θ′)

π(θ(t))Kω(θ′|θ(t)),

θ(t) otherwise,



Alphabet soup

ABC-MCMC

Markov chain (θ(t)) created via the transition function

θ(t+1) =

θ′ ∼ Kω(θ′|θ(t)) if x ∼ f(x|θ′) is such that x = y

and u ∼ U(0, 1) ≤ π(θ′)Kω(θ(t)|θ′)

π(θ(t))Kω(θ′|θ(t)),

θ(t) otherwise,

has the posterior π(θ|y) as stationary distribution[Marjoram et al, 2003]



Alphabet soup

ABC-MCMC (2)

Algorithm 3 Likelihood-free MCMC sampler

Use Algorithm 2 to get (θ(0), z(0))for t = 1 to N doGenerate θ′ from Kω

(·|θ(t−1)

),

Generate z′ from the likelihood f(·|θ′),Generate u from U[0,1],

if u ≤ π(θ′)Kω(θ(t−1)|θ′)

π(θ(t−1)Kω(θ′|θ(t−1))IAǫ,y

(z′) then

set (θ(t), z(t)) = (θ′, z′)else(θ(t), z(t))) = (θ(t−1), z(t−1)),

end ifend for



Alphabet soup

Why does it work?

Acceptance probability that does not involve the calculation of thelikelihood and

πǫ(θ′, z′|y)

πǫ(θ(t−1), z(t−1)|y) ×Kω(θ

(t−1)|θ′)f(z(t−1)|θ(t−1))

Kω(θ′|θ(t−1))f(z′|θ′)

=π(θ′) f(z′|θ′) IAǫ,y

(z′)

π(θ(t−1)) f(z(t−1)|θ(t−1))IAǫ,y(z(t−1))

× Kω(θ(t−1)|θ′) f(z(t−1)|θ(t−1))

Kω(θ′|θ(t−1)) f(z′|θ′)

=π(θ′)Kω(θ

(t−1)|θ′)π(θ(t−1)Kω(θ′|θ(t−1))

IAǫ,y(z′) .



Alphabet soup

ABCµ

[Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]

Use of a joint density

f(θ, ǫ|y) ∝ ξ(ǫ|y, θ)× πθ(θ)× πǫ(ǫ)

where y is the data, and ξ(ǫ|y, θ) is the prior predictive density ofρ(η(z), η(y)) given θ and x when z ∼ f(z|θ)



Alphabet soup

ABCµ

[Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]

Use of a joint density

f(θ, ǫ|y) ∝ ξ(ǫ|y, θ)× πθ(θ)× πǫ(ǫ)

where y is the data, and ξ(ǫ|y, θ) is the prior predictive density ofρ(η(z), η(y)) given θ and x when z ∼ f(z|θ)Warning! Replacement of ξ(ǫ|y, θ) with a non-parametric kernelapproximation.



Alphabet soup

ABCµ details

Multidimensional distances ρk (k = 1, . . . ,K) and errorsǫk = ρk(ηk(z), ηk(y)), with

ǫk ∼ ξk(ǫ|y, θ) ≈ ξk(ǫ|y, θ) =1

Bhk

∑

b

K[{ǫk−ρk(ηk(zb), ηk(y))}/hk]

then used in replacing ξ(ǫ|y, θ) with mink ξk(ǫ|y, θ)



Alphabet soup

ABCµ details

Multidimensional distances ρk (k = 1, . . . ,K) and errorsǫk = ρk(ηk(z), ηk(y)), with

ǫk ∼ ξk(ǫ|y, θ) ≈ ξk(ǫ|y, θ) =1

Bhk

∑

b

K[{ǫk−ρk(ηk(zb), ηk(y))}/hk]

then used in replacing ξ(ǫ|y, θ) with mink ξk(ǫ|y, θ)ABCµ involves acceptance probability

π(θ′, ǫ′)

π(θ, ǫ)

q(θ′, θ)q(ǫ′, ǫ)

q(θ, θ′)q(ǫ, ǫ′)

mink ξk(ǫ′|y, θ′)

mink ξk(ǫ|y, θ)



Alphabet soup

ABCµ multiple errors

[ c© Ratmann et al., PNAS, 2009]



Alphabet soup

ABCµ for model choice

[ c© Ratmann et al., PNAS, 2009]



Alphabet soup

Questions about ABCµ

For each model under comparison, marginal posterior on ǫ used toassess the fit of the model (HPD includes 0 or not).



Alphabet soup

Questions about ABCµ

For each model under comparison, marginal posterior on ǫ used toassess the fit of the model (HPD includes 0 or not).

◮ Is the data informative about ǫ? [Identifiability]

◮ How is the prior π(ǫ) impacting the comparison?

◮ How is using both ξ(ǫ|x0, θ) and πǫ(ǫ) compatible with astandard probability model? [remindful of Wilkinson]

◮ Where is the penalisation for complexity in the modelcomparison?

[X, Mengersen & Chen, 2010, PNAS]



Alphabet soup

A PMC version

Generate a sample at iteration t by

πt(θ(t)) ∝

N∑

j=1

ω(t−1)j Kt(θ

(t)|θ(t−1)j )

modulo acceptance of the associated xt, and use an importance

weight associated with an accepted simulation θ(t)i

ω(t)i ∝ π(θ

(t)i )/πt(θ

(t)i ) .

c© Still likelihood free[Beaumont et al., 2009]



Alphabet soup

The ABC-PMC algorithm

Given a decreasing sequence of approximation levels ǫ1 ≥ . . . ≥ ǫT ,

1. At iteration t = 1,

For i = 1, ..., N

Simulate θ(1)i ∼ π(θ) and x ∼ f(x|θ(1)i ) until (x, y) < ǫ1

Set ω(1)i = 1/N

Take τ2 as twice the empirical variance of the θ(1)i ’s

2. At iteration 2 ≤ t ≤ T ,

For i = 1, ..., N , repeat

Pick θ⋆i from the θ(t−1)j ’s with probabilities ω

(t−1)j

generate θ(t)i |θ⋆i ∼ N (θ⋆i , σ

2t ) and x ∼ f(x|θ(t)i )

until (x, y) < ǫt

Set ω(t)i ∝ π(θ

(t)i )/

∑N

j=1 ω(t−1)j ϕ

(σ−1t

{θ(t)i − θ

(t−1)j )

})

Take τ2t+1 as twice the weighted empirical variance of the θ(t)i ’s



Alphabet soup

Sequential Monte Carlo

SMC is a simulation technique to approximate a sequence ofrelated probability distributions πn with π0 “easy” and πT astarget.Iterated IS: particles moved from time n to time n via kernel Kn

and use of a sequence of extended targets πn

πn(z0:n) = πn(zn)n∏

j=0

Lj(zj+1, zj)

where the Lj ’s are backward Markov kernels [check that πn(zn) isa marginal]

[Del Moral, Doucet & Jasra, Series B, 2006]



Alphabet soup

Sequential Monte Carlo (2)

Algorithm 4 SMC sampler

sample z(0)i ∼ γ0(x) (i = 1, . . . , N)

compute weights w(0)i = π0(z

(0)i ))/γ0(z

(0)i )

for t = 1 to N doif ESS(w(t−1)) < NT thenresample N particles z(t−1) and set weights to 1

end ifgenerate z

(t−1)i ∼ Kt(z

(t−1)i , ·) and set weights to

w(t)i = W

(t−1)i−1

πt(z(t)i ))Lt−1(z

(t)i ), z

(t−1)i ))

πt−1(z(t−1)i ))Kt(z

(t−1)i ), z

(t)i ))

end for

[Del Moral, Doucet & Jasra, Series B, 2006]



Alphabet soup

ABC-SMC

[Del Moral, Doucet & Jasra, 2009]

True derivation of an SMC-ABC algorithmUse of a kernel Kn associated with target πǫn and derivation of thebackward kernel

Ln−1(z, z′) =

πǫn(z′)Kn(z

′, z)

πn(z)

Update of the weights

win ∝ wi(n−1)

∑Mm=1 IAǫn

(xmin)∑Mm=1 IAǫn−1

(xmi(n−1))

when xmin ∼ K(xi(n−1), ·)



Alphabet soup

ABC-SMCM

Modification: Makes M repeated simulations of the pseudo-data z

given the parameter, rather than using a single [M = 1]simulation, leading to weight that is proportional to the number ofaccepted zis

ω(θ) =1

M

M∑

i=1

Iρ(η(y),η(zi))<ǫ

[limit in M means exact simulation from (tempered) target]



Alphabet soup

Properties of ABC-SMC

The ABC-SMC method properly uses a backward kernel L(z, z′) tosimplify the importance weight and to remove the dependence onthe unknown likelihood from this weight. Update of importanceweights is reduced to the ratio of the proportions of survivingparticles



Alphabet soup

Properties of ABC-SMC

The ABC-SMC method properly uses a backward kernel L(z, z′) tosimplify the importance weight and to remove the dependence onthe unknown likelihood from this weight. Update of importanceweights is reduced to the ratio of the proportions of survivingparticlesAdaptivity in ABC-SMC algorithm only found in on-lineconstruction of the thresholds ǫt, slowly enough to keep a largenumber of accepted transitions



Alphabet soup

Semi-automatic ABC

Fearnhead and Prangle (2010) study ABC and the selection of thesummary statistic in close proximity to Wilkinson’s proposal

ABC then considered from a purely inferential viewpoint andcalibrated for estimation purposes.Use of a randomised (or ‘noisy’) version of the summary statistics

η(y) = η(y) + τǫ

Derivation of a well-calibrated version of ABC, i.e. an algorithmthat gives proper predictions for the distribution associated withthis randomised summary statistic.



Alphabet soup

Summary statistics

Optimality of the posterior expectations of the parameters ofinterest as summary statistics!



Alphabet soup

Summary statistics

Optimality of the posterior expectations of the parameters ofinterest as summary statistics!Use of the standard quadratic loss function

(θ − θ0)TA(θ − θ0) .



Calibration of ABC

Which summary?

Fundamental difficulty of the choice of the summary statistic whenthere is no non-trivial sufficient statistics [except when done by theexperimenters in the field]



Calibration of ABC

Which summary?

Fundamental difficulty of the choice of the summary statistic whenthere is no non-trivial sufficient statistics [except when done by theexperimenters in the field]Starting from a large collection of summary statistics is available,Joyce and Marjoram (2008) consider the sequential inclusion intothe ABC target, with a stopping rule based on a likelihood ratiotest.



Calibration of ABC

Which summary?

Fundamental difficulty of the choice of the summary statistic whenthere is no non-trivial sufficient statistics [except when done by theexperimenters in the field]Starting from a large collection of summary statistics is available,Joyce and Marjoram (2008) consider the sequential inclusion intothe ABC target, with a stopping rule based on a likelihood ratiotest.

◮ Does not taking into account the sequential nature of the tests

◮ Depends on parameterisation

◮ Order of inclusion matters.






The Gibbs Sampler


ABC for model choiceModel choiceGibbs random fieldsModel choice via ABCIllustrationsGeneric ABC model choice



Model choice

Bayesian model choice

Several models M1,M2, . . . are considered simultaneously for adataset y and the model index M is part of the inference.Use of a prior distribution. π(M = m), plus a prior distribution onthe parameter conditional on the value m of the model index,πm(θm)Goal is to derive the posterior distribution of M , challengingcomputational target when models are complex.



Model choice

Generic ABC for model choice

Algorithm 5 Likelihood-free model choice sampler (ABC-MC)

for t = 1 to T dorepeatGenerate m from the prior π(M = m)Generate θm from the prior πm(θm)Generate z from the model fm(z|θm)

until ρ{η(z), η(y)} < ǫSet m(t) = m and θ

(t) = θm

end for



Model choice

ABC estimates

Posterior probability π(M = m|y) approximated by the frequencyof acceptances from model m

1

T

T∑

t=1

Im(t)=m .

Issues with implementation:

◮ should tolerances ǫ be the same for all models?

◮ should summary statistics vary across models (incl. theirdimension)?

◮ should the distance measure ρ vary as well?



Model choice

ABC estimates

Posterior probability π(M = m|y) approximated by the frequencyof acceptances from model m

1

T

T∑

t=1

Im(t)=m .

Issues with implementation:

◮ should tolerances ǫ be the same for all models?

◮ should summary statistics vary across models (incl. theirdimension)?

◮ should the distance measure ρ vary as well?

Extension to a weighted polychotomous logistic regression estimateof π(M = m|y), with non-parametric kernel weights

[Cornuet et al., DIYABC, 2009]



Model choice

The Great ABC controversy

On-going controvery in phylogeographic genetics about the validityof using ABC for testing

Against: Templeton, 2008,2009, 2010a, 2010b, 2010cargues that nested hypothesescannot have higher probabilitiesthan nesting hypotheses (!)



Model choice

The Great ABC controversy

On-going controvery in phylogeographic genetics about the validityof using ABC for testing

Against: Templeton, 2008,2009, 2010a, 2010b, 2010cargues that nested hypothesescannot have higher probabilitiesthan nesting hypotheses (!)

Replies: Fagundes et al., 2008,Beaumont et al., 2010, Berger etal., 2010, Csillery et al., 2010point out that the criticisms areaddressed at [Bayesian]model-based inference and havenothing to do with ABC...



Gibbs random fields

Gibbs random fields

Gibbs distribution

The rv y = (y1, . . . , yn) is a Gibbs random field associated withthe graph G if

f(y) =1

Zexp

{−∑

c∈C

Vc(yc)

},

where Z is the normalising constant, C is the set of cliques of Gand Vc is any function also called potentialU(y) =

∑c∈C

Vc(yc) is the energy function



Gibbs random fields

Gibbs random fields

Gibbs distribution

The rv y = (y1, . . . , yn) is a Gibbs random field associated withthe graph G if

f(y) =1

Zexp

{−∑

c∈C

Vc(yc)

},

where Z is the normalising constant, C is the set of cliques of Gand Vc is any function also called potentialU(y) =

∑c∈C

Vc(yc) is the energy function

c© Z is usually unavailable in closed form



Gibbs random fields

Potts model

Potts model

Vc(y) is of the form

Vc(y) = θS(y) = θ∑

l∼i

δyl=yi

where l∼i denotes a neighbourhood structure



Gibbs random fields

Potts model

Potts model

Vc(y) is of the form

Vc(y) = θS(y) = θ∑

l∼i

δyl=yi

where l∼i denotes a neighbourhood structure

In most realistic settings, summation

Zθ =∑

x∈X

exp{θTS(x)}

involves too many terms to be manageable and numericalapproximations cannot always be trusted

[Cucala, Marin, CPR & Titterington, 2009]



Model choice via ABC

Bayesian Model Choice

Comparing a model with potential S0 taking values in Rp0 versus a

model with potential S1 taking values in Rp1 can be done through

the Bayes factor corresponding to the priors π0 and π1 on eachparameter space

Bm0/m1(x) =

∫exp{θT

0 S0(x)}/Zθ0,0π0(dθ0)∫

exp{θT1 S1(x)}/Zθ1,1

π1(dθ1)




Bayesian Model Choice

Comparing a model with potential S0 taking values in Rp0 versus a

model with potential S1 taking values in Rp1 can be done through

the Bayes factor corresponding to the priors π0 and π1 on eachparameter space

Bm0/m1(x) =

∫exp{θT

0 S0(x)}/Zθ0,0π0(dθ0)∫

exp{θT1 S1(x)}/Zθ1,1

π1(dθ1)

Use of Jeffreys’ scale to select most appropriate model




Neighbourhood relations

Choice to be made between M neighbourhood relations

im∼ i′ (0 ≤ m ≤ M − 1)

withSm(x) =

∑

im∼i′

I{xi=xi′}

driven by the posterior probabilities of the models.




Model index

Formalisation via a model index M that appears as a newparameter with prior distribution π(M = m) andπ(θ|M = m) = πm(θm)




Model index

Formalisation via a model index M that appears as a newparameter with prior distribution π(M = m) andπ(θ|M = m) = πm(θm)Computational target:

P(M = m|x) ∝∫

Θm

fm(x|θm)πm(θm) dθm π(M = m) ,




Sufficient statistics

By definition, if S(x) sufficient statistic for the joint parameters(M, θ0, . . . , θM−1),

P(M = m|x) = P(M = m|S(x)) .






P(M = m|x) = P(M = m|S(x)) .For each model m, own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) also sufficient.






P(M = m|x) = P(M = m|S(x)) .For each model m, own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) also sufficient.For Gibbs random fields,

x|M = m ∼ fm(x|θm) = f1m(x|S(x))f2

m(S(x)|θm)

=1

n(S(x))f2m(S(x)|θm)

wheren(S(x)) = ♯ {x ∈ X : S(x) = S(x)}

c© S(x) is therefore also sufficient for the joint parameters[Specific to Gibbs random fields!]




ABC model choice Algorithm

ABC-MC◮ Generate m∗ from the prior π(M = m).

◮ Generate θ∗m∗ from the prior πm∗(·).◮ Generate x∗ from the model fm∗(·|θ∗m∗).

◮ Compute the distance ρ(S(x0), S(x∗)).

◮ Accept (θ∗m∗ ,m∗) if ρ(S(x0), S(x∗)) < ǫ.

Note When ǫ = 0 the algorithm is exact




ABC approximation to the Bayes factor

Frequency ratio:

BFm0/m1(x0) =

P(M = m0|x0)

P(M = m1|x0)× π(M = m1)

π(M = m0)

=♯{mi∗ = m0}♯{mi∗ = m1}

× π(M = m1)

π(M = m0),




ABC approximation to the Bayes factor

Frequency ratio:

BFm0/m1(x0) =

P(M = m0|x0)

P(M = m1|x0)× π(M = m1)

π(M = m0)

=♯{mi∗ = m0}♯{mi∗ = m1}

× π(M = m1)

π(M = m0),

replaced with

BFm0/m1(x0) =

1 + ♯{mi∗ = m0}1 + ♯{mi∗ = m1}

× π(M = m1)

π(M = m0)

to avoid indeterminacy (also Bayes estimate).



Illustrations

Toy example

iid Bernoulli model versus two-state first-order Markov chain, i.e.

f0(x|θ0) = exp

(θ0

n∑

i=1

I{xi=1}

)/{1 + exp(θ0)}n ,

versus

f1(x|θ1) =1

2exp

(θ1

n∑

i=2

I{xi=xi−1}

)/{1 + exp(θ1)}n−1 ,

with priors θ0 ∼ U(−5, 5) and θ1 ∼ U(0, 6) (inspired by “phasetransition” boundaries).



Illustrations

Toy example (2)

(left) Comparison of the true BFm0/m1(x0) with BFm0/m1

(x0)(in logs) over 2, 000 simulations and 4.106 proposals from theprior. (right) Same when using tolerance ǫ corresponding to the1% quantile on the distances.



Generic ABC model choice

Back to sufficiency

If η1(x) sufficient statistic for model m = 1 and parameter θ1 andη2(x) sufficient statistic for model m = 2 and parameter θ2,(η1(x), η2(x)) is not always sufficient for (m, θm)




Back to sufficiency

If η1(x) sufficient statistic for model m = 1 and parameter θ1 andη2(x) sufficient statistic for model m = 2 and parameter θ2,(η1(x), η2(x)) is not always sufficient for (m, θm)

c© Potential loss of information at the testing level[X, Cornuet, Marin, and Pillai, 2011]




Limiting behaviour of B12 (T → ∞)

ABC approximation

B12(y) =

∑Tt=1 Imt=1 Iρ{η(zt),η(y)}≤ǫ∑Tt=1 Imt=2 Iρ{η(zt),η(y)}≤ǫ

,

where the (mt, zt)’s are simulated from the (joint) prior




Limiting behaviour of B12 (T → ∞)

ABC approximation

B12(y) =

∑Tt=1 Imt=1 Iρ{η(zt),η(y)}≤ǫ∑Tt=1 Imt=2 Iρ{η(zt),η(y)}≤ǫ

,

where the (mt, zt)’s are simulated from the (joint) priorAs T go to infinity, limit

Bǫ12(y) =

∫Iρ{η(z),η(y)}≤ǫπ1(θ1)f1(z|θ1) dz dθ1∫Iρ{η(z),η(y)}≤ǫπ2(θ2)f2(z|θ2) dz dθ2

=

∫Iρ{η,η(y)}≤ǫπ1(θ1)f

η1 (η|θ1) dη dθ1∫

Iρ{η,η(y)}≤ǫπ2(θ2)fη2 (η|θ2) dη dθ2

,

where fη1 (η|θ1) and fη

2 (η|θ2) distributions of η(z)




Limiting behaviour of B12 (ǫ → 0)

When ǫ goes to zero,

Bη12(y) =

∫π1(θ1)f

η1 (η(y)|θ1) dθ1∫

π2(θ2)fη2 (η(y)|θ2) dθ2

,




Limiting behaviour of B12 (ǫ → 0)

When ǫ goes to zero,

Bη12(y) =

∫π1(θ1)f

η1 (η(y)|θ1) dθ1∫

π2(θ2)fη2 (η(y)|θ2) dθ2

,

Bayes factor based on the sole observation of η(y)




Limiting behaviour of B12 (under sufficiency)

If η(y) sufficient statistic for both models,

fi(y|θi) = gi(y)fηi (η(y)|θi)

Thus

B12(y) =

∫Θ1

π(θ1)g1(y)fη1 (η(y)|θ1) dθ1∫

Θ2π(θ2)g2(y)f

η2 (η(y)|θ2) dθ2

=g1(y)

∫π1(θ1)f

η1 (η(y)|θ1) dθ1

g2(y)∫π2(θ2)f

η2 (η(y)|θ2) dθ2

=g1(y)

g2(y)Bη

12(y) .

[Didelot, Everitt, Johansen & Lawson, 2011]




Limiting behaviour of B12 (under sufficiency)

If η(y) sufficient statistic for both models,

fi(y|θi) = gi(y)fηi (η(y)|θi)

Thus

B12(y) =

∫Θ1

π(θ1)g1(y)fη1 (η(y)|θ1) dθ1∫

Θ2π(θ2)g2(y)f

η2 (η(y)|θ2) dθ2

=g1(y)

∫π1(θ1)f

η1 (η(y)|θ1) dθ1

g2(y)∫π2(θ2)f

η2 (η(y)|θ2) dθ2

=g1(y)

g2(y)Bη

12(y) .

[Didelot, Everitt, Johansen & Lawson, 2011]

c© No discrepancy only when cross-model sufficiency




Poisson/geometric example

Samplex = (x1, . . . , xn)

from either a Poisson P(λ) or from a geometric G(p) Then

S =n∑

i=1

yi = η(x)

sufficient statistic for either model but not simultaneously

Discrepancy ratio

g1(x)

g2(x)=

S!n−S/∏

i yi!

1/(

n+S−1S

)




Poisson/geometric discrepancy

Range of B12(x) versus Bη12(x) B12(x): The values produced have

nothing in common.




Formal recovery

Creating an encompassing exponential family

f(x|θ1, θ2, α1, α2) ∝ exp{θT1 η1(x)+ θT1 η1(x)+α1t1(x)+α2t2(x)}

leads to a sufficient statistic (η1(x), η2(x), t1(x), t2(x))[Didelot, Everitt, Johansen & Lawson, 2011]




Formal recovery




In the Poisson/geometric case, if∏

i xi! is added to S, nodiscrepancy




Formal recovery




Only applies in genuine sufficiency settings...

c© Inability to evaluate loss brought by summary statistics




Meaning of the ABC-Bayes factor

In the Poisson/geometric case, if E[yi] = θ0 > 0,

limn→∞

Bη12(y) =

(θ0 + 1)2

θ0e−θ0




MA(q) divergence

Evolution [against ǫ] of ABC Bayes factor, in terms of frequencies ofvisits to models MA(1) (left) and MA(2) (right) when ǫ equal to10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sampleof 50 points from a MA(2) with θ1 = 0.6, θ2 = 0.2. True Bayes factorequal to 17.71.




MA(q) divergence

Evolution [against ǫ] of ABC Bayes factor, in terms of frequencies ofvisits to models MA(1) (left) and MA(2) (right) when ǫ equal to10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sampleof 50 points from a MA(1) model with θ1 = 0.6. True Bayes factor B21

equal to .004.




A population genetics evaluation

Population genetics example with

◮ 3 populations

◮ 2 scenari

◮ 15 individuals

◮ 5 loci

◮ single mutation parameter




A population genetics evaluation

Population genetics example with

◮ 3 populations

◮ 2 scenari

◮ 15 individuals

◮ 5 loci

◮ single mutation parameter

◮ 24 summary statistics

◮ 2 million ABC proposal

◮ importance [tree] sampling alternative




Stability of importance sampling




Comparison with ABC

Use of 24 summary statistics and DIY-ABC logistic correction




Comparison with ABC





Comparison with ABC





The only safe cases

Besides specific models like Gibbs random fields,

using distances over the data itself escapes the discrepancy...[Toni & Stumpf, 2010; Sousa et al., 2009]




The only safe cases

Besides specific models like Gibbs random fields,

using distances over the data itself escapes the discrepancy...[Toni & Stumpf, 2010; Sousa et al., 2009]

...and so does the use of more informal model fitting measures[Ratmann et al., 2009, 2011]

Education

Shanghai tutorial