Columbia workshop [ABC model choice]

ABC Methods for Bayesian Model Choice


Christian P. Robert

Universite Paris-Dauphine, IuF, & CRESThttp://www.ceremade.dauphine.fr/~xian

Computational Methods in Applied Sciences, ColumbiaUniversity, New York, September 24, 2011

Joint work(s) with Jean-Marie Cornuet, Aude Grelaud,Jean-Michel Marin, Jean-Michel Marin, Natesh Pillai, Judith

Rousseau

http://www.ceremade.dauphine.fr/~xian


Approximate Bayesian computation



ABC for model choice

Gibbs random fields

Generic ABC model choice

Model choice consistency



Regular Bayesian computation issues

When faced with a non-standard posterior distribution

π(θ|y) ∝ π(θ)L(θ|y)

the standard solution is to use simulation (Monte Carlo) toproduce a sample

θ1, . . . , θT

from π(θ|y) (or approximately by Markov chain Monte Carlomethods)

[Robert & Casella, 2004]



Untractable likelihoods

Cases when the likelihood function f(y|θ) is unavailable and whenthe completion step

f(y|θ) =

∫Zf(y, z|θ) dz

is impossible or too costly because of the dimension of zc© MCMC cannot be implemented!



Untractable likelihoods

c© MCMC cannot be implemented!



The ABC method

Bayesian setting: target is π(θ)f(x|θ)

When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:

ABC algorithm

For an observation y ∼ f(y|θ), under the prior π(θ), keep jointlysimulating

θ′ ∼ π(θ) , z ∼ f(z|θ′) ,

until the auxiliary variable z is equal to the observed value, z = y.

[Tavare et al., 1997]



The ABC method

Bayesian setting: target is π(θ)f(x|θ)When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:

ABC algorithm


θ′ ∼ π(θ) , z ∼ f(z|θ′) ,





The ABC method

Bayesian setting: target is π(θ)f(x|θ)When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:

ABC algorithm


θ′ ∼ π(θ) , z ∼ f(z|θ′) ,





A as approximative

When y is a continuous random variable, equality z = y is replacedwith a tolerance condition,

%(y, z) ≤ ε

where % is a distance

Output distributed from

π(θ)Pθ{%(y, z) < ε} ∝ π(θ|%(y, z) < ε)



A as approximative

When y is a continuous random variable, equality z = y is replacedwith a tolerance condition,

%(y, z) ≤ ε

where % is a distanceOutput distributed from

π(θ)Pθ{%(y, z) < ε} ∝ π(θ|%(y, z) < ε)



ABC algorithm

Algorithm 1 Likelihood-free rejection sampler

for i = 1 to N dorepeat

generate θ′ from the prior distribution π(·)generate z from the likelihood f(·|θ′)

until ρ{η(z), η(y)} ≤ εset θi = θ′

end for

where η(y) defines a (maybe in-sufficient) statistic



Output

The likelihood-free algorithm samples from the marginal in z of:

πε(θ, z|y) =π(θ)f(z|θ)IAε,y(z)∫

Aε,y×Θ π(θ)f(z|θ)dzdθ,

where Aε,y = {z ∈ D|ρ(η(z), η(y)) < ε}.

The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution:

πε(θ|y) =

∫πε(θ, z|y)dz ≈ π(θ|y) .



Output

The likelihood-free algorithm samples from the marginal in z of:

πε(θ, z|y) =π(θ)f(z|θ)IAε,y(z)∫

Aε,y×Θ π(θ)f(z|θ)dzdθ,

where Aε,y = {z ∈ D|ρ(η(z), η(y)) < ε}.

The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution:

πε(θ|y) =

∫πε(θ, z|y)dz ≈ π(θ|y) .



MA example

Consider the MA(q) model

xt = εt +

q∑i=1

ϑiεt−i

Simple prior: uniform prior over the identifiability zone, e.g.triangle for MA(2)



MA example (2)

ABC algorithm thus made of

1. picking a new value (ϑ1, ϑ2) in the triangle

2. generating an iid sequence (εt)−q<t≤T

3. producing a simulated series (x′t)1≤t≤T

Distance: basic distance between the series

ρ((x′t)1≤t≤T , (xt)1≤t≤T ) =T∑t=1

(xt − x′t)2

or between summary statistics like the first q autocorrelations

τj =T∑

t=j+1

xtxt−j



MA example (2)

ABC algorithm thus made of

1. picking a new value (ϑ1, ϑ2) in the triangle

2. generating an iid sequence (εt)−q<t≤T

3. producing a simulated series (x′t)1≤t≤T

Distance: basic distance between the series

ρ((x′t)1≤t≤T , (xt)1≤t≤T ) =T∑t=1

(xt − x′t)2

or between summary statistics like the first q autocorrelations

τj =

T∑t=j+1

xtxt−j



Comparison of distance impact

Evaluation of the tolerance on the ABC sample against bothdistances (ε = 100%, 10%, 1%, 0.1%) for an MA(2) model




0.0 0.2 0.4 0.6 0.8

01

23

4

θ1

−2.0 −1.0 0.0 0.5 1.0 1.5

0.00.5

1.01.5

θ2





0.0 0.2 0.4 0.6 0.8

01

23

4

θ1

−2.0 −1.0 0.0 0.5 1.0 1.5

0.00.5

1.01.5

θ2







Gibbs random fields





Bayesian model choice

PrincipleSeveral models M1,M2, . . . are considered simultaneously fordataset y and model index M central to inference.Use of a prior π(M = m), plus a prior distribution on theparameter conditional on the value m of the model index, πm(θm)Goal is to derive the posterior distribution of M, a challengingcomputational target when models are complex.



Generic ABC for model choice

Algorithm 2 Likelihood-free model choice sampler (ABC-MC)

for t = 1 to T dorepeat

Generate m from the prior π(M = m)Generate θm from the prior πm(θm)Generate z from the model fm(z|θm)

until ρ{η(z), η(y)} < εSet m(t) = m and θ(t) = θm

end for

[Toni, Welch, Strelkowa, Ipsen & Stumpf, 2009]



ABC estimates

Posterior probability π(M = m|y) approximated by the frequencyof acceptances from model m

1

T

T∑t=1

Im(t)=m .

Early issues with implementation:

I should tolerances ε be the same for all models?

I should summary statistics vary across models? incl. theirdimension?

I should the distance measure ρ vary across models?

Extension to a weighted polychotomous logistic regression estimateof π(M = m|y), with non-parametric kernel weights

[Cornuet et al., DIYABC, 2009]



ABC estimates

Posterior probability π(M = m|y) approximated by the frequencyof acceptances from model m

1

T

T∑t=1

Im(t)=m .

Early issues with implementation:

I ε then needs to become part of the model

I Varying statistics incompatible with Bayesian model choiceproper

I ρ then part of the model

Extension to a weighted polychotomous logistic regression estimateof π(M = m|y), with non-parametric kernel weights

[Cornuet et al., DIYABC, 2009]



The great ABC controversy

On-going controvery in phylogeographic genetics about the validityof using ABC for testing

Against: Templeton, 2008,2009, 2010a, 2010b, 2010c, &tcargues that nested hypothesescannot have higher probabilitiesthan nesting hypotheses (!)



The great ABC controversy

On-going controvery in phylogeographic genetics about the validityof using ABC for testing

Against: Templeton, 2008,2009, 2010a, 2010b, 2010c, &tcargues that nested hypothesescannot have higher probabilitiesthan nesting hypotheses (!)

Replies: Fagundes et al., 2008,Beaumont et al., 2010, Berger etal., 2010, Csillery et al., 2010point out that the criticisms areaddressed at [Bayesian]model-based inference and havenothing to do with ABC...


Gibbs random fields

Potts model

Potts model∑c∈C Vc(y) is of the form∑

c∈C

Vc(y) = θS(y) = θ∑l∼i

δyl=yi

where l∼i denotes a neighbourhood structure

In most realistic settings, summation

Zθ =∑x∈X

exp{θTS(x)}

involves too many terms to be manageable and numericalapproximations cannot always be trusted

[Cucala et al., 2009]


Gibbs random fields

Potts model

Potts model∑c∈C Vc(y) is of the form∑

c∈C

Vc(y) = θS(y) = θ∑l∼i

δyl=yi

where l∼i denotes a neighbourhood structure

In most realistic settings, summation

Zθ =∑x∈X

exp{θTS(x)}

involves too many terms to be manageable and numericalapproximations cannot always be trusted

[Cucala et al., 2009]


Gibbs random fields

Neighbourhood relations

SetupChoice to be made between M neighbourhood relations

im∼ i′ (0 ≤ m ≤M − 1)

withSm(x) =

∑im∼i′

I{xi=xi′}

driven by the posterior probabilities of the models.


Gibbs random fields

Model index

Computational target:

P(M = m|x) ∝∫

Θm

fm(x|θm)πm(θm) dθm π(M = m)

If S(x) sufficient statistic for the joint parameters(M, θ0, . . . , θM−1),

P(M = m|x) = P(M = m|S(x)) .


Gibbs random fields

Model index

Computational target:

P(M = m|x) ∝∫

Θm

fm(x|θm)πm(θm) dθm π(M = m)

If S(x) sufficient statistic for the joint parameters(M, θ0, . . . , θM−1),

P(M = m|x) = P(M = m|S(x)) .


Gibbs random fields

Sufficient statistics in Gibbs random fields

Each model m has its own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) is also (model-)sufficient.Explanation: For Gibbs random fields,

x|M = m ∼ fm(x|θm) = f1m(x|S(x))f2

m(S(x)|θm)

=1

n(S(x))f2m(S(x)|θm)

wheren(S(x)) = ] {x ∈ X : S(x) = S(x)}

c© S(x) is therefore also sufficient for the joint parameters


Gibbs random fields


Each model m has its own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) is also (model-)sufficient.

Explanation: For Gibbs random fields,


m(S(x)|θm)

=1


wheren(S(x)) = ] {x ∈ X : S(x) = S(x)}



Gibbs random fields


Each model m has its own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) is also (model-)sufficient.Explanation: For Gibbs random fields,


m(S(x)|θm)

=1


wheren(S(x)) = ] {x ∈ X : S(x) = S(x)}



Gibbs random fields

Toy example

iid Bernoulli model versus two-state first-order Markov chain, i.e.

f0(x|θ0) = exp

(θ0

n∑i=1

I{xi=1}

)/{1 + exp(θ0)}n ,

versus

f1(x|θ1) =1

2exp

(θ1

n∑i=2

I{xi=xi−1}

)/{1 + exp(θ1)}n−1 ,

with priors θ0 ∼ U(−5, 5) and θ1 ∼ U(0, 6) (inspired by “phasetransition” boundaries).


Gibbs random fields

Toy example (2)

−40−20

010

−5 0 5

BF01

BF01

−40−20

010

−10 −5 0 5 10

BF01

BF01

(left) Comparison of the true BFm0/m1(x0) with BFm0/m1

(x0)(in logs) over 2, 000 simulations and 4.106 proposals from theprior. (right) Same when using tolerance ε corresponding to the1% quantile on the distances.



Back to sufficiency

‘Sufficient statistics for individual models are unlikely tobe very informative for the model probability. This isalready well known and understood by the ABC-usercommunity.’

[Scott Sisson, Jan. 31, 2011, ’Og]

If η1(x) sufficient statistic for model m = 1 and parameter θ1 andη2(x) sufficient statistic for model m = 2 and parameter θ2,(η1(x), η2(x)) is not always sufficient for (m, θm)

c© Potential loss of information at the testing level



Back to sufficiency







Back to sufficiency







Limiting behaviour of B12 (T →∞)

ABC approximation

B12(y) =

∑Tt=1 Imt=1 Iρ{η(zt),η(y)}≤ε∑Tt=1 Imt=2 Iρ{η(zt),η(y)}≤ε

,

where the (mt, zt)’s are simulated from the (joint) prior

As T go to infinity, limit

Bε12(y) =

∫Iρ{η(z),η(y)}≤επ1(θ1)f1(z|θ1) dz dθ1∫Iρ{η(z),η(y)}≤επ2(θ2)f2(z|θ2) dz dθ2

=

∫Iρ{η,η(y)}≤επ1(θ1)fη1 (η|θ1) dη dθ1∫Iρ{η,η(y)}≤επ2(θ2)fη2 (η|θ2) dη dθ2

,

where fη1 (η|θ1) and fη2 (η|θ2) distributions of η(z)



Limiting behaviour of B12 (T →∞)

ABC approximation

B12(y) =

∑Tt=1 Imt=1 Iρ{η(zt),η(y)}≤ε∑Tt=1 Imt=2 Iρ{η(zt),η(y)}≤ε

,

where the (mt, zt)’s are simulated from the (joint) priorAs T go to infinity, limit

Bε12(y) =

∫Iρ{η(z),η(y)}≤επ1(θ1)f1(z|θ1) dz dθ1∫Iρ{η(z),η(y)}≤επ2(θ2)f2(z|θ2) dz dθ2

=

∫Iρ{η,η(y)}≤επ1(θ1)fη1 (η|θ1) dη dθ1∫Iρ{η,η(y)}≤επ2(θ2)fη2 (η|θ2) dη dθ2

,

where fη1 (η|θ1) and fη2 (η|θ2) distributions of η(z)



Limiting behaviour of B12 (ε→ 0)

When ε goes to zero,

Bη12(y) =

∫π1(θ1)fη1 (η(y)|θ1) dθ1∫π2(θ2)fη2 (η(y)|θ2) dθ2

Bayes factor based on the sole observation of η(y)



Limiting behaviour of B12 (ε→ 0)

When ε goes to zero,

Bη12(y) =

∫π1(θ1)fη1 (η(y)|θ1) dθ1∫π2(θ2)fη2 (η(y)|θ2) dθ2

Bayes factor based on the sole observation of η(y)



Limiting behaviour of B12 (under sufficiency)

If η(y) sufficient statistic in both models,

fi(y|θi) = gi(y)fηi (η(y)|θi)

Thus

B12(y) =

∫Θ1π(θ1)g1(y)fη1 (η(y)|θ1) dθ1∫

Θ2π(θ2)g2(y)fη2 (η(y)|θ2) dθ2

=g1(y)

∫π1(θ1)fη1 (η(y)|θ1) dθ1

g2(y)∫π2(θ2)fη2 (η(y)|θ2) dθ2

=g1(y)

g2(y)Bη

12(y) .

[Didelot, Everitt, Johansen & Lawson, 2011]

c© No discrepancy only when cross-model sufficiency



Limiting behaviour of B12 (under sufficiency)

If η(y) sufficient statistic in both models,

fi(y|θi) = gi(y)fηi (η(y)|θi)

Thus

B12(y) =

∫Θ1π(θ1)g1(y)fη1 (η(y)|θ1) dθ1∫

Θ2π(θ2)g2(y)fη2 (η(y)|θ2) dθ2

=g1(y)

∫π1(θ1)fη1 (η(y)|θ1) dθ1

g2(y)∫π2(θ2)fη2 (η(y)|θ2) dθ2

=g1(y)

g2(y)Bη

12(y) .

[Didelot, Everitt, Johansen & Lawson, 2011]

c© No discrepancy only when cross-model sufficiency



Poisson/geometric example

Samplex = (x1, . . . , xn)

from either a Poisson P(λ) or from a geometric G(p)Sum

S =

n∑i=1

xi = η(x)

sufficient statistic for either model but not simultaneously

Discrepancy ratio

g1(x)

g2(x)=S!n−S/

∏i xi!

1/(

n+S−1S

)



Poisson/geometric discrepancy

Range of B12(x) versus Bη12(x): The values produced have

nothing in common.



Formal recovery

Creating an encompassing exponential family

f(x|θ1, θ2, α1, α2) ∝ exp{θT1 η1(x) + θT

1 η1(x) +α1t1(x) +α2t2(x)}

leads to a sufficient statistic (η1(x), η2(x), t1(x), t2(x))[Didelot, Everitt, Johansen & Lawson, 2011]



Formal recovery



1 η1(x) +α1t1(x) +α2t2(x)}


In the Poisson/geometric case, if∏i xi! is added to S, no

discrepancy



Formal recovery



1 η1(x) +α1t1(x) +α2t2(x)}


Only applies in genuine sufficiency settings...

c© Inability to evaluate loss brought by summary statistics



The Pitman–Koopman lemma

Efficient sufficiency is not such a common occurrence:

Lemma

A necessary and sufficient condition for the existence of a sufficientstatistic with fixed dimension whatever the sample size is that thesampling distribution belongs to an exponential family.

[Pitman, 1933; Koopman, 1933]

Provision of fixed support (consider U(0, θ) counterexample)



The Pitman–Koopman lemma

Efficient sufficiency is not such a common occurrence:

Lemma

A necessary and sufficient condition for the existence of a sufficientstatistic with fixed dimension whatever the sample size is that thesampling distribution belongs to an exponential family.

[Pitman, 1933; Koopman, 1933]

Provision of fixed support (consider U(0, θ) counterexample)



Meaning of the ABC-Bayes factor

‘This is also why focus on model discrimination typically(...) proceeds by (...) accepting that the Bayes Factorthat one obtains is only derived from the summarystatistics and may in no way correspond to that of thefull model.’


In the Poisson/geometric case, if E[yi] = θ0 > 0,

limn→∞

Bη12(y) =

(θ0 + 1)2

θ0e−θ0



Meaning of the ABC-Bayes factor

‘This is also why focus on model discrimination typically(...) proceeds by (...) accepting that the Bayes Factorthat one obtains is only derived from the summarystatistics and may in no way correspond to that of thefull model.’


In the Poisson/geometric case, if E[yi] = θ0 > 0,

limn→∞

Bη12(y) =

(θ0 + 1)2

θ0e−θ0



MA example

1 2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

1 2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

1 2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

1 2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Evolution [against ε] of ABC Bayes factor, in terms of frequencies ofvisits to models MA(1) (left) and MA(2) (right) when ε equal to10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sampleof 50 points from a MA(2) with θ1 = 0.6, θ2 = 0.2. True Bayes factorequal to 17.71.



MA example

1 2

0.0

0.2

0.4

0.6

1 2

0.0

0.2

0.4

0.6

1 2

0.0

0.2

0.4

0.6

1 2

0.0

0.2

0.4

0.6

0.8

Evolution [against ε] of ABC Bayes factor, in terms of frequencies ofvisits to models MA(1) (left) and MA(2) (right) when ε equal to10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sampleof 50 points from a MA(1) model with θ1 = 0.6. True Bayes factor B21

equal to .004.



A population genetics evaluation

Population genetics example with

I 3 populations

I 2 scenari

I 15 individuals

I 5 loci

I single mutation parameter

I 24 summary statistics

I 2 million ABC proposal

I importance [tree] sampling alternative



A population genetics evaluation

Population genetics example with

I 3 populations

I 2 scenari

I 15 individuals

I 5 loci

I single mutation parameter

I 24 summary statistics

I 2 million ABC proposal

I importance [tree] sampling alternative



Stability of importance sampling

●

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

●

0.0

0.2

0.4

0.6

0.8

1.0

●

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0



Comparison with ABC

Use of 24 summary statistics and DIY-ABC logistic correction

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

importance sampling

AB

C d

irect

and

logi

stic

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●● ●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●



Comparison with ABC


●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

● ●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

● ●

●

●

●●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

−4 −2 0 2 4 6

−4

−2

02

46

importance sampling

AB

C d

irect



Comparison with ABC


●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

● ●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

● ●

●

●

●●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

−4 −2 0 2 4 6

−4

−2

02

46

importance sampling

AB

C d

irect

and

logi

stic

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

● ●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●



Comparison with ABC


●●

●●

●●●

●

●●●●●

●●●

●●

●●

●

●●

●●

●●●

●

●

●

●●●●

●

●●

●●●●

●

●

●

●

●

●●●●

●●

●●●●●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

0 20 40 60 80 100

−2

−1

01

2

index

log−

ratio

●

●

●

●●●●

●●●●●●

●

●●

●●

●

●

●

●

●

●

●

●●●

●

●

●●●●●

●

●●●●●

●●●●

●

●

●●

●

●

●

●

●●●●

●

●

●●●●

●

●

●

●

●●

●●

●

●●

●

●

●●●●●

●

●

●

●

●

●●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●●●●

●●●●●●

●

●●

●●

●

●

●

●

●

●

●

●●●

●

●

●●●●●

●

●●●●●

●●●●

●

●

●●

●

●

●

●

●●●●

●

●

●●●●

●

●

●

●

●●

●●

●

●●

●

●

●●●●●

●

●

●

●

●

●●

●

●

●●●●

●

●

●

●

●

●●●

●●

●●●

●

●●●●●

●●●

●●

●●

●

●●

●●

●●●

●

●

●

●●●●

●

●●

●●●●

●

●

●

●

●

●●●●

●●

●●●●●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●



A second population genetics experiment

I three populations, two divergent 100 gen. ago

I two scenarios [third pop. recent admixture between first twopop. / diverging from pop. 1 5 gen. ago]

I In scenario 1, admixture rate 0.7 from pop. 1

I 100 datasets with 100 diploid individuals per population, 50independent microsatellite loci.

I Effective population size of 1000 and mutation rates of0.0005.

I 6 parameters: admixture rate (U [0.1, 0.9]), three effectivepopulation sizes (U [200, 2000]), the time of admixture/seconddivergence (U [1, 10]) and time of first divergence (U [50, 500]).



Results

IS algorithm performed with 100 coalescent trees per particle and50,000 particles [12 calendar days using 376 processors]Using ten times as many loci and seven times as many individualsdegrades the confidence in the importance sampling approximationbecause of an increased variability in the likelihood.

●●

●

●●●●●

●

●● ● ●●

●

●

●

●●●●●●

●

●

●

●

●

●●●●● ● ●●●

●●

●

●●●●●

●

●●●●●● ●

●● ●●●●●●● ●●

●

●●●●●●●●●●●●●● ●●●●●●●●●●●●

●●●●●

●

●

●●

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Importance Sampling estimates ofP(M=1|y) with 50,000 particles

Impo

rtan

ce S

ampl

ing

estim

ates

of

P(M

=1|

y) w

ith 1

,000

par

ticle

s



Results

Blurs potential divergence between ABC and genuine posteriorprobabilities because both are overwhelmingly close to one, due tothe high information content of the data.

● ●

●

●●●●●

●

●●● ●●

●

●

●

●●●● ● ●

●

●

●●

●

● ●● ●● ●●●● ●●

●

● ●●●●

●

●●●

●●●●●●●●●●●●●

●●

●

●●●●●●●●●●● ●●● ●●● ●●●●●●●● ●●●● ●●

●

●

●●

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

ABC estimates of P(M=1|y)

Impo

rtan

ce S

ampl

ing

estim

ates

of P

(M=

1|y)

with

50,

000

part

icle

s



The only safe cases???

Besides specific models like Gibbs random fields,

using distances over the data itself escapes the discrepancy...[Toni & Stumpf, 2010;Sousa et al., 2009]

...and so does the use of more informal model fitting measures[Ratmann, Andrieu, Richardson and Wiujf, 2009]



The only safe cases???

Besides specific models like Gibbs random fields,

using distances over the data itself escapes the discrepancy...[Toni & Stumpf, 2010;Sousa et al., 2009]

...and so does the use of more informal model fitting measures[Ratmann, Andrieu, Richardson and Wiujf, 2009]



ABC model choice consistency



Gibbs random fields





The starting point

Central question to the validation of ABC for model choice:

When is a Bayes factor based on an insufficient statisticT (y) consistent?

Note: conclusion drawn on T (y) through BT12(y) necessarily differs

from the conclusion drawn on y through B12(y)



The starting point

Central question to the validation of ABC for model choice:

When is a Bayes factor based on an insufficient statisticT (y) consistent?

Note: conclusion drawn on T (y) through BT12(y) necessarily differs

from the conclusion drawn on y through B12(y)



A benchmark if toy example

Comparison suggested by referee of PNAS paper:[X, Cornuet, Marin, & Pillai, Aug. 2011]

Model M1: y ∼ N (θ1, 1) opposed to model M2:y ∼ L(θ2, 1/

√2), Laplace distribution with mean θ2 and scale

parameter 1/√

2 (variance one).







parameter 1/√

2 (variance one).Four possible statistics

1. sample mean y (sufficient for M1 but not M2);

2. sample median med(y) (insufficient);

3. sample variance var(y) (ancillary);

4. median absolute deviation mad(y) = med(y −med(y));







parameter 1/√

2 (variance one).

0.1 0.2 0.3 0.4 0.5 0.6 0.7

01

23

45

6

posterior probability

Den

sity







parameter 1/√

2 (variance one).

0.1 0.2 0.3 0.4 0.5 0.6 0.7

01

23

45

6

posterior probability

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

01

23

probability

Den

sity



Framework

Starting from sample y = (y1, . . . , yn) be the observed sample, notnecessarily iid with true distribution y ∼ PnSummary statistics T (y) = T n = (T1(y), T2(y), · · · , Td(y)) ∈ Rdwith true distribution T n ∼ Gn.



Framework

Comparison of

– under M1, y ∼ F1,n(·|θ1) where θ1 ∈ Θ1 ⊂ Rp1

– under M2, y ∼ F2,n(·|θ2) where θ2 ∈ Θ2 ⊂ Rp2

turned into

– under M1, T (y) ∼ G1,n(·|θ1), and θ1|T (y) ∼ π1(·|T n)

– under M2, T (y) ∼ G2,n(·|θ2), and θ2|T (y) ∼ π2(·|T n)



Assumptions

A collection of asymptotic “standard” assumptions:

[A1] There exist a sequence {vn} converging to +∞,an a.c. distribution Q with continuous bounded density q(·),a symmetric, d× d positive definite matrix V0and a vector µ0 ∈ Rd such that

vnV−1/20 (T n − µ0)

n→∞ Q, under Gn

and for all M > 0

supvn|t−µ0|<M

∣∣∣|V0|1/2v−dn gn(t)− q(vnV

−1/20 {t− µ0}

)∣∣∣ = o(1)



Assumptions


[A2] For i = 1, 2, there exist d× d symmetric positive definite matricesVi(θi) and µi(θi) ∈ Rd such that

vnVi(θi)−1/2(T n − µi(θi))

n→∞ Q, under Gi,n(·|θi) .



Assumptions


[A3] For i = 1, 2, there exist sets Fn,i ⊂ Θi and constants εi, τi, αi > 0such that for all τ > 0,

supθi∈Fn,i

Gi,n

[|T n − µ(θi)| > τ |µi(θi)− µ0| ∧ εi |θi

]. v−αi

n (|µi(θi)− µ0| ∧ εi)−αi

withπi(Fcn,i) = o(v−τin ).



Assumptions


[A4] For (u > 0)

Sn,i(u) ={θi ∈ Fn,i; |µ(θi)− µ0| ≤ u v−1n

}if inf{|µi(θi)− µ0|; θi ∈ Θi} = 0, there exist constants di < τi ∧ αi − 1such that

πi(Sn,i(u)) ∼ udiv−din , ∀u . vn



Assumptions


[A5] If inf{|µi(θi)− µ0|; θi ∈ Θi} = 0, there exists U > 0 such that forany M > 0,

supvn|t−µ0|<M

supθi∈Sn,i(U)

∣∣∣|Vi(θi)|1/2v−dn gi(t|θi)

−q(vnVi(θi)

−1/2(t− µ(θi))∣∣∣ = o(1)

and

limM→∞

lim supn

πi

(Sn,i(U) ∩

{||Vi(θi)−1||+ ||Vi(θi)|| > M

})πi(Sn,i(U))

= 0 .



Assumptions


[A1]–[A2] are standard central limit theorems[A3] controls the large deviations of the estimator T n from theestimand µ(θ)[A4] is the standard prior mass condition found in Bayesianasymptotics[A5] controls more tightly convergence esp. when µi is notone-to-one



Asymptotic marginals

Asymptotically, under [A1]–[A5]

mi(t) =

∫Θi

gi(t|θi)πi(θi) dθi

is such that(i) if inf{|µi(θi)− µ0|; θi ∈ Θi} = 0,

Clvd−din ≤ mi(T

n) ≤ Cuvd−din

and(ii) if inf{|µi(θi)− µ0|; θi ∈ Θi} > 0

mi(Tn) = oPn [vd−τin + vd−αin ].



Within-model consistency

Under same assumptions, if inf{|µi(θi)− µ0|; θi ∈ Θi} = 0, theposterior distribution of µi(θi) given T n is consistent at rate 1/vnprovided αi ∧ τi > di.

Note: di can be seen as an effective dimension of the model underthe posterior πi(.|T n), since if µ0 ∈ {µi(θi); θi ∈ Θi},

mi(Tn) ∼ vd−din and n(T n) ∼ vdn



Within-model consistency

Under same assumptions, if inf{|µi(θi)− µ0|; θi ∈ Θi} = 0, theposterior distribution of µi(θi) given T n is consistent at rate 1/vnprovided αi ∧ τi > di.

Note: di can be seen as an effective dimension of the model underthe posterior πi(.|T n), since if µ0 ∈ {µi(θi); θi ∈ Θi},

mi(Tn) ∼ vd−din and n(T n) ∼ vdn



Between-model consistency

Consequence of above is that asymptotic behaviour of the Bayesfactor is driven by the asymptotic mean value of T n under bothmodels. And only by this mean value!Indeed, if

inf{|µ0 − µ2(θ2)|; θ2 ∈ Θ2} = inf{|µ0 − µ1(θ1)|; θ1 ∈ Θ1} = 0

then

Clv−(d1−d2)n ≤ m1(T n)

m2(T n)≤ Cuv−(d1−d2)

n ,

where Cl, Cu = OPn(1), irrespective of the true model. Onlydepends on the difference d1 − d2



Between-model consistency

Consequence of above is that asymptotic behaviour of the Bayesfactor is driven by the asymptotic mean value of T n under bothmodels. And only by this mean value!Else, if

inf{|µ0 − µ2(θ2)|; θ2 ∈ Θ2} > inf{|µ0 − µ1(θ1)|; θ1 ∈ Θ1} = 0

thenm1(T n)

m2(T n)≥ Cu min

(v−(d1−α2)n , v−(d1−τ2)

n

),



Consistency theorem

If

{|µ0 − µ2(θ2)|; θ2 ∈ Θ2} = inf{|µ0 − µ1(θ1)|; θ1 ∈ Θ1} = 0,

Bayes factor BT12 is O(v

−(d1−d2)n ) irrespective of the true model. It

is consistent iff Pn is within the model with the smallest dimension

If Pn belongs to one of the two models and if µ0 cannot beattained by the other one :

0 = min (inf{|µ0 − µi(θi)|; θi ∈ Θi}, i = 1, 2)

< max (inf{|µ0 − µi(θi)|; θi ∈ Θi}, i = 1, 2) ,

then the Bayes factor BT12 is consistent



Consistency theorem

If

{|µ0 − µ2(θ2)|; θ2 ∈ Θ2} = inf{|µ0 − µ1(θ1)|; θ1 ∈ Θ1} = 0,

Bayes factor BT12 is O(v

−(d1−d2)n ) irrespective of the true model. It

is consistent iff Pn is within the model with the smallest dimensionIf Pn belongs to one of the two models and if µ0 cannot beattained by the other one :

0 = min (inf{|µ0 − µi(θi)|; θi ∈ Θi}, i = 1, 2)

< max (inf{|µ0 − µi(θi)|; θi ∈ Θi}, i = 1, 2) ,

then the Bayes factor BT12 is consistent



Consequences on summary statistics

Bayes factor driven by the means µi(θi) and the relative position ofµ0 wrt both sets {µi(θi); θi ∈ Θi}, i = 1, 2.For ABC, this implies the most likely statistics T n are ancillarystatistics with different mean values under both modelsElse, if T n asymptotically depends on some of the parameters ofthe models, it is quite likely that there exists θi ∈ Θi such thatµi(θi) = µ0 even though model M1 is misspecified




Bayes factor driven by the means µi(θi) and the relative position ofµ0 wrt both sets {µi(θi); θi ∈ Θi}, i = 1, 2.Toy example Laplace versus Gauss: If

T n = n−1n∑i=1

X4i

and the true distribution is Laplace with mean 0, so that µ0 = 6.Since under the Gaussian model

µ(θ) = 3 + θ4 + 6θ2

the value θ∗ = 2√

3− 3 leads to µ0 = µ(θ∗) and a Bayes factorassociated with such a statistic is not consistent (hered1 = d2 = d = 1).




Bayes factor driven by the means µi(θi) and the relative position ofµ0 wrt both sets {µi(θi); θi ∈ Θi}, i = 1, 2.

0.0 0.2 0.4 0.6 0.8 1.0

02

46

8

probability

Fourth moment




Bayes factor driven by the means µi(θi) and the relative position ofµ0 wrt both sets {µi(θi); θi ∈ Θi}, i = 1, 2.

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

6

probability

Fourth and sixth moments



Embedded models

When M1 submodel of M2, and if the true distribution belongs to

the smaller model M1, Bayes factor is of order v−(d1−d2)n .

If summary statistic only informative on a parameter that is thesame under both models, i.e if d1 = d2, then the Bayes factor isnot consistentElse, d1 < d2 and Bayes factor is consistent under M1. If truedistribution not in M1, then Bayes factor is consistent only ifµ1 6= µ2 = µ0

Education

Columbia workshop [ABC model choice]