QUOL: Qualitative, Objective Likelihoodism

Likelihoodism Objectivity Bayesian Objectivity QUOL: Qualitative, Objective Likelihoodism References Qualitative Conditional Independence

QUOL: Qualitative, Objective Likelihoodism

Conor Mayo-Wilson, Soham Pardeshi, and Aditya Saraf

University of Washington

FEWJune 30, 2020

1 / 65



Bernstein on Blood Types

Question: How do genes determine bloodtype?

H1 : Single-locus hypothesis.Three alleles A, B, and O determine type.

H2 : Two locus hypothesis.Two alleles A/a and B/b determine type.

2 / 65



Bernstein’s Data

H1 : Single Locus Hypothesis

Type Genotypes Observed % Expected % under H1

A AA, AO .422 .4112B BB, BO .206 .1943AB AB .078 .0911O OO .294 .3034

H2 : Two Locus Hypothesis

Type Genotypes Observed % Expected % under H2

A AAbb, Aabb .422 .358B aaBB, aaBb .206 .142AB AABB,AaBB,

AABb, AaBb.078 .142

O aabb .294 .358

Sample Size = 502

3 / 65



Bernstein and Law of Likelihood

According to Edwards, Bernstein’s data is about 400 million timesmore likely if H1 is true than if H2 is!

PH1(E ) ≈ 4 · 108 · PH2(E )

where E is Bernstein’s data.

4 / 65



Law of Likelihood

[Edwards, 1984] uses this example to motivate . . .

5 / 65



Law of Likelihood

Law of Likelihood (ll): E favors θ1 to θ2 if and only if

Pθ1(E ) > Pθ2(E )

Notes:

Above, θ1 and θ2 are simple (i.e., not composite) statisticalhypotheses.

As we use the term, ll is strictly weaker than the thesis that thelikelihood ratio is a numerical strength of evidence.

6 / 65



Likelihoodism: What is it?

Likelihoodism is often glossed as the thesis that “all theinformation that about an unknown parameter is contained in thelikelihood function.”

In practice, it’s taken to be the conjunction of ll and thelikelihood principle (lp), which is best illustrated by anuncontroversial consequence . . .

7 / 65



Conditionality Principle

Suppose Gertrude conducts experiment E and gets data E .

Allan can’t decide whether to conduct experiment E or F. He flips acoin and decides to conduct experiment E too. He gets the exactsame data E as Gertrude.

Conditionality Principle: Gertrude and Allan have equivalentevidence.

8 / 65



Conditionality Principle and LP

The conditionality principle follows from the likelihood principle . . .

9 / 65



Likelihood Principle

Experiment 1

Observe A Observe B

θ1 10% 90%θ2 20% 80%θ3 5% 95%

Experiment 2

Observe D . . .

θ1 20% . . .θ2 40% . . .θ3 10% . . .

Likelihood Principle (lp): For any two statistical experiments E and F and

any pair of observations E and F from E and F respectively, if there is some

constant c > 0 such that PEθ (E) = c · PF

θ (F ) for all simple hypotheses θ ∈ Θ,

then E and F are evidentially equivalent.

10 / 65




Experiment 1

Observe A Observe B

θ1 10% 90%θ2 20% 80%θ3 5% 95%

Experiment 2

Observe D . . .

θ1 20% . . .θ2 40% . . .θ3 10% . . .






10 / 65



LP entails Conditionality

Example:

Let F be the experiment in which one flips a coin of known biasc > 0 to decide whether to conduct E or E′.

Then the outcome F = 〈Heads,E 〉 of F is always c times theprobability of obtaining E if one had simply conducted theexperiment E.

So E and F = 〈Heads,E 〉 are evidentially equivalent by lp.

11 / 65



We have two overarching goals . . .

12 / 65



Favoring, Equivalence, and Objectivity

Goal 1: Clarify the importance of “favoring” and “evidentialequivalence”

If “favoring” is not pre-theoretic and does not tell us what tobelieve or how to act, why explicate it?

In what sense are ll and lp objective?

13 / 65



Likelihoodism: A complete theory

Goal 2: Provide a complete likelihoodist theory of evidence

1 Extend ll to composite hypotheses, including those with nuisanceparameters.

2 Characterize comparative favoring (i.e., when E favors H1 over H2

at least as much as F does) so that the analysis

1 Unifies ll and lp,2 Entails truisms about evidential strength that are not entailed by ll

and lp (e.g., larger iid samples are typically stronger evidence)

3 Motivate qualitative likelihoodist principles that generalize ll, lp,and the comparative principle that unifies them.

14 / 65











14 / 65











14 / 65











14 / 65



1 Likelihoodism

2 Objectivity

3 Bayesian ObjectivityKey Concept: Comparative Bayesian FavoringLaw of LikelihoodLikelihood Principle

4 QUOL: Qualitative, Objective LikelihoodismKey ConceptQualitative LLQualitative LP

15 / 65



Objectivity and Agreement

16 / 65



Likelihoodism and Objectivity

Question: Putting aside the question of what “favoring” and“evidentially equivalent” mean, why do authors often claim that lland lp provide objective standards of evidence?

Answer: ll and lp employ only likelihoods – probabilities of theform Pθ(E ) – to characterize “favoring” and “equivalence.” Okay,but why does that matter?

17 / 65




Question: Why does employing only likelihoods – probabilities ofthe form Pθ(E ) – make a principle more “objective”?

The Standard Answer:

Claim 1: Pθ describe frequencies or chances, i.e., “objective”features of the world that are independent of the experimenter.

E.g., In the blood-types example, Pθ describes frequencies of allelesin a population.

Claim 2: Posterior probabilities of the form P(θ|E ) describe aparticular agent’s credences, which are considered “subjective.”

18 / 65




Question: Why does employing only likelihoods – probabilities ofthe form Pθ(E ) – make a principle more “objective”?

The Standard Answer:

Claim 1: Pθ describe frequencies or chances, i.e., “objective”features of the world that are independent of the experimenter.

E.g., In the blood-types example, Pθ describes frequencies of allelesin a population.

Claim 2: Posterior probabilities of the form P(θ|E ) describe aparticular agent’s credences, which are considered “subjective.”

18 / 65



Clarifying Objectivity

Problem: We think the standard answer conflates two notions ofobjectivity.

1 Objective measures of evidence do not depend upon a singleagent’s credences (i.e., a particular credence function).

2 Objective measures of evidence depend only upon frequencies orchances.

The first claim is reasonable; we reject the second.

19 / 65



Clarifying Objectivity

Problem: We think the standard answer conflates two notions ofobjectivity.

1 Objective measures of evidence do not depend upon a singleagent’s credences (i.e., a particular credence function).

2 Objective measures of evidence depend only upon frequencies orchances.

The first claim is reasonable; we reject the second.

19 / 65




Our Thesis: ll and lp can be understood as answering questionsabout how all rational agents updates their credences.

In [Douglas, 2009]’s terminology, ll and lp characterizeconcordantly objective updating procedures.

20 / 65



The Key Concept: Comparative Bayesian Favoring

21 / 65



Posteriors and Likelihoods

Definition: Given a prior Q and likelihood functions Pθ(E )θ∈Θ,the posterior of Q(·|E ) is defined by

Q(H|E ) =Q(E |H) · Q(H)

Q(E )

=

∑θ∈H Pθ(E ) · Q(θ)∑θ∈Θ Pθ(E ) · Q(θ)

In other words, we assume agents agree to update using commonlikelihood functions.

Question: Where else will they agree?

22 / 65



Definition: Say that E comparatively Bayesian favors H1 to H2

more than F does if

Q(H1|E ∩ (H1 ∪ H2)) ≥ Q(H1|F ∩ (H1 ∪ H2))

for all priors such that that Q(H1),Q(H2) > 0.

Notation: Write E BH1H2

F in this case.

Idea: Learning E raises one’s confidence in H1 at least as much aslearning F no matter one’s initial degrees of belief.

23 / 65



Theorem

Suppose H1 and H2 are finite and disjoint. Let E and F beoutcomes of experiments E and F respectively. Then Ecomparatively Bayesian favors H1 to H2 at least as much as F iff

minθ∈H1 PEθ (E )

maxθ∈H2 PEθ (E )

≥maxθ∈H1 P

Fθ (F )

minθ∈H2 PFθ (F )

.

Corollary

Suppose H1 = θ1 and H2 = θ2 are simple hypotheses. Then Ecomparatively Bayesian favors H1 to H2 at least as much as F iff

PEθ1

(E )

PEθ2

(E )≥

PFθ2

(F )

PFθ1

(F ).

24 / 65



Theorem




≥maxθ∈H1 P

Fθ (F )


.

Corollary

Suppose H1 = θ1 and H2 = θ2 are simple hypotheses. Then Ecomparatively Bayesian favors H1 to H2 at least as much as F iff

PEθ1

(E )

PEθ2

(E )≥

PFθ2

(F )

PFθ1

(F ).

24 / 65



Application: Favoring Larger Samples

Let E be 75 heads in 100 tosses.

Let F be three heads in four tosses.

θ1 = The coin’s bias is 3/4.

θ2 = The coin’s bias is a specific value other than 3/4.

Then E comparatively favors θ1 to θ2 more than F does, but notvice versa.

25 / 65



LL and Agreement

26 / 65



Favoring

Definition: Say that E strictly Bayesian favors H1 to H2 if

Q(H1|E ∩ (H1 ∪ H2)) > Q(H1|(H1 ∪ H2))

for all priors such that that Q(H1),Q(H2) > 0.

Equivalently: E strictly favors H1 to H2 more than F = Ω, thesure event.

27 / 65



Theorem




≥maxθ∈H1 P

Fθ (F )


.

Corollary

Suppose H1 and H2 are finite. Then E strictly Bayesian favors H1

to H2 if and only if

minθ∈H1

PEθ (E ) > max

θ∈H2

PEθ (E ).

If H1 = θ1 and H2 = θ2 are simple, then E strictly Bayesianfavors H1 to H2 if and only if PE

θ1(E ) > PE

θ2(E ).

28 / 65



Theorem




≥maxθ∈H1 P

Fθ (F )


.

Corollary

Suppose H1 and H2 are finite. Then E strictly Bayesian favors H1

to H2 if and only if

minθ∈H1

PEθ (E ) > max

θ∈H2

PEθ (E ).

If H1 = θ1 and H2 = θ2 are simple, then E strictly Bayesianfavors H1 to H2 if and only if PE

θ1(E ) > PE

θ2(E ).

28 / 65



Moral: The law of likelihood answers the question: “When will allBayesians’ credences in H1 increase upon learning E , if theyassume either H1 or H2 is true?”

29 / 65



LP and Agreement

30 / 65




Experiment 1

Observe A Observe B

θ1 10% 90%θ2 20% 80%θ3 5% 95%

Experiment 2

Observe D . . .

θ1 20% . . .θ2 40% . . .θ3 10% . . .






31 / 65



Bayesian Favoring Equivalence

Definition: Say E and F are Bayesian-favoring equivalent ifE B

H1H2

F and F BH1H2

E for all disjoint hypotheses H1 and H2,i.e.,

Q(H1|E ∩ (H1 ∪ H2)) = Q(H1|F ∩ (H1 ∪ H2))

for all disjoint hypotheses H1 and H2 and for all priors Q such thatQ(H1),Q(H2) > 0.

32 / 65



Theorem

Let E and F be outcomes of experiments E and F respectively.Then the following are equivalent:

1 E and F are Bayesian favoring equivalent,

2 Q(H|E ) = Q(H|F ) for all hypotheses H and priors Q, and

3 lp entails E and F are evidentially equivalent, i.e., there is somec > 0 such that PE

θ (E ) = c · PFθ (F ) for all θ.

33 / 65



LP and Bayesian Evidential Equivalence

Morals:

1 lp characterizes precisely when every member of the hypotheticalcommunity of Bayesian scientists would agree two pieces of evidenceare equivalent.

2 The notions of “favoring” and “evidential equivalence” axiomatizedby ll and lp respectively can be interpreted as special cases of thenotion of comparative Bayesian-favoring.

34 / 65



Further Applications

Fans of Bayesianism:

1 The methodology suggests a way for finding other statisticalprinciples about favoring and equivalence.

E.g., When H1 and H2 are composite, the question of how to extendll is debated.E.g., When two experiments E and F have different parameterspaces, lp says nothing.

2 The “social” perspective (i.e., viewing statistical principles asimplicit agreements) tells us when we’ve found all the relevantstatistical principles for the Bayesian community.

35 / 65



Further Applications

Fans of Bayesianism:

1 The methodology suggests a way for finding other statisticalprinciples about favoring and equivalence.

E.g., When H1 and H2 are composite, the question of how to extendll is debated.E.g., When two experiments E and F have different parameterspaces, lp says nothing.

2 The “social” perspective (i.e., viewing statistical principles asimplicit agreements) tells us when we’ve found all the relevantstatistical principles for the Bayesian community.

35 / 65



There are at least two related ways in which we think the Bayesianmotivation for likelihoodist principles is limited . . .

36 / 65



Qualitative Evidence

Follow Jupiter in Retrograde Don’t follow

Orbit Jupiter Likely ImpossibleDon’t orbit Very unlikely Extremely likely

In most cases in the history of science, we can’t assign preciseprobabilities to various observations. Yet we seem capable ofdrawing conclusions about what the evidence favors.

37 / 65



Bayesianism Entails Logical OmniscienceBayesianism:

1 Bayesianism requires agents to have degrees of belief that obey theprobability axioms.

2 Probability axioms: maximal probability and additivity.

3 Probability domain: an algebra of sets (events) or a logically closedset of formulae

Logical Omniscience:

1 Agents must be maximally sure of any tautologies.

2 Agents must recognize any logically equivalent formulae/differentdescriptions of the same set.

For sets, this follows from the fact that two sets are equivalent ifthey have the same elements (and that the probability function iswell-defined).For formulae, it can easily be derived from the axioms that logicallyequivalent formulae should be assigned the same probability.

38 / 65



Bayesianism Entails Logical OmniscienceBayesianism:

1 Bayesianism requires agents to have degrees of belief that obey theprobability axioms.

2 Probability axioms: maximal probability and additivity.

3 Probability domain: an algebra of sets (events) or a logically closedset of formulae

Logical Omniscience:

1 Agents must be maximally sure of any tautologies.

2 Agents must recognize any logically equivalent formulae/differentdescriptions of the same set.

For sets, this follows from the fact that two sets are equivalent ifthey have the same elements (and that the probability function iswell-defined).For formulae, it can easily be derived from the axioms that logicallyequivalent formulae should be assigned the same probability.

38 / 65



Why Keep Bayesianism?

The previous argument seems like an equally good argument fornot endorsing a Bayesian view of LL/LP. Indeed, if you thinknothing more needs to be clarified regarding the notions of favoringand evidential equivalence, there’s no need to move to qualitativeprobability. However, both the unifying and explanatory nature ofour Bayesian framework makes it seem more attractive to addressthese issues without abandoning Bayesian ideas entirely.

39 / 65




40 / 65



Statistical problems

Typical statistical problems have three components:

A set Θ of simple hypotheses,

A set Ω consisting of possible data sequences, and

For each hypothesis θ, there is a probability measure Pθ over sets ofdata sequences in Ω.

Bayesians claim that agents’ degrees of belief can be representedby a function Q that assigns numerical probabilities to Θ× Ω.

Often, Bayesians assume Q(·|H) = Pθ so that the likelihoods areintersubjectively shared.

41 / 65



Statistical problems

Typical statistical problems have three components:

A set Θ of simple hypotheses,

A set Ω consisting of possible data sequences, and

For each hypothesis θ, there is a probability measure Pθ over sets ofdata sequences in Ω.

Bayesians claim that agents’ degrees of belief can be representedby a function Q that assigns numerical probabilities to Θ× Ω.

Often, Bayesians assume Q(·|H) = Pθ so that the likelihoods areintersubjectively shared.

41 / 65



Our Idea: We relax numerical probability functions to qualitativeorderings. This affects

The likelihood functions, and

Agents’ degrees of belief

42 / 65



Qualitative Likelihoods

Instead of Pθθ∈Θ, we will model likelihoods using a relation v.

Informally: A|θ v B|θ′ means “B is more likely if θ′ is true than Awould be if θ were true.”

Notation: A|θ ≡ B|θ′ is defined as holding when A|θ v B|θ′ andvice versa.

43 / 65



Qualitative Likelihoods

Instead of Pθθ∈Θ, we will model likelihoods using a relation v.

Informally: A|θ v B|θ′ means “B is more likely if θ′ is true than Awould be if θ were true.”

Notation: A|θ ≡ B|θ′ is defined as holding when A|θ v B|θ′ andvice versa.

43 / 65



Qualitative Priors

Instead of a prior (or posterior) Q, we we will model degrees ofbelief using a relation .

Notation: H|E ∼ H ′|F is defined as holding when H|E H ′|F andvice versa.

44 / 65



Qualitative Priors

Instead of a prior (or posterior) Q, we we will model degrees ofbelief using a relation .

Notation: H|E ∼ H ′|F is defined as holding when H|E H ′|F andvice versa.

44 / 65



Qualitative Priors

Let F be an algebra containing events in Θ× Ω and let N ⊆ F .

An agent’s beliefs are represented by a binary relation onF × (F \ N ) such that

C0. A|H B|H ′ if and only if A|H v B|H ′, i.e., agents agree upon thelikelihood of various data given the various hypotheses,

C1. is a weak ordering,C2. A ∈ N if and only if A|Θ ∼ ∅|Θ,C3. A|A ∼ B|B for all A,B,C4. A ∩ B|B ∼ A|B,C5. If A1|C1 A2|C2 and B1|C1 B2|C2 and A1 ∩ B1 = A2 ∩ B2 = ∅,

then A1 ∪ B1|C1 A2 ∪ B2|C2.C6. Suppose C ⊆ B ⊆ A and C ′ ⊆ B ′ ⊆ A′. If B|A C ′|B ′ and

C |B B ′|A′, then C |A C ′|A′.

Axioms C1-C6 are some of [Krantz et al., 2006]’s axioms forqualitative conditional probability.

45 / 65



How Does Qualitative Probability AvoidLogical Omniscience?

Agents are still required to have orderings that agree with somesubset of the axioms. However:

1 There is no maximal probability associated with the orderings*. Soagents need not be equally or maximally sure of any tautology.

2 When we move from sets/events to formulae, there will be norequirement that logically equivalent formulae have the sameprobability.

46 / 65



Further Motivation

Frequentists and likelihoodists often don’t want to assignprobabilities to hypotheses, but it seems like they will admit tohaving or wanting an ordering of hypotheses.

Saying that the evidence ”favors” one hypothesis or that we oughtto reject the null hypothesis seems to imply some sort of ordering.

Note:

These axioms are not sufficient for to be representable as aconditional probability function.

47 / 65



The Key Concept: Comparative Qualitative Favoring

48 / 65



Qualitative Comparative Favoring

Definition: Say E comparatively favors H1 to H2 at least as muchas F if

H1|E ∩ (H1 ∪ H2) H1|F ∩ (H1 ∪ H2)

for all the orderings such that H1|Θ,H2|Θ ∅|Θ.

49 / 65



Comparative Favoring, Equivalence, andNon-Comparative

As in the quantitative case, we can use this comparative notion todefine notions of

E strictly qualitatively favors H1 to H2, and

E and F are qualitative favoring equivalent

See the handout for precise definitions.

50 / 65



Qualitative LL

51 / 65



Community Favoring

Qualitative Law of Likelihood (qll): E favors θ1 to θ2 if andonly if E |θ2 < E |θ1.

52 / 65



Theorem

Suppose H1 and H2 are finite and disjoint. Then E strictly favorsH1 to H2 if and only if E |θ2 < E |θ1 for all θ1 ∈ H1 and all θ2 ∈ H2.

If H1 = θ1 and H2 = θ2 are simple, then E strictlyqualitatively favors H1 to H2 if and only if E |θ2 < E |θ1, i.e., iffqll entails so.

53 / 65



Qualitative LP

54 / 65



Motivation

Number of balls

A B C1 C2 Total

Urn Type 1 15 30 50 5 100Urn Type 2 10 20 20 50 100

By lp, pulling an A is equivalent to pulling a B on the first draw,as Pθ(A) = 1/2 · Pθ(B) for any simple hypothesis θ (about the urntype).

55 / 65



Motivation

Number of ballsA B C1 C2 Total

Urn Type 1 15 30 50 5 100Urn Type 2 10 20 20 50 100

Let C1 and C2 respectively denote the events that, after replacingthe first draw, one draws C1/C2 on the second draw.

Then Pθ1 (B ∩ C1) = Pθ1 (B) · Pθ1 (C1) = 1/2 · Pθ1 (B) = Pθ1 (A) byindependence of the draws, if the urn is Type 1.

Similarly, Pθ2 (B ∩ C2) = Pθ2 (B) · Pθ2 (C2) = 1/2 · Pθ2 (B) = Pθ2 (A).

By lp, because Pθ1 (C1) = Pθ2 (C2) > 0 andPθi (A) = Pθi (B ∩ Ci ) = Pθi (Ci ) · Pθi (B) for all i , we know A and Bare evidentially equivalent.

56 / 65



Motivation


Urn Type 1 15 30 50 5 100Urn Type 2 10 20 20 50 100





56 / 65



Motivation


Urn Type 1 15 30 50 5 100Urn Type 2 10 20 20 50 100





56 / 65



Motivation


Urn Type 1 15 30 50 5 100Urn Type 2 10 20 20 50 100





56 / 65



Motivation

Number of balls

A B C1 C2 Total

Urn Type 1 15 30 50 5 100Urn Type 2 10 20 20 50 100

Fact: A and B are equivalent according to lp if there is a set ofevents Cθθ∈Θ such that

1 Pθ(A) = Pθ(B ∩ Cθ) for all θ,

2 Pθ(Cθ) = Pθ′(Cθ′) > 0 for all θ, θ′ ∈ Θ, and

3 B and Cθ are independent given Pθ

57 / 65



Qualitative LP

Proposition

E and F are favoring equivalent if there is a set of events Cθθ∈Θ

such that

1 E |θ ≡ F ∩ Cθ|θ for all θ,

2 Cθ|θ ≡ Cυ|υ for all θ and υ, and

3 Cθ and F are qualitatively conditionally independent given θ for all θ

A partial converse is stated on the handout.

58 / 65



Philosophical Morals

59 / 65



Upshots 1

Morals: The type of agreement that ll and lp characterizeextends to qualitative settings, i.e., to characterize agreementamong agents whose beliefs satisfy relatively few coherenceconstraints.

Our work, therefore, provides a better rational reconstruction ofqualitative assessments of evidence in science, especially thosewhich predate probability theory,

And classical statisticians might agree with such coherenceconstraints . . .

60 / 65



Upshots 2

Upshots for Philosophy of Statistics: Qualitative conditions likethat provided by the previous theorem entail statistical principles(e.g., stopping rule principle) the classical statisticians often reject.

61 / 65



References I

Douglas, H. (2009). Science, policy, and the value-free ideal. Universityof Pittsburgh Press.

Edwards, A. W. F. (1984). Likelihood. CUP Archive.

Fine, T. L. (2014). Theories of probability: An examination offoundations. Academic Press.

Krantz, D. H., Luce, R. D., Suppes, P., and Tversky, A. (2006).Foundations of Measurement Volume II: Geometrical, Threshold, andProbabilistic Representations. Dover Publications, Mineola, N.Y.

62 / 65



Thanks.

Suggestions? Questions? Concerns?

63 / 65



Qualitative Conditional Independence

Let’s return to the qualitative setting . . .

Definition: Given an ordering representing an agent’s degrees ofbelief, define: A⊥CB to hold if and only if

A|B ∩ C ∼ A|C , and

B|A ∩ C ∼ B|C .

The idea is that A⊥CB represents the claim that A and B areconditionally independent given C .

64 / 65



Qualitative Independence

we must be cognizant of the fact that invocations of[stochastic independence] are usually not founded uponempirical or objective knowledge of probabilities. Quitethe contrary. Independence is adduced to permit us tosimplify and reduce the family of possible probabilisticdescriptions for a given experiment.

[Fine, 2014, p. 80]

65 / 65


Documents

QUOL: Qualitative, Objective Likelihoodism