Upload
jemille6
View
7.788
Download
0
Tags:
Embed Size (px)
Citation preview
Final APS Mayo / 1
Error Statistical Control: Forfeit at your Peril Deborah Mayo
• A central task of philosophers of science is to address
conceptual, logical, methodological discomforts of scientific practices—still we’re rarely called in
• Psychology has always been more self-conscious than most • To its credit, replication crises led to programs to restore
credibility: fraud busting, reproducibility studies. • There are proposed methodological reforms––many
welcome, some of them quite radical • Without a better understanding of the problems, many
reforms are likely to leave us worse off.
Final APS Mayo / 2
1. A Paradox for Significance Test Critics
Critic1: It’s much too easy to get a small P-value. Critic2: We find it very difficult to replicate the small P-values
others found.
Is it easy or is it hard?
R.A. Fisher: it’s easy to lie with statistics by selective reporting (he called it the “political principle”) Sufficient finagling—cherry-picking, P-hacking, significance seeking—may practically guarantee a researcher’s preferred hypothesis gets support, even if it’s unwarranted by evidence.
Final APS Mayo / 3
2. Bad Statistics Severity Requirement: If data x0 agree with a hypothesis H, but the test procedure had little or no capability, i.e., little or no probability of finding flaws with H (even if H is incorrect), then x0 provide poor evidence for H.
Such a test we would say fails a minimal requirement for a stringent or severe test.
• This seems utterly uncontroversial.
Final APS Mayo / 4
• Methods that scrutinize a test’s capabilities, according to their severity, I call error statistical.
• Existing error probabilities (confidence levels, significance
levels) may but need not provide severity assessments.
• New name: frequentist, sampling theory, Fisherian, Neyman-Pearsonian—are too associated with hard line views.
Final APS Mayo / 5
3. Two main views of the role of probability in inference Probabilism. To provide an assignment of degree of probability, confirmation, support or belief in a hypothesis, absolute or comparative, given data x0. (e.g., Bayesian, likelihoodist)—with due regard for inner coherency Performance. To ensure long-run reliability of methods, coverage probabilities, control the relative frequency of erroneous inferences in a long-run series of trials (behavioristic Neyman-Pearson) What happened to using probability to assess the error probing capacity by the severity criterion?
Final APS Mayo / 6
• Neither “probabilism” nor “performance” directly captures it.
• Good long-run performance is a necessary, not a sufficient, condition for avoiding insevere tests.
• The problems with selective reporting, cherry picking,
stopping when the data look good, P-hacking, barn hunting, are not problems about long-runs—
• It’s that we cannot say about the case at hand that it has
done a good job of avoiding the sources of misinterpreting data.
Final APS Mayo / 7
• Probabilism says H is not warranted unless it’s true or probable (or increases probability, makes firmer–some use Bayes rule, but its use doesn’t make you Bayesian).
• Performance says H is not warranted unless it stems from a method with low long-‐run error.
• Error Statistics (Probativism) says H is not warranted unless something (a fair amount) has been done to probe ways we can be wrong about H.
Final APS Mayo / 8
• If you assume probabilism is required for inference, it follows error probabilities are relevant for inference only by misinterpretation. False! • Error probabilities have a crucial role in appraising well-‐testedness, which is very different from appraising believability, plausibility, confirmation.
• It’s crucial to be able to say, H is highly believable or plausible but poorly tested. [Both H and not-‐H may be poorly tested)
• Probabilists can allow for the distinct task of severe testing
Final APS Mayo / 9
• It’s not that I’m keen to defend many common uses of significance tests; my work in philosophy of statistics has been to provide the long sought “evidential interpretation” (Birnbaum) of frequentist methods, to avoid classic fallacies.
• It’s just that the criticisms are based on serious
misunderstandings of the nature and role of these methods; consequently so are many “reforms”.
• Note: The severity construal blends testing and subsumes
(improves) interval estimation, but I keep to testing talk to underscore the probative demand.
Final APS Mayo / 10
4. Biasing selection effects:
One function of severity is to identify which selection effects are problematic (not all are) Biasing selection effects: when data or hypotheses are selected or generated (or a test criterion is specified), in such a way that the minimal severity requirement is violated, seriously altered or incapable of being assessed.
Picking up on these alterations is precisely what enables error statistics to be self-correcting— Let me illustrate.
Final APS Mayo / 11
Capitalizing on Chance
We often see articles on fallacious significance levels: When the hypotheses are tested on the same data that suggested them and when tests of significance are based on such data, then a spurious impression of validity may result. The computed level of significance may have almost no relation to the true level…Suppose that twenty sets of differences have been examined, that one difference seems large enough to test and that this difference turns out to be ‘significant at the 5 percent level.’ ….The actual level of significance is not 5 percent, but 64 percent! (Selvin, 1970, p. 104)
Final APS Mayo / 12
• This is from a contributor to Morrison and Henkel’s Significance Test Controversy way back in 1970!
• They were clear on the fallacy: blurring the “computed” or “nominal” significance level, and the “actual” or “warranted” level.
• There are many more ways you can be wrong with hunting (different sample space).
• Nowadays, we’re likely to see the tests blamed for permitting such misuses (instead of the testers).
• Even worse are those statistical accounts where the abuse vanishes!
Final APS Mayo / 13
What defies scientific sense? On some views, biasing selection effects are irrelevant….
Stephen Goodman (epidemiologist): Two problems that plague frequentist inference: multiple comparisons and multiple looks, or, as they are more commonly called, data dredging and peeking at the data. The frequentist solution to both problems involves adjusting the P-‐value…But adjusting the measure of evidence because of considerations that have nothing to do with the data defies scientific sense, belies the claim of ‘objectivity’ that is often made for the P-‐value.” (1999, p. 1010).
Final APS Mayo / 14
5. Likelihood Principle (LP) The vanishing act takes us to the pivot point around which much debate in philosophy of statistics revolves: In probabilisms, the import of the data is via the ratios of likelihoods of hypotheses
P(x0;H1)/P(x0;H0)
Different forms: posterior probabilities, positive B-‐boost, Bayes factor The data x0 is fixed, while the hypotheses vary
Final APS Mayo / 15
Savage on the LP:
“According to Bayes’s theorem, P(x|µ)...constitutes the entire evidence of the experiment, that is, it tells all that the experiment has to tell. More fully and more precisely, if y is the datum of some other experiment, and if it happens that P(x|µ) and P(y|µ) are proportional functions of µ (that is, constant multiples of each other), then each of the two data x and y have exactly the same thing to say about the values of µ… (Savage 1962, p. 17).
Final APS Mayo / 16
All error probabilities violate the LP (even without selection effects): “Sampling distributions, significance levels, power, all depend on something more [than the likelihood function]–something that is irrelevant in Bayesian inference–namely the sample space”. (Lindley 1971, p. 436) That is why properties of the sampling distribution of test statistic d(X) disappear for accounts that condition on the particular data x0
Final APS Mayo / 17
Paradox of Optional Stopping: Error probing capabilities are altered not just by cherry picking and data dredging, but also via data dependent stopping rules:
We have a random sample from a Normal distribution with mean µ and standard deviation σ, Xi ~ N(µ,σ2), 2-‐sided
H0: µ = 0 vs. H1: µ ≠ 0.
Instead of fixing the sample size n in advance, in some tests, n is determined by a stopping rule:
Keep sampling until H0 is rejected at the .05 level
i.e., keep sampling until |𝑋| ≥ 1.96 σ/ 𝑛.
Final APS Mayo / 18
“Trying and trying again”: having failed to rack up a 1.96 SD difference after, say, 10 trials, the researcher went on to 20, 30 and so on until finally obtaining a 1.96 SD difference. Nominal vs. Actual significance levels: with n fixed the type 1 error probability is .05. With this stopping rule the actual significance level differs from, and will be greater than .05.
Final APS Mayo / 19
Jimmy Savage (1959 forum) audaciously declared:
“optional stopping is no sin”
so the problem must be with significance levels (because they pick up on it).
On the other side: Peter Armitage, who had brought up the problem, also uses biblical language
“thou shalt be misled”
if thou dost not know the person tried and tried again. (72)
Final APS Mayo / 20
Where the Bayesians here claim:
“This irrelevance of stopping rules to statistical inference restores a simplicity and freedom to experimental design that had been lost by classical emphasis on significance levels” (in the sense of Neyman and Pearson) (Edwards, Lindman, Savage 1963, p. 239). The frequentists: While it may restore "simplicity and freedom" it does so at the cost of being unable to adequately control probabilities of misleading interpretations of data) (Birnbaum).
Final APS Mayo / 21
6. Current Reforms are Probabilist Probabilist reforms to replace tests (and CIs) with likelihood ratios, Bayes factors, HPD intervals, or just lower the P-value (so that the maximal likely alternative gets .95 posterior)
while ignoring biasing selection effects, will fail The same p-hacked hypothesis can occur in Bayes factors; optional stopping can exclude true nulls from HPD intervals
With one big difference: Your direct basis for criticism and possible adjustments has just vanished To repeat: Properties of the sampling distribution d(x) disappear for accounts that condition on the particular data.
Final APS Mayo / 22
7. How might probabilists block intuitively unwarranted inferences? (Consider first subjective) When we hear there’s statistical evidence of some unbelievable claim (distinguishing shades of grey and being politically moderate, ovulation and voting preferences), some probabilists claim—you see, if our beliefs were mixed into the interpretation of the evidence, we wouldn’t be fooled.
We know these things are unbelievable, a subjective Bayesian might say. That could work in some cases (though it still wouldn’t show what researchers had done wrong)—battle of beliefs.
Final APS Mayo / 23
It wouldn’t help with our most important problem: (2 critics) How to distinguish the warrant for a single hypothesis H with different methods (e.g., one has biasing selection effects, another, registered results and precautions)? Besides, as committees investigating questionable practices know, researchers really do sincerely believe their hypotheses. So now you’ve got two sources of flexibility, priors and biasing selection effects (which can no longer be criticized).
Final APS Mayo / 24
8. Conventional: Bayesian-‐Frequentist reconciliations?
The most popular probabilisms these days are non-subjective, default, reference:
• because of the difficulty of eliciting subjective priors, • the reluctance of scientists to allow subjective beliefs to
overshadow the information provided by data. Default, or reference priors are designed to prevent prior beliefs from influencing the posteriors.
• A classic conundrum: no general non-‐informative prior exists, so most are conventional.
Final APS Mayo / 25
“The priors are not to be considered expressions of uncertainty, ignorance, or degree of belief. Conventional priors may not even be probabilities…” (Cox and Mayo 2010, p. 299)
• Prior probability: An undefined mathematical construct for obtaining posteriors (giving highest weight to data, or satisfying invariance, or matching frequentists, or….).
Leading conventional Bayesians (J. Berger) still tout their methods as free of concerns with selection effects, stopping rules (stopping rule principle)
Final APS Mayo / 26
There are some Bayesians who don’t see themselves as fitting under either the subjective or conventional heading, and may even reject probabilism…..
Final APS Mayo / 27
Before concluding…I don’t ignore fallacies of current methods 9. How the severity analysis avoids classic fallacies Fallacies of Rejection: Statistical vs. Substantive Significance
i. Take statistical significance as evidence of substantive theory H* that explains the effect.
ii. Infer a discrepancy from the null beyond what the test warrants.
(i) Handled with severity: flaws in the substantive alternative H* have not been probed by the test, the inference from a statistically significant result to H* fails to pass with severity.
Final APS Mayo / 28
Merely refuting the null hypothesis is too weak to corroborate substantive H*, “we have to have ‘Popperian risk’, ‘severe test’ [as in Mayo], or what philosopher Wesley Salmon called ‘a highly improbable coincidence’” (Meehl and Waller 2002,184).
• NHSTs (supposedly) allow moving from statistical to substantive; if so, they exist only as abuses of tests: they are not licensed by any legitimate test.
• Severity applies informally: Much more attention to these quasi-‐formal statistical substantive links: Do those proxy variables capture the intended treatments? Do the measurements reflect the theoretical phenomenon?
Final APS Mayo / 29
Fallacies of Rejection: (ii) Infer a discrepancy beyond what’s warranted: Severity sets up a discrepancy parameter γ (never just report P-‐value)
A statistically significant effect may not warrant a meaningful effect — especially with n sufficiently large: large n problem.
• Severity tells us: an α-‐significant difference is indicative of less of a discrepancy from the null if it results from larger (n1) rather than a smaller (n2) sample size (n1 > n2 )
What’s more indicative of a large effect (fire), a fire alarm that goes off with burnt toast or one so insensitive that it doesn’t go off unless the house is fully ablaze? The larger sample size is like the one that goes off with burnt toast.)
Final APS Mayo / 30
Fallacy of Non-‐Significant results: Insensitive tests • Negative results do not warrant 0 discrepancy from the null, but we can use severity to rule out discrepancies that, with high probability, would have resulted in a larger difference than observed
• akin to power analysis (Cohen) but sensitive to x0
• We hear sometimes negative results are uninformative: not so.
No point in running replication research if your account views negative results as uninformative
Final APS Mayo / 31
Confidence Intervals also require supplementing There’s a duality between tests and intervals: values within the (1 -‐ α) CI are non-‐rejectable at the α level (but that doesn’t make them well-‐warranted) • Still too dichotomous: in /out, plausible/not plausible (Permit fallacies of rejection/non-‐rejection)
• Justified in terms of long-‐run coverage • All members of the CI treated on par • Fixed confidence levels (SEV need several benchmarks) • Estimation is important but we need tests for distinguishing real and spurious effects, and checking assumptions of statistical models
Final APS Mayo / 32
10. Error Statistical Control: Forfeit at your Peril
• The role of error probabilities in inference, is not long-‐run error control, but to severely probe flaws in your inference today.
• It’s not a high posterior probability in H that’s wanted (however construed) but a high probability our procedure would have unearthed flaws in H.
• Many reforms are based on assuming a philosophy of probabilism
• The danger is that some reforms may enable rather than directly reveal illicit inferences due to biasing selection effects.
Final APS Mayo / 33
Mayo and Cox (2010): Frequentist Principle of Evidence (FEV); SEV: Mayo and Spanos (2006)
• FEV/SEV: insignificant result: A moderate P-‐value is evidence of the absence of a discrepancy δ from H0, only if there is a high probability the test would have given a worse fit with H0 (i.e., d(X) > d(x0) ) were a discrepancy δ to exist.
• FEV/SEV significant result d(X) > d(x0) is evidence of discrepancy δ from H0, if and only if, there is a high probability the test would have d(X) < d(x0) were a discrepancy as large as δ absent.
Final APS Mayo / 34
Test T+: Normal testing: H0: µ < µ0 vs. H1: µ > µ0 σ known (FEV/SEV): If d(x) is not statistically significant, then µ < M0 + kεσ/√𝑛 passes the test T+ with severity (1 – ε). (FEV/SEV): If d(x) is statistically significant, then µ > M0 + kεσ/√𝑛 passes the test T+ with severity (1 – ε). where P(d(X) > kε) = ε.
Final APS Mayo / 35
REFERENCES: Armitage, P. 1962. “Contribution to Discussion.” In The Foundations of Statistical Inference:
A Discussion, edited by L. J. Savage. London: Methuen. Barnard, G. A. 1972. “The Logic of Statistical Inference (review of ‘The Logic of Statistical
Inference’ by Ian Hacking).” British Journal for the Philosophy of Science 23 (2) (May 1): 123–132.
Berger, J. O. 2006. “The Case for Objective Bayesian Analysis.” Bayesian Analysis 1 (3): 385–402.
Birnbaum, A. 1970. “Statistical Methods in Scientific Inference (letter to the Editor).” Nature 225 (5237) (March 14): 1033.
Cox, D. R., and Deborah G. Mayo. 2010. “Objectivity and Conditionality in Frequentist Inference.” In Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability, and the Objectivity and Rationality of Science, edited by Deborah G. Mayo and Aris Spanos, 276–304. Cambridge: Cambridge University Press.
Edwards, W., H. Lindman, and L. J. Savage. 1963. “Bayesian Statistical Inference for Psychological Research.” Psychological Review 70 (3): 193–242.
Fisher, R. A. 1955. “Statistical Methods and Scientific Induction.” Journal of the Royal Statistical Society, Series B (Methodological) 17 (1) (January 1): 69–78.
Goodman SN. Toward evidence-based medical statistics. 2: The Bayes factor. Annals of Internal Medicine 1999; 130:1005 –1013.
Final APS Mayo / 36
Lindley, D. V. 1971. “The Estimation of Many Parameters.” In Foundations of Statistical Inference, edited by V. P. Godambe and D. A. Sprott, 435–455. Toronto: Holt, Rinehart and Winston.
Mayo, D. G. 1996. Error and the Growth of Experimental Knowledge. Science and Its Conceptual Foundation. Chicago: University of Chicago Press.
Mayo, D. G., and A. Spanos. 2006. “Severe Testing as a Basic Concept in a Neyman–Pearson Philosophy of Induction.” British Journal for the Philosophy of Science 57 (2) (June 1): 323–357.
Mayo, D. G., and A. Spanos. 2011. “Error Statistics.” In Philosophy of Statistics, edited by Prasanta S. Bandyopadhyay and Malcom R. Forster, 7:152–198. Handbook of the Philosophy of Science. The Netherlands: Elsevier.
Meehl, Paul E., and Niels G. Waller. 2002. “The Path Analysis Controversy: A New Statistical Approach to Strong Appraisal of Verisimilitude.” Psychological Methods 7 (3): 283–300.
Morrison, Denton E., and Ramon E. Henkel, ed. 1970. The Significance Test Controversy: A Reader. Chicago: Aldine De Gruyter.
Pearson, E.S. & Neyman, J. (1930). On the problem of two samples. Joint Statistical Papers by J. Neyman & E.S. Pearson, 99-115 (Berkeley: U. of Calif. Press). First published in Bul. Acad. Pol.Sci. 73-96.
Savage, L. J. 1962. The Foundations of Statistical Inference: A Discussion. London: Methuen. Selvin, H. 1970. “A critique of tests of significance in survey research. In The significance test
controversy, edited by D. Morrison and R. Henkel, 94-106. Chicago: Aldine De Gruyter.