Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Significance, P-value and t-tests
Question: Whether an observation is significantly different?
Solution: Hypothesis testing
1. State the null hypothesis(H0)
2. Choose appropriate test ,test statistic T
3. Derive the distribution of T under H0
4. Compute the observed value tobs of T
5. Calculate the p-value
6. Reject H0 if the p-value is less than the selected significance
P-value tells us how strong the evidence is against H0.
P-value is not the probability that H0 is True
To calculate P-value, we need to know the distribution under H0
• Normal distribution
The t-distribution
t-distribution take into account that most sample will underestimate population’s variability
Interpreting P Values
Compatibility
“A P value measures a sample’s compatibility with a hypothesis, not the truth of the hypothesis”
Requires combinations of other measures to make correct decisions
Bayesian cutoffs
P-value-based FDR calculations (eFDR)
P values and effect sizes
Bayesian Analysis
Bayes factor: B Ratio of average likelihood under alternative hypothesis and null
hypothesis
Requires specification of a prior distribution
Upper bound for Bayes factor: 𝐵 Provides the maximum alternative to null likelihood ratio
𝐵 ≥ 20 indicates strong evidence for alternative hypothesis
𝐵 ≤ −1/(𝑒𝑃𝑙𝑛(𝑃)) 𝑃 < 𝛼 = 0.05 → 𝐵 ≤ 2.5
𝑃 < 𝛼 = 0.025 → 𝐵 ≤ 3.9
𝐵 > 20 → 𝛼 < 0.0032
False Discovery Rate
FDR is the expected proportion of rejected null hypotheses that are false rejections
Let 𝜋0 represent the true proportion of tests that are truly null 𝛼𝜋0 is the expected number of false rejections
𝛽(1 − 𝜋0) is the number of non-null tests that we end up rejecting, where 𝛽 is the power of the test (𝛽 = 𝑃(𝐻0 𝑟𝑒𝑗𝑒𝑐𝑡𝑒𝑑|𝐻1 𝑡𝑟𝑢𝑒))
i.e. the number of correct rejections
𝑒𝐹𝐷𝑅 = 𝛼𝜋0/(𝛼𝜋0+ 𝛽(1 − 𝜋0) ) Reasonable estimate for FDR
Effect Size Strong support for consideration of effect size when interpreting
P values
Provide confidence interval for parameter of interest Based on confidence level
P Values
Random Variables Random samples result in random distribution for P values
Null true
P value ~ Unif(0,1), 𝜇 = 0.5, 𝜎 = 0.29
Alternative true
Increased power results in decreased variability