Upload
dana-harrison
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Statistical PowerThe ability to find a difference when
one really exists.
Statistical Power
The probability of rejecting a false null hypothesis (H0).
The probability of obtaining a value of t (or z) that is large enough to reject H0 when H0 is actually false
We always test the null hypothesis against an alternative/research hypothesis
Usually the goal is to reject the null hypothesis in favor of the alternative
Why is Power Important?
As researchers, we put a lot of effort into designing and conducting our research. This effort may be wasted if we do not have sufficient power in our studies to find the effect of interest.
Type I versus Type II Error
A researcher can make two types of error when reporting the results of a statistical test.
Actual State of Reality
Researcher Decision
H0 is true H0 is false
Reject H0 Type I error ()Correct Decision
(1 – β)
Accept H0Correct Decision
(1 – )Type II error (β)
Type I Error
The probability of a type I Error is determined by the alpha () level set by the researcher
Actual State of Reality
Researcher Decision
H0 is true H0 is false
Reject H0 Type I error ()Correct Decision
(1 – β)
Accept H0Correct Decision
(1 – )Type II error (β)
Type II Error
A type II Error (β) results when the researcher finds that there isn’t a difference, when there really is one.
Actual State of Reality
Researcher Decision
H0 is true H0 is false
Reject H0 Type I error ()Correct Decision
(1 – β)
Accept H0Correct Decision
(1 – )Type II error (β)
Statistical Power
Power is the ability of a test to detect a real effect. It is measured as a probability that equals 1 – β.
Actual State of Reality
Researcher Decision
H0 is true H0 is false
Reject H0 Type I error ()Correct Decision
(1 – β)
Accept H0Correct Decision
(1 – )Type II error (β)
Power depends on…
To discuss power we need to understand the variables that affect its size.
1. The alpha level set by the researcher
2. The sample size (N)
3. The effect size (e.g., Cohen’s d)
Power and Alpha ()
An increase in alpha, say from .05 to .1, artificially increases the power of a study.
Increasing alpha reduces the risk of making a type II error, but increases that of a type I.
Increasing the risk of making a type I error, in many cases, may be worse than making a type II error. E.g., replacing an effective chemotherapy drug
with one that is, in reality, less effective.
Power and Sample Size (N)
Power increases as N increases. The more independent scores that are
measured or collected, the more likely it is that the sample mean represents the true mean.
Prior to a study, researchers rearrange the power calculation to determine how many scores (subjects or N) are needed to achieve a certain level of power (usually 80%).
Power and Effect Size
Effect size is a measure of the difference between the means of two groups of data.
For example, the difference in mean jump ht. between samples of vball and bball players.
As effect size increases, so does power. For example, if the difference in mean jump
ht. was very large, then it would be very likely that a t-test on the two samples would detect that true difference.
A Little More on Effect Size
While a p-value indicates the statistical significance of a test, the effect size indicates the “practical” significance.
If the units of measurement are meaningful (e.g., jump height in cm), then the effect size can simply be portrayed as the difference between two means.
If the units of measurement are not meaningful (questionnaire on behaviour), then a standardized method of calculating effect size is useful.
Cohen’s d Cohen’s d is a common effect size index It describes the difference between two means in
terms of number of standard deviations The standard deviation (σpooled) represents a
weighted average variance from both samples
pooled
uud
21
)1()1(
)1()1(
21
222
211
NN
NNpooled
Hypothetical Example
To understand statistical power the following slides provide a hypothetical example.
Assume that we know the actual effect size. The actual difference between the means.
Jump Height Example
Basketball vs. Volleyball, who jumps higher? We have 16 athletes in each sample (N=16) We know the population means are:
Basketball: Mean jump ht = 30 5.7 in. Volleyball: Mean jump ht = 36 5.7 in.
Alpha = .05
Using the above information we can graphically demonstrate statistical power.
Knowing there is a difference, how many times out of 100 tests would we be correct?
Step 1. What’s t-critical for the study?
For what t score will we consider there to be a significant difference between bball and vball?
We know, N=32 (df = 30), =.05, and 2-tailed Use =tinv() in Excel tinv(.05/2, 30) = 2.36….t critical = 2.36
t critical: +2.36
-3-4 -1-2 0 1 2 3 4
t-distribution for 30
degrees of freedom / 2 = .025
t critical: -2.36
/ 2 = .025
We know the mean jump height for bball is 30, We know the mean jump height for bball is 30, then what would vball need to be to get t = 2.36?then what would vball need to be to get t = 2.36?
Use the independent t-test equationUse the independent t-test equation
22BV
BV
SEMSEM
XXt
For both groups
N = 16, Stdev = 5.7
0.2
3036.2
VX 8.34VX
Step 2Step 2
On the actual distribution ofOn the actual distribution ofXXVV,which has a mean ,which has a mean
of 36, what would be the t-value for 34.8 in? of 36, what would be the t-value for 34.8 in?
Can you calculate the probability (area) of getting Can you calculate the probability (area) of getting a mean a mean 34.8 if the real mean is 36?...Type 2 34.8 if the real mean is 36?...Type 2 ErrorError
Critical value ofXV: 34.8 in
2422 2826 30 32 34 36 38
Distribution of values ofXV
if H0 is true (V = 30) and
SEDiff is 2
t critical: +2.36
-3-4 -1-2 0 1 2 3 4
/ 2 = .025 / 2 = .025
t critical: -2.36t-distribution for 30
degrees of freedom
Step 3Step 3
β = .20Power
(1- β) = .80
Step 4Step 4
Distribution of values
ofXV if V = 36 and
SEMean is 1.43.
34.8
31.730.3 33.2 36 37.4 38.9 40.3 41.734.6
43.1
8.3436 t 87.0t
tdist(t,df,tails)
tdist(.87,15,1) = .20
-3-4 -1-2 0 1 2 3 4
t-distribution for 15
degrees of freedom
t = - 0.87
Step 4Step 4
Distribution of values
ofX if H0 is true ( = 30)
and SEDiff is 2.
t critical: +2.36
-3-4 -1-2 0 1 2 3 4
2422 2826 30 32 34 36 38
Critical value ofX: 34.8 in
/ 2 = .025 / 2 = .025
t critical: -2.36t-distribution for 30
degrees of freedom
β = .20Power
(1- β) = .80
Distribution of values
ofXV if V = 36 and
SEMean is 1.43.
34.8
31.730.3 33.2 36 37.4 38.9 40.3 41.734.6
Power Calculator
http://www.stat.ubc.ca/~rollin/stats/ssize/n2.html