Embed Size (px)
Citation preview
Balkin, R. S.(2008). 1
Effect Size
Rick Balkin, Ph.D., LPCDepartment of Counseling
Texas A&M [email protected]
Balkin, R. S.(2008). 2
Statistical vs. PracticalSignificance
Statistical significance refers to the probabilitythat the rejection of the null hypothesisoccurred outside the realm of chance (alphalevel).
Practical significance refers to themeaningfulness of the differences, byspecifying the magnitude of the differencesbetween the means or the strength of theassociation between the independentvariable(s) and the dependent variable.
Balkin, R. S.(2008). 3
Practical significance: why weneed it.
A school counselor wants to compare aset of scores on the SAT to the nationalnorm. The population has a mean of500 and a standard deviation of 100.
Balkin, R. S.(2008). 4
Practical significance: why weneed it.
If the school counselor has 25 students in the sample
with a mean (
X
) of 520, then the z-test would be
conducted as follows: 00.1
20
20
25
100
500520==
!=
!=
!=
n
XXz
X"
µ
"
µ
With an alpha level of .05 (non -directional) and z crit = 1.96, there is no statistic ally significant difference
between the sample group and the population ( z = 1.00,
p > .05).
Balkin, R. S.(2008). 5
Practical significance: why weneed it.
Now, take the same scores, but increase the sample size
to 100.
00.210
20
100
100
500520==
!=
!=
!=
n
XXz
X"
µ
"
µ
With an alpha level of .05 (non -directional) and z crit =
1.96, there is a statistically significant difference
between the sample group and the population ( z = 2.00,
p < .05). The observed value is greater than the critical
value (2.00> 1.96).
Balkin, R. S.(2008). 6
Practical significance: why weneed it.
Did you notice that with the smaller samplesize you did not have statistical significancebut with the larger sample size you did?
Although the magnitude of the meandifferences did not change, the interpretationof the results changed strictly based on theincrease in sample size. When sample sizeincreased, the error decreased.
Balkin, R. S.(2008). 7
Practical significance: why weneed it.
Thus, statistically significant differencesare more likely to occur when largesamples are utilized.
Nearly any null hypothesis can berejected when a large enough sample isattained.
Balkin, R. S.(2008). 8
Practical significance: why weneed it.
Practical significance is importantbecause it addresses the magnitude ofa treatment effect without thecomplication of sample size, therebyproviding more meaningful informationthat has usefulness to practitioners andresearchers (Kirk, 1995).
Balkin, R. S.(2008). 9
Practical significance: why weneed it.
The following procedures are utilized toprovide measures of effect size todetermine practical significance.
Currently, statistical packages do notcompute Cohen’s d or Cohen’s f, whichmeasure effect size in standarddeviation units. However, they arerelatively simple computations.
Balkin, R. S.(2008). 10
Practical significance: why weneed it.
The reporting of practical significance is veryimportant when reporting results andmandatory in many social science journals.
“For the reader to fully understand theimportance of your findings, it is almostalways necessary to include some index ofeffect size or strength of relationship in yourResults section” (APA, 2001, p. 25).
Balkin, R. S.(2008). 11
Cohen’s d
Cohen’s d is used to determine the effect sizefor the differences between two groups, suchas in a t-test or pairwise comparisons (i.e.Tukey post hoc), and is expressed in standarddeviation units.
Cohen (1988) created the followingcategories to interpret d: Small = .2 Medium = .5 Large = .8
Balkin, R. S.(2008). 12
Cohen’s d
Cohen’s d =
)1()1(21
212
2
2
21
!+!
+==
!
""nn
xxss
or
MS
or
s
XX
ww
error
error
The numerator value is the difference between two group means. The denominatoris the error term, which can be expressed in one of three ways.
Balkin, R. S.(2008). 13
Computing Cohen’s dTable 2.
Tukey post hoc analysis
Group
Comparisons Mean Difference p d
1 2 -3.00 0.0815 1.65
3 -1.00 0.8215 0.55
4 3.60 * 0.0301 1.97
2 3 2.00 0.3393 1.09
4 6.60* 0.0002 3.62
3 4 4.60* 0.0052 2.52
*p < .05
From the ANOVA example in thenotepack, the first group comparison would be computed asfollows:
65.1325.3
3=
!
The -3 came from subtractionof the means from groups 1 and 2 in the ANOVA example.
Balkin, R. S.(2008). 14
Understanding Cohen’s d
So, if d = 1.65, then the differencebetween the groups is 1.65 standarddeviation units.
This would be considered a very largeeffect size, as it is greater than .8.
Balkin, R. S.(2008). 15
Cohen’s f
Cohen’s f also expresses effect size instandard deviation units, but does sofor two or more groups.
When conducting an ANOVA, Cohen’s fcan be computed to determine thepractical significance in the differencesamong the groups.
Balkin, R. S.(2008). 16
Cohen’s f
Like the ANOVA, the Cohen’s f will identifythe magnitude of the differences among thegroups, but it will not explain differencesbetween specific groups.
To identify differences between specificgroups, a Tukey post hoc analysis followed byCohen’s d for each pairwise comparisonwould be necessary.
Balkin, R. S.(2008). 17
Cohen’s f
Cohen (1988) created the followingcategories to interpret f: Small = .10 Medium = .25 Large = .40
Balkin, R. S.(2008). 18
Computing Cohen’s f
31.130.13
)69.1381.41.801(.
325.3)4(
])1.64.2()1.67()1.69()1.66[(
)(
)( 22222
=+++
=
!+!+!+!=
!="
error
j
MSJf
µµ
So, a large effect size was found among the four
groups with an effect size of approximately 1.31
standard deviations.
Balkin, R. S.(2008). 19
Omega squared ω2 and Eta-squared η2
Practical significance is not alwaysmeasured in standard deviation unitsand may be expressed in variance units.
There are mathematical relationshipsbetween effect sizes expressed instandard deviation units and strengthsof association expressed in varianceunits.
2!
Balkin, R. S.(2008). 20
Omega squared ω2 and Eta-squared η2
However, when conducting parametric statistics, inwhich the focus of the study is on group differences,it is best practice to express effect size in standarddeviation units as it better compliments thedescriptive data, such as means and standarddeviations.
As a rule of thumb, Cohen’s d and Cohen’s f may bemore informative for ANOVA. However, manystatistical packages provide measures of strength ofassociation, especially η2 and ω2, and so they arewidely used.
Balkin, R. S.(2008). 21
Omega squared ω2 and Eta-squared η2
Cohen (1988) created the followingcategories to interpret strength ofassociation: Small = .02 Medium = .13 Large = .26
Balkin, R. S.(2008). 22
Eta-squared: η2
Eta-squared refers to strength of associationbetween the independent variable(s) and thedependent variable.
It indicates the amount of variance accountedfor in the dependent variable by theindependent variable(s).
If the strength of association is weak, or low,the independent variable(s) have lessmeaning/relevance to the dependent variable.
Balkin, R. S.(2008). 23
Eta-squared: η2
Similar to Cohen’s f , .683 is a verylarge effect size. The IV accounts for63% of the variance in the DV.
68.
8.167
6.142=
1==
TOT
B
SS
SS!
Balkin, R. S.(2008). 24
Omega squared: ω2
The computation of
2!
also uses terms from the
ANOVA computation:
WTOT
WB
MSSS
MSjSS
+
""=
))(1(2!
where SSTOT is the sum of SS B + SSW and j is the
number of groups.
Balkin, R. S.(2008). 25
Omega squared: ω2
61.125.171
625.104
325.38.167
)325.3)(14(6.1142 ==+
!!="
From this statistic, we can conclude that the four
groups of students account for 61% of the variance in
self-efficacy scores .
Balkin, R. S.(2008). 26
Effect size summary
Keep in mind, effect size is alwayscomputed when a statistical test isconducted.
Even if your F-test is not significant, youshould still report a Cohen’s F.However, since a TMC is not conductedfor a non-significant test, no furtheranalysis is necessary.
