View
227
Download
0
Category
Tags:
Preview:
Citation preview
Effect Size Estimation in Fixed Factors Between-Groups
ANOVA
Contrast Review Given a design with a single factor A with 3 or
more levels (conditions) The omnibus comparison concerns all levels (i.e., dfA >
2) A focused comparison or contrast concerns just two
levels (i.e.,df = 1) The omnibus effect is often relatively
uninteresting compared with specific contrasts (e.g., treatment 1 vs. placebo control)
A large omnibus effect can also be misleading if due to a single discrepant mean that is not of substantive interest
Comparing Groups Traditional approach is to analyze the omnibus
effect followed by analysis of all possible pairwise contrasts (i.e. compare each condition to every other condition)
However, this approach is typically incorrect (Wilkinson & TFSI,1999)—for example, it is rare that all such contrasts are interesting Also, use of traditional methods for post hoc comparisons (e.g. Newman-Keuls) reduces power for every contrast, and power may already be low
Contrast specification and tests A contrast is a directional effect that corresponds to a
particular facet of the omnibus effect In a sample, a contrast is calculated as:
a1, a2, ... , aj is the set of weights that specifies the contrast As we have mentioned
Contrast weights must sum to zero and weights for at least two different means should not equal zero
Means assigned a weight of zero are excluded from the contrast
Means with positive weights are compared with means given negative weights
1 1 2 2... k k j ja X a X a X a X
Contrast specification and tests For effect size estimation with the d family, we
generally want a standard set of contrast weights that will better allow comparison across study
In a one-way design, the sum of the absolute values of the weights in a standard set equals two (i.e., ∑ |aj| = 2) E.g. 4 groups comparing 1 and 2 vs. 3 and 4
Use weights of .5 .5 -.5 -.5
Mean difference scaling permits the interpretation of a contrast as the difference between the averages of two subsets of means
Contrast specification and tests An exception to the need for mean difference scaling is for
trends (polynomials) specified for a quantitative factor (e.g., drug dosage)
There are default sets of weights that define trend components (e.g. linear, quadratic, etc.) that are not typically based on mean difference scaling
Not usually a problem because effect size for trends is generally estimated with the r family (measures of association) Measures of association for contrasts of any kind generally
correct for the scale of the contrast weights
Orthogonal Contrasts Two contrasts are orthogonal if they each
reflect an independent aspect of the omnibus effect
For balanced designs and unbalanced designs (latter)
1 21
1 2
1
0
0
c
i
c
i
a a
a a
n
Orthogonal Contrasts Recall that for a set of all possible orthogonal
pairwise contrasts, the SSA = the total SS from the contrasts, and their eta-squares will sum to the SSA eta-square
That is, the omnibus effect can be broken down into a − 1 independent directional effects
The maximum number of orthogonal contrasts is one less than the number of groups dfA = a − 1
However, it is more important to analyze contrasts of substantive interest even if they are not orthogonal
Contrast specification and tests t-test for a contrast against the nil
hypothesis
The F is
w/in
2
t( )
w
dfs
weighted mean difference
as MS
n
2/
/ in
2
2
(1, )w inw
SSF df t
MS
SSan
Dependent Means Test statistics for dependent mean contrasts usually have
error terms based on only the two conditions compared—for example:
s2 here refers to the variance of the contrast difference scores
This error term does not assume sphericity
2
( 1)
D
t ns
ss
n
Confidence Intervals Approximate confidence intervals for
contrasts are generally fine The general form of an individual
confidence interval for Ψ is:
dferror is specific to that contrast
[ ( )]cv errors t df
Contrast specification and tests There are also corrected confidence intervals for
contrasts that adjust for multiple comparisons (i.e., inflated Type I error) Known as simultaneous or joint confidence intervals
Their widths are generally wider compared with individual confidence intervals because they are based on a more conservative critical value Examples in R using the MBESS package1
ci.c(means=c(2, 4, 9, 13), error.variance=1, c.weights=c(1, -1, -1, 1), n=c(3, 3, 3, 3), N=12, conf.level=.95) ci.c(means=c(94, 91, 92, 83), error.variance=67.375, c.weights=c(1, -1, 0, 0), n=c(4, 6, 5, 5), N=20, conf.level=.95)
Standardized contrasts The general form for standardized
contrasts (in terms of population parameters)
pooled
Standardized contrasts There are three general ways to estimate σ (i.e., the
standardizer) for contrasts between independent means:
1. Calculate d as Glass’s Δ i.e., use the standard deviation of the control/reference group
2. Calculate d as Hedge’s g i.e., use the square root of the pooled within-conditions
variance for just the two groups being compared 3. Calculate d as an extension of g
Where the standardizer is the square root of MSW based on all groups
Assumes we have met homogeneity of variance assumption Generally recommended
Standardized contrasts Calculate from a d from a tcontrast for a paper not
reporting effect size like they should If they report an F instead, which is very common,
simply take it’s square root to get the t
Recall the weights should sum to 2 CIs
Once the d is calculated one can easily obtain exact confidence intervals via the MBESS package in R as you have done in lab
2
( )a
g or d tn
Cohen’s f Cohen’s f1 provides what can
interpreted as the average standardized mean difference across the groups in question
It has a direct relation to a measure of association
As with Cohen’s d, there are guidelines regarding Cohen’s f .10, .25, .40 for small, moderate and
large effect sizes These correspond to eta-square values
of: .01, .06, .14
Again though, one should conduct the relevant literature for effect size estimation
2 22 2
2 2
2
2
1 1
1
ff
f
f
Measures of Association A measure of association describes the amount of
the covariation between the independent and dependent variables
It is expressed in an unsquared metric or a squared metric—the former is a correlation or multiple correlation if more than one predictor, the latter a variance-accounted-for effect size
A squared multiple correlation (R2) calculated in ANOVA is also called the correlation ratio or estimated eta-squared (2)
Eta-squared A measure of the degree to which variability
among observations can be attributed to conditions
Example: 2 = .50 50% of the variability seen in the scores is due to the
independent variable
2 2treatpb
total
SSR
SS
More than One factor It is a fairly common practice to calculate
eta2 (correlation ratio) for the omnibus effect but to calculate the partial correlation ratio for each contrast
As we have noted before
2partial η treat
treat error
SS
SS SS
Problem Eta-squared (since it is R-squared) is an
upwardly biased measure of association (just like R-squared was)
As such it is better used descriptively than inferentially
Omega-squared ω2 is another effect size measure that is less
biased and interpreted in the same way as eta-squared It is our adjusted R2 for the ANOVA setting
So why do we not see omega-squared so much? People don’t like small values Stat packages don’t provide it by default
2 ( 1)effect error
total error
SS k MS
SS MS
Omega-squared Put differently
2
2
( )
/ ( ) ( 1)
/ ( ) ( 1)
effect effect error
total error
effect effect error effect effect
effect effect error error effect effect
df MS MS
SS MS
df kn MS MS df Fpartial
df kn MS MS MS df F kn
Omega-squared Assumes a balanced design
eta2 does not assume a balanced design When unbalanced perhaps stick with eta or maybe use
the harmonic mean in the kn part in the previous formula
Though the omega values are generally lower than those of the corresponding correlation ratios for the same data, their values converge as the sample size increases
Note that the values can be negative—if so, interpret as though the value were zero
Comparing effect size measures Consider our previous example with item
difficulty and arousal regarding performance
Tests of Between-Subjects Effects
Dependent Variable: Score
240.000a 5 48.000 9.600 .000 .667
120.000 1 120.000 24.000 .000 .500
60.000 2 30.000 6.000 .008 .333
60.000 2 30.000 6.000 .008 .333
120.000 24 5.000
360.000 29
SourceB/t groups
Difficulty
Arousal
Difficulty * Arousal
Error
Total
Type IIISum ofSquares df Mean Square F Sig.
Partial EtaSquared
R Squared = .667 (Adjusted R Squared = .597)a.
Comparing effect size measures
2 ω2 Partial 2
f
B/t groups .67 .59 .67 1.42
Difficulty .33 .32 .50 .71
Arousal .17 .14 .33 .45
Interaction .17 .14 .33 .45
Slight differences due to rounding, f based on eta-squared. Given the balanced design, when looking at specific effects eta-squared serve as the more appropriate semi-partial correlation squared.
No p-values As before, programs are available to
calculate confidence intervals for an effect size measure
Example using the MBESS package for the overall effect 95% CI on ω2: .20 to .69
No p-values Ask yourself as we have before, if the null
hypothesis is true, what would our effect size be (standardized mean difference or proportion of variance accounted for)?
0 Rather than do traditional hypothesis testing, one
can simply see if our CI for the effect size contains the value of zero (or, in eta-squared case, gets really close)
If not, reject H0 This is superior in that we can use the NHST
approach, get a confidence interval reflecting the precision of our estimates, focus on effect size, and de-emphasize the p-value
Recommended