Effect Size Calculation for Effect Size Calculation for Meta-AnalysisMeta-Analysis

Effect Size Calculation for Effect Size Calculation for Meta-AnalysisMeta-Analysis

Robert M. Bernard Robert M. Bernard

Centre for the Study of Learning and Performance Centre for the Study of Learning and Performance

Concordia UniversityConcordia University

February 24, 2010February 24, 2010

04/21/23 2

Main Purposes of a Main Purposes of a Meta-AnalysisMeta-Analysis

Main Purposes of a Main Purposes of a Meta-AnalysisMeta-Analysis

A meta-analysis attempts to …

04/21/23 3

What is an Effect size?What is an Effect size?What is an Effect size?What is an Effect size?

04/21/23 4

Types of Effect SizesTypes of Effect SizesTypes of Effect SizesTypes of Effect Sizes

Most reviews use …

• d-family of effect sizes, including the standardized mean difference, or

• r-family of effect sizes, including the correlation coefficient, or

• the odds ratio (OR) family of effect sizes, including proportions and other measures for categorical data.

04/21/23 5

Effect Size ExtractionEffect Size ExtractionEffect Size ExtractionEffect Size Extraction

Effect size (ES) extraction involves …

• Locating descriptive or other statistical information contained in studies.

• Converting statistical information into a standard metric (effect size) by which studies can be compared and/or combined.

04/21/23 6

Choice of an Effect SizeChoice of an Effect Size

When we have…

• continuous univariate data for two groups, we typically compute a raw mean difference or a standardized difference – an effect size from the d-family,

• continuous bivariate data, we typically compute a correlation (from the r-family), or

• binary data (the patient lived or died, the student passed or failed), we typically compute an odds ratio, a risk ratio, or a risk difference.

04/21/23 7

d-d-Family: Family: Zero Effect SizeZero Effect Size

d-d-Family: Family: Zero Effect SizeZero Effect Size

ES = 0.00

Control Condition

Treatment Condition

Overlapping Distributions

04/21/23 8

d-d-Family: Family: Moderate Effect SizeModerate Effect Size

d-d-Family: Family: Moderate Effect SizeModerate Effect Size

Control Condition

Treatment Condition

ES = 0.40

04/21/23 9

d-d-Family: Family: Large Effect SizeLarge Effect Size

d-d-Family: Family: Large Effect SizeLarge Effect Size

Control Condition

Treatment Condition

ES = 0.85

Effect Size InterpretationEffect Size Interpretation

04/21/23 10

04/21/23 11

Research designs for d-Family Statistics

Research designs for d-Family Statistics

Independent Groups (posttest-only) EXP YPost

(Randomized or Non-randomized) CT YPost

One-group (pretest-posttest) YPre EXP YPost

Independent Groups (pre-post) YPre EXP YPost

(Randomized or Non-randomized) YPre CT YPost

EXP = Experimental Condition CT = Control ConditionEXP = Experimental Condition CT = Control Condition

04/21/23 12

Statistics for d-Family Effect Size Extraction

Statistics for d-Family Effect Size Extraction

Effect sizes can be extracted using the following reported statistics:

• Descriptive statistics (means, SDs, sample sizes) Preferred (by far).

• Exact test statistics (t-values, F-values, etc.)

• Exact probability values (p = .013, etc.)

• Approximate comparisons of p to α (p < .05, etc.) By far, the least exact.

04/21/23 13

ΔGlass =YExperimental − YControl

SDControl

dd--Family with Independent Groups Family with Independent Groups (Basic Equation)(Basic Equation)

dd--Family with Independent Groups Family with Independent Groups (Basic Equation)(Basic Equation)

dCohen =YExperimental −YControl

SDPooled

SDpooled =(SD2

E (nE −1)) + (SD2C (nC −1))

NTotal −2

Note: this equation is the same as adding two SSs and dividing by dfTotalNote: this equation is the same as adding two SSs and dividing by dfTotal

04/21/23

dd Family Statistics: Family Statistics: Means and Standard DeviationsMeans and Standard Deviations

dd Family Statistics: Family Statistics: Means and Standard DeviationsMeans and Standard Deviations

Procedure: 1) Calculate Pooled SD 2) Calculate d

04/21/23 15

Alternative Methods of Alternative Methods of ESES Extraction: Extraction: tt-values-values and and F-ratiosF-ratios

Alternative Methods of Alternative Methods of ESES Extraction: Extraction: tt-values-values and and F-ratiosF-ratios

Important Note: Report must indicate direction of the effect (+/–)

04/21/23

Alternative Methods of Alternative Methods of ESES Extraction: Extraction: Exact Exact pp-value-value

Alternative Methods of Alternative Methods of ESES Extraction: Extraction: Exact Exact pp-value-value

Study Reports: t(60) is sig. p = .01

Look up t-value for p = .01 (df = 60)

t = 2.66

Important Note: Report must indicate direction of the effect (+/–)

Alternative Methods of Alternative Methods of ESES Extraction: Extraction: pp < α < α

Alternative Methods of Alternative Methods of ESES Extraction: Extraction: pp < α < α

Study Reports: p < .05, nT = 31, nC = 31

Important Note: Report must indicate direction of the effect (+/–)

Estimate +t(60) = +2.00

04/21/23

Compared with 0.676, this ES is only 75% accurate.Compared with 0.676, this ES is only 75% accurate.

04/21/23 18

dd Family: Adjustment for Family: Adjustment for Small SamplesSmall Samples

dd Family: Adjustment for Family: Adjustment for Small SamplesSmall Samples

Recommendation: If there are small samples and large samples, convert all d-family statistics to g.

N = 60, g is 99% of d N = 40, g is 98% of d

N = 20, g is 96% of d N = 10, g is 90% of d

N = 60, g is 99% of d N = 40, g is 98% of d

N = 20, g is 96% of d N = 10, g is 90% of d

04/21/23 19

d-d-Family Statistics with dependent Family Statistics with dependent Groups (pre-post)Groups (pre-post)

d-d-Family Statistics with dependent Family Statistics with dependent Groups (pre-post)Groups (pre-post)

Relationship Between Effect Size and Pre-Post Correlation

Relationship Between Effect Size and Pre-Post Correlation

04/21/23 20

Means and SDs: d = 0.21

SD Change: d = 0.21, using r = 0.80

Co

rrel

atio

n

04/21/23 21

d-d-Family Statistics with Independent Family Statistics with Independent Groups Groups (pre-post)(pre-post)

d-d-Family Statistics with Independent Family Statistics with Independent Groups Groups (pre-post)(pre-post)

Calculate the pooled SD.

04/21/23 22

Calculating Standard ErrorCalculating Standard Error

Standard Error:

σ̂ g =131

+131

+0.6752

2(31+ 31)1−

34(31+ 31)−9

⎛⎝⎜

⎞⎠⎟

σ̂ g = 0.076 ⋅1−0.0126( )

σ̂ g = 0.266)⋅(0.987( )

σ̂ g =0.263

The standard error of g is an estimate of the “standard deviation” of the population, based on the sampling distribution of an infinite number of samples all with a given sample size. Smaller samples tend to have larger standard errors and larger samples have smaller standard errors.

The standard error of g is an estimate of the “standard deviation” of the population, based on the sampling distribution of an infinite number of samples all with a given sample size. Smaller samples tend to have larger standard errors and larger samples have smaller standard errors.

σ̂ g =1ne

+1nc

+g2

2(ne +nc)1−

34(ne +nc)−9

⎛

⎝⎜⎞

⎠⎟

04/21/23 23

95th% Confidence Interval95th% Confidence Interval

95th Confidence Interval

Upper:

Lower:

CIU =0.687 + (1.96 ⋅0.26)CIU =+1.97

CI L =0.687 −(1.96 ⋅0.26)CI L =+0.177

Conclusion: Confidence interval does not cross 0 (g falls within the 95th confidence interval).Conclusion: Confidence interval does not cross 0 (g falls within the 95th confidence interval).

The 95th Confidence Interval is the range within which it can be stated with reasonable confidence that the true population mean exists. As the standard error decreases (the sample size increases), the confidence interval decreases in width.

The 95th Confidence Interval is the range within which it can be stated with reasonable confidence that the true population mean exists. As the standard error decreases (the sample size increases), the confidence interval decreases in width.

CIUL =g±(1.96 ⋅σ̂ i )

04/21/23 24

Forest PlotForest Plot

04/21/23 25

Other Important StatisticsOther Important Statistics

σ̂ 2g =(σ̂ g)

2

σ̂ 2g =(0.262)2

σ̂ 2g =0.069

wi =1 σ̂ 2

wi =1 0.069wi =14.54

Weightedg =(wi )(gi ) =14.54 ⋅0.687 =9.99

Variance:

Inverse Variance (w):

Weighted g (g*w):

The variance is the standard error squared.The variance is the standard error squared.

The inverse variance (w) provides a weight that is proportional to the sample size. Larger samples are more heavily weighted than small samples.

The inverse variance (w) provides a weight that is proportional to the sample size. Larger samples are more heavily weighted than small samples.

Weighted g is the weight (w) times the value of g. It can be + or –, depending on the sign of g.Weighted g is the weight (w) times the value of g. It can be + or –, depending on the sign of g.

04/21/23 26

Hedges’ g

Standard Error (σ̂g) Variance

(σ̂2g) 95th Lo werLimit

95th Upper Limit z-Value p-Value

Weights (wi) Weighted

g (wi)(gi) 2.44 0.22 0.05 2.00 2.88 10.89 0.00 19.94 48.65 2.31 0.17 0.03 1.98 2.64 13.59 0.00 34.60 79.93 1.38 0.30 0.09 0.79 1.97 4.60 0.00 11.11 15.33 1.17 0.19 0.04 0.80 1.54 6.16 0.00 27.70 32.41 0.88 0.17 0.03 0.55 1.21 5.18 0.00 34.60 30.45 0.81 0.12 0.01 0.57 1.05 6.75 0.00 69.44 56.25 0.80 0.08 0.01 0.64 0.96 10.00 0.00 156.25 125.00 0.68 0.18 0.03 0.33 1.03 3.78 0.00 30.86 20.99 0.63 0.51 0.26 -0.37 1.63 1.24 0.22 3.84 2.42 0.60 0.13 0.02 0.35 0.85 4.62 0.00 59.17 35.50 0.58 0.29 0.08 0.01 1.15 2.00 0.05 11.89 6.90 0.32 0.11 0.01 0.10 0.54 2.91 0.00 82.64 26.45 0.25 0.08 0.01 0.09 0.41 3.13 0.00 156.25 39.06 0.24 0.20 0.04 -0.15 0.63 1.20 0.23 25.00 6.00 0.24 0.15 0.02 -0.05 0.53 1.60 0.11 44.44 10.67 0.19 0.12 0.01 -0.05 0.43 1.58 0.11 69.44 13.19 0.11 0.12 0.01 -0.13 0.35 0.92 0.36 69.44 7.64 0.09 0.08 0.01 -0.07 0.25 1.13 0.26 156.25 14.06 0.02 0.24 0.06 -0.45 0.49 0.08 0.93 17.36 0.35 0.02 0.17 0.03 -0.31 0.35 0.12 0.91 34.60 0.69 0.02 0.26 0.07 -0.49 0.53 0.08 0.94 14.79 0.30 -0.11 0.24 0.06 -0.58 0.36 -0.46 0.65 17.36 -1.91 -0.11 0.28 0.08 -0.66 0.44 -0.39 0.69 12.76 -1.40 -0.18 0.22 0.05 -0.61 0.25 -0.82 0.41 20.66 -3.72 -0.30 0.06 0.00 -0.42 -0.18 -5.00 0.00 277.78 -83.33 0.330 0.03 0.00 0.28 0.38 12.62 0.00 1458.21* 481.87*

g+ =wi∑

(wi )(gi )∑g+ =

481.871458.21

g+ =0.333

Average g (g+) is the sum of the weights divided by the sum of the weighted gs.

Average g (g+) is the sum of the weights divided by the sum of the weighted gs.

04/21/23 27

Selected ReferencesSelected ReferencesSelected ReferencesSelected References

Borenstein, M. Hedges, L.V., Higgins, J.P..,& Rothstein, H.R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley.

Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. [ERIC Document Reproduction Service No. ED 309 952].