The Power of Pairing in Cluster Randomized Block Designs: A Monte Carlo Simulation

The Power of Pairing in Cluster Randomized Block Designs: A Monte Carlo Simulation

Nianbo Dong & Mark Lipsey

03/04/2010

Power Analysis for Group-Randomized Experiments

1. Two Big Design Families (Kirk, 1995)

1) Hierarchical Design

2) Cluster Randomized Block Design

2. Three Ways to do Power Analysis

1) Software, e.g., Optimal Design 2.0 (Spybrook, Raudenbush, Congdon, & Martinez, 2009)

2) MDES formula (Bloom, 2006; Schochet, 2008)

3) Power table using operational effect size(Hedges, 2009; Konstantopoulos, 2009)

2

Matched-Pair Cluster-Randomized Design (1)

1. Advantages1) Avoiding bad randomization, 2) Face validity

2. Cost: Loss of degree of freedom

3

3. But, the gain in predictive power may outweigh the loss of degrees of freedom (Billewicz, 1965; Bloom, 2007; Hedges, 2009; Martin, Diehr, Perrin, & Koepsell, 1993; Raudenbush, Martinez, & Spybrook, 2007)

4. Break-even point using MDES for fixed pair effect model (Bloom, 2007)


5. MDES Comparison

4

MDES for 2-level hierarchical design, w/o covariance adjustment

(Bloom, 2006)

MDES for 3-level matched-pair cluster-randomized design, random pair effect model w/o covariance adjustment, VC

JnJ

RMMDES B

J）-1（4)1(4 *2*

12

M (x) = / 2t + 1t

J: # of clusters; n: average # of individuals

* : ICC for hierarchical design;

: Pair-level ICC for matched-pair cluster-randomized design

3

(SE from Raudenbush & Liu, 2000)

JnJMMDES J

）-1（44 **

2

*32

BR


5. MDES Comparison

5

MDES for 2-level hierarchical design, w/o covariance adjustment

(Bloom, 2006)

MDES for 3-level matched-pair cluster-randomized design, random pair effect model w/o covariance adjustment, VC

JnJ

RMMDES B

J）-1（4)1(4 *2*

12

M (x) = / 2t + 1t

J: # of clusters; n: average # of individuals

* : ICC for hierarchical design;

: Pair-level ICC for matched-pair cluster-randomized design

3

(SE from Raudenbush & Liu, 2000)

JnJMMDES J

）-1（44 **

2

*32

BR

6

W/o cov-adjRandom pair

effect

With cov-adj

W/o cov-adjFixed pair effect

With cov-adj

Ignoring pair With cov-adj

YesW/o cov-adj

Random pair effect

With cov-adj


With cov-adj


No With cov-adj

Able to make close matching

Pre-randomization

Matching Unable to make close matching

L1 L2 L3 L4

Conceptual Model: Options & Decisions

7


effect

With cov-adj


With cov-adj


YesW/o cov-adj

Random pair effect

With cov-adj


With cov-adj


No With cov-adj

Able to make close matching

Pre-randomization


L1 L2 L3 L4

Conceptual Model: Options & Decisions

Research Questions

The Overall QuestionAre there design and analysis options other than increasing the sample size that might keep pre-randomization matching from degrading power relative to the analogous unmatched design?

8

Four Sub-Questions

9

How much difference does it make to statistical power:

1. If we are able to make close matches or unable to do so?2. If we are treating the pairwise blocks as fixed effects vs. random effects?3. If we ignore the pairwise blocking entirely (and does this compromise the Type I error rate)?4. If we also use the blocking variable as a covariate in the analysis?

Simulation: Parameter Combinations

10

R2 = 0.2

R2 = 0.5

R2 = 0.2

R2 = 0.5

R2 = 0.2

R2 = 0.5

10

30

60

10

30

60

Note . n =20

ESV=0.01

0.2

J

0.1

ICC

ES=0, 0.3, 0.5

ESV=0 ESV=0.05


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj

.75/.13Ignoring pair With cov-adj

Yes


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


No With cov-adj

Pre-randomization


.74/.14

.66/.08

.71/.08

.66/.10

.70/.08Able to make close matching

.71/.12

.63/.08

.69/.12

.75/.13

.69/.12

.71/.08

.75/.13

.58/.10

.77/.13

.55/.09

.73/.08

.76/.13

.74/.14

.69/.08

Results: Median and Minimum Power on Each Branch

11

Median/MinSE =.009-.016


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


Yes


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


No With cov-adj

Pre-randomization


.74/.14

.66/.08

.71/.08

.66/.10


.71/.12

.63/.08

.69/.12

.75/.13

.69/.12

.71/.08

.75/.13

.58/.10

.77/.13

.55/.09

.73/.08

.76/.13

.74/.14

.69/.08


12



effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


Yes


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


No With cov-adj

Pre-randomization


.74/.14

.66/.08

.71/.08

.66/.10


.71/.12

.63/.08

.69/.12

.75/.13

.69/.12

.71/.08

.75/.13

.58/.10

.77/.13

.55/.09

.73/.08

.76/.13

.74/.14

.69/.08


13



effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


Yes


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


No With cov-adj

Pre-randomization


.74/.14

.66/.08

.71/.08

.66/.10


.71/.12

.63/.08

.69/.12

.75/.13

.69/.12

.71/.08

.75/.13

.58/.10

.77/.13

.55/.09

.73/.08

.76/.13

.74/.14

.69/.08


14



effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


Yes


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


No With cov-adj

Pre-randomization


.74/.14

.66/.08

.71/.08

.66/.10


.71/.12

.63/.08

.69/.12

.75/.13

.69/.12

.71/.08

.75/.13

.58/.10

.77/.13

.55/.09

.73/.08

.76/.13

.74/.14

.69/.08


15



effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


Yes


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


No With cov-adj

Pre-randomization


.74/.14

.66/.08

.71/.08

.66/.10


.71/.12

.63/.08

.69/.12

.75/.13

.69/.12

.71/.08

.75/.13

.58/.10

.77/.13

.55/.09

.73/.08

.76/.13

.74/.14

.69/.08


16



effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


Yes


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


No With cov-adj

Pre-randomization


.74/.14

.66/.08

.71/.08

.66/.10


.71/.12

.63/.08

.69/.12

.75/.13

.69/.12

.71/.08

.75/.13

.58/.10

.77/.13

.55/.09

.73/.08

.76/.13

.74/.14

.69/.08


17



effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


Yes


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


No With cov-adj

Pre-randomization


.74/.14

.66/.08

.71/.08

.66/.10


.71/.12

.63/.08

.69/.12

.75/.13

.69/.12

.71/.08

.75/.13

.58/.10

.77/.13

.55/.09

.73/.08

.76/.13

.74/.14

.69/.08


18



effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


Yes


effect

With cov-adj

W/o cov-adj

Fixed pair effect

With cov-adj


No With cov-adj

Pre-randomization


.74/.14

.66/.08

.71/.08

.66/.10


.71/.12

.63/.08

.69/.12

.75/.13

.69/.12

.71/.08

.75/.13

.58/.10

.77/.13

.55/.09

.73/.08

.76/.13

.74/.14

.69/.08


19


Conclusions•The advantages of pre-randomization matching do not have to come at the cost of reduced power– even when the matching is not very good.

20

•The most important technique for maintaining power is to also use the matching variable as a covariate.

•The random effects model does not necessarily have less power than the fixed effects alternative.

•Ignoring the pairwise blocking variable in the analysis, though not faithful to the actual design used, does not appear to cause problems with either the Type I or Type II error rate. (Consistent with Diehr, Martin, Koepsell, & Cheadle, 1995; Lynn & McCulloch, 1992; Proschan, 1996)

Thanks

21

22

Statistical Power from Simulation (J=30, n=20, ES=.3, 2

1R = 22R =.5, = .05 for two-tailed test)

Combination ESV = 0 ESV = .05

ICC = .1 ICC = .2 ICC = .1 ICC = .2 1 .81 .61 .70 .54 2 .69 .53 .58 .46 3 .78 .57 .64 .47 4 .70 .53 .58 .46 5 .85 .63 .71 .55 6 .51 .36 .45 .34 7 .78 .58 .64 .49 8 .56 .39 .49 .37 9 .82 .63 .70 .55

10 .84 .62 .69 .53 11 .81 .63 .69 .54

Appendix 1:

23

Appendix 2:Statistical Power from Simulation (J=10, n=20, ES=.3, 2

1R = 22R =.2, = .05 for two-tailed test)

Combination ESV = 0 ESV = .05

ICC = .1 ICC = .2 ICC = .1 ICC = .2 1 .19 .14 .19 .14 2 .15 .13 .15 .12 3 .09 .09 .11 .10 4 .15 .12 .15 .13 5 .18 .14 .16 .14 6 .14 .12 .13 .09 7 .10 .11 .10 .08 8 .16 .13 .14 .10 9 .17 .15 .15 .13

10 .22 .15 .20 .15 11 .22 .15 .18 .13

Appendix 3: MDES for Matched-Pair Cluster-Randomized Design

24

Random Pair Effect Model - Unconditional model (without covariance adjustment)

ijkjkkjkkjktijk euTREATTREATy )()(0

),0(~ 2eijk Ne , ),0(~ 2

2Nu jk ,

22

23

0

0~

Nk

k

Given the variance component (VC) variance-covariance matrix

2 2 2 2 2 2 2 2 2 2 2 22 3 2 2 3

1 1 1 1*

2 4 2 4Total e e Total

ESV = 2 2 22 / Total

Recall that in the hierarchical design: 2Total = 2 2 21

4e .

Thus, we will have: 2 2 2 22 3 2

1

2 , i.e., 2 2 2 2 2

2 3

1*

2 Total

i.e., 22 3

1

2

MDES for matched-pair cluster-randomized design: without covariate:

JnJMMDES J

)1(424

*2

2

12

i.e.,

JnJ

MMDES J

)1(44 *3

*

12

JnJ

MMDES J

)1(4R14 *2B

*

12

, where *32

BR

.

Documents

The Power of Pairing in Cluster Randomized Block Designs: A Monte Carlo Simulation