The Multi-Site RCT• A fleet of RCTs!• Each conducted in a different social setting• Since 2002, IES has funded 175 randomized

trials;• Vast majority are multi-site trials (Spybrook, 2013)

Opportunities

• Can assess generalizability of impact– Gauge variation in effect– Model heterogeneity– Does the intervention equalize outcomes?

But we must formulate appropriate analyses.This is trickier than it might seem!

Some Recent MS TrialsStudy Levels Assigned

UnitsSites Fixed or

Random sites

National Head Start Eval.

2 Children 300+ Program Sites

Random

Moving to Opportunity

2 Families 5 cities Fixed

BostonCharter School Lotteries

2 Children Lottery pools Random

Tennessee STAR

3 Teachers Schools Random

Double-Dose Algebra

2 Children Schools Random

Theoretical Model

bjj

bjjj

jjj

Tijijijjjij

buCovbbB

uuU

rrTBUY

00

2

200000

20

),(),0(~,

),0(~,SitesBetween

),0(~,SitesWithin

4. Unbalanced Designs and Targets of Inference

Two kinds of imbalance

• Unequal n per site• Unequal propensity score (probability of

assignment to the treatment group)

Targets of Inference

**

**

**

*

*

*

10

100

1

2

1

2

11

01

0*0

2

1*

2

1*

/))((

/)(

/

Personsof populationa toGeneralize

))((1

)(1

1

Sites of ona Populati toGeneralize

J

jjj

J

jjjpersonsb

J

jj

J

jjjpersonsb

J

jj

J

jjjpersons

j

J

jjsitesb

J

jjsitesb

J

jjsites

NBUN

NBN

NBN

BUJ

BJ

BJ

Identification

lotteriesschoolchartere.g.,

).|(),.|(

)|()|()|(

21

0

00

00

jjjj

ijjijijjijijij

ijijjjijij

TbETuEaboutWorry

TbETTuETTYE

eTbuTYlevelandlevelCombine

Parameters to be estimated

Properties

a)Fixed effects β Biased if precision related to B

b)Centering HLM B,Var(B) Bias in β if precision is related to B (less so than fixed effects )

c) Control propensityScore

B,Var(B)?? Similar to Centering for β,

d)Weighting (“IPTW”) with HLM

B,Var(B),Cov(B,U0) Removes bias (but may be imprecise!)

a) Fixed Effects Model

jjj

J

jjjj

J

jjjjj

ijjijij

YYB

where

TTn

BTTn

eTY

01

1

1

ˆ

).1(.

ˆ).1(.ˆ

Potential Bias if heterogeneous impact,and unbalanced design

averageweightedprecisionA

TTn

BTTnETBE

j

J

jjj

jj

J

jjj

).1(.

).1(.),|ˆ(

1

1

b) Centering the Predictor AND the Outcome (“FIRC” Model)

),0(~

).(

2bjjj

ijjijjjij

NbbB

eTTBYY

Robustness (compared to fixed effects)

.).1(.,,

).1(.

).1(.),,,|ˆ(

2

1

12

2

1

12

2

22

decreasesTTnonrelianceincreasesincreasesityheterogeneAs

TTn

BTTn

ETE

jjj

b

J

j jjjb

j

J

j jjjb

b

Estimation of heterogeneity

.).1(.,,

).1(.

])ˆ[().1(.

2

1

2)1(2

2)1(

2)1(

1

2)1(2

2)1(

2)(

decreasesTTnonrelianceincreasesincreasesityheterogeneAs

TTn

VBTTn

jjj

b

J

j jjj

mm

b

jm

j

J

j jjj

mm

bm

b

c) Control the Propensity Score

ijijjjij eTBUYLevel

0

1

2

000

,

.2

b

jj

jjj

ofestimategoodnoeffecttreatmentaverageforcenteringtosimilarVery

bBuTU

Level

Variance Estimation

)1(

)1(1

)1(00

)1()1(

1

1)1()1(1)1()1(

1

1

1)1()1(1)1()1()(

ˆˆ

)(*

)(

mj

jmm

jj

Tmj

mj

J

j

mj

mmj

m

J

j

mj

mmj

mm

BTU

d

vec

vec

ddVτVτ

VτVττ

Summary so far

Four commonly used options* limit what we can estimate * produce inconsistent estimates of

what we can estimate.

6.HLM with Inverse Probability of Treatment Weighting

• IPTW• Embedding within HLM• Always produces consistent estimates• May be quite imprecise

Inverse Probability of Treatment Weights (Robins and Greenland, 2000)

jij

jijij

jijij

jijij

TTT

TTTw

soTTwTIf

TTwTIf

11)1(

)1/()1(,0

/,1

Weighting Schemes for HLM

Level-2 weight

Level-1 weight Result

Weights site-specific estimates of impact by

Weights site-specific estimates equally

The picture can't be displayed.

j

ij

j

ij

TTT

TTT

.1)1)(1(

.

j

ij

j

ij

j TTT

TTT

nn

.1)1)(1(

.1

jn

nn j

Estimation via Weighted log Likelihood(weighted complete-data log likelihood)

,

|)|2log(

)2log(/)(

:

)](ln[)]|(ln[)],(ln[

),0(~),,0(~,

12

1 1

1

122

1

1 1

22221

1 1

2

JwNwwhere

Juuw

NuZXYwl

weightingAfter

upuYfuYh

NeNueuZXY

J

jj

J

j

n

iij

J

jj

Tjj

J

j

n

irj

Tij

Tijijij

J

j

n

iijw

ijjijjTij

Tijj

j

jj

Two examples of weighting schemes

Equally weight people Equally weight sitesLevel-1 weight Level-1 weight

Level-2 weight Level-2 weight1

jij

jij T

TTTTT

11)1(

JBJ

jj

1

/ˆ̂

J

jj

J

jjj nBn

111 /ˆ̂

nn j /

jj

J

jnn

b VBJ j

2

1

12 )ˆˆ(ˆ

J

jjjb VBJ

1

212 )ˆˆ(ˆ

jjj

J

jnn

b CUBJ j

)ˆˆ)(ˆˆ(ˆ 001

10

j

ijj

ijnn

TTT

TTT

j 11)1(

J

jjjjb CUBJ

100

10 )ˆˆ)(ˆˆ(ˆ

7. DiscussionA Class of Estimators…

We are approximating various balanced designs by maximizing the weighted log likelihood

This gives us a family of method-of-moments estimators

Hence no reliance on normality or homoscedasticity

May be quite imprecise

Extensions• Extend easily to multi-site clustered

randomized designs• Balance propensity within covariate classes

in randomized studies• Extend to observational studies• Extend to weighted mediation models (Hong

et al, 2012)• Application to local average treatment effects

(Raudenbush, Nomi, and Reardon, 2016)