Challenges in the Design and

Analysis of a Randomized, Phased

Implementation (Stepped-Wedge)

Study in Brazil

Design of Experiments in Healthcare Conference

Isaac Newton Institute, Cambridge

Lawrence H. Moulton

Professor, Departments of International Health and Biostatistics

Johns Hopkins Bloomberg School of Public Health

www.LarryMoulton.com

Co-Authors: A. Pacheco3, V. Saraceni1, J. Golub4 , S. Cohn4, S. Cavalcante1,3,

B. Durovni1,2, R. Chaisson4.1Rio de Janeiro City Health Secretariat, Rio de Janeiro, Brazil, 2Federal University of Rio de Janeiro, Rio de Janeiro, Brazil, 3Fiocruz, Rio de Janeiro, Brazil, 4Johns Hopkins School of Medicine, Baltimore, United States © 2011 Lawrence H. Moulton

2

Outline

• Description: Brazil TB clinic-based study

• Analytic approach

• Statistical features of design

– Power: stepped wedge, unequal cluster size,

within-clinic correlation

– Randomization

• Special analytic features and results

3

Impact of Widespread Use of TB Preventive Therapy for Patients

with Access to Antiretroviral Therapy in Rio de Janeiro, Brazil:

A Phased Implementation Trial

4

Rio de Janeiro • Betina Durovni• Solange Cavalcante• Valeria Saraceni• Antonio Pacheco• Rita Ferreira• Giselle Israel• Vitoria Vellozo• Lilian Lauria

JHU• Richard Chaisson• Jonathan Golub• Larry Moulton• Silvia Cohn• Bonnie King• Anne Efron• Susan Dorman

THRio Study Team

Funding: Bill and Melinda Gates Foundation, NIAID, Fogarty International Center

5

Key features:

• Want to perform a strengthening of health services intervention, getting clinic personnel to do TB testing, and put TST positives on prophylactic isoniazid regimen

• 29 clinics, HIV+ clientele, most on HAART; clinics enter the intervention status two at a time every two months

• For samples size/power, need to account for:

--loss of efficiency due to stepped-wedge design

--loss of efficiency due to within-clinic correlation

--variable size of clinics

• Need to constrain the randomized order of entry, to avoid serious covariate imbalances over time

6

THRio Study Timeline

Stepped-Wedge Design

Intervention and

Follow-up Period

(for all clinics)

Sep 05 Jan 08

48 60

Dec 09

7

First—Need to Consider Analytic Approach

• Study will take place over 2.5 years, and there may be a strong

temporal trend in TB incidence

• Perfectly control for calendar time by comparing, ON EACH DAY, TB

incidence in clinics that are still in control status to incidence in clinics

that are in intervention status

• Assume Poisson process with time-varying intensity:

)exp( ititit zn

where itn is the person-days of exposure in the ith clinic on the tth day,

t represents the effect of the tth day

is the log rate ratio comparing those in the intervention status ( =1)iz

to those in control status ( =0)iz

8

Analytic Approach (continued)

• Condition on each day’s risk set; form partial

likelihood, comparing covariates of incident

cases to those of the other patients: eliminates

• Use gamma frailty model to account for within-

clinic correlation over time

t

9

iTd

iTY

idiYand

number of incident cases in the ith month in the intervention clinics and

number of persons at risk in those clinics, and

the cases and persons in both Treatment and Comparison clinics

in the ith month. Then the log-rank test statistic is given by:

28

1

28

1

[ ( / )]

11

i i

i i

T T i ii

T T i iii

i i i

d Y d YZ

Y Y Y dd

Y Y Y

Accounting for Wedging in the Power Calculations

Use log-rank weights to approximate partial likelihood

weighting of events:

10

EZ

SWZ

Accounting for Wedging, Continued

Inflation Factors for Stepped-Wedge Design

Calculate Z in two ways:

Assumes equal ratio 14 Treatment: 14 Comparison

clinics throughout the study

Assumes stepped-wedge design, 2 clinics entering

Intervention status every 2 months

Excel spreadsheet gives: / 1.2E SWZ Z

(virtually the same under null and alternative hypotheses)

So, for 80% power, instead of using 0.8416, use 0.8426 x 1.2 = 1.01

(nominal 93.8% power)

For Type I error of 5%, use 2.352 instead of 1.96 (nominal 1.9% size)

(=1.44 add’l

deff factor)

11

Estimation of k, the Coefficient of Variation

• See Hayes & Bennett 1999 Int J Epidemiology

• In each of 10 clinics, sampled 24 patient charts, obtained

historical information on TB incidence

• Calculation result based on random effects model: k = 0.10

• Seems too good to be true; if the true rate is 3.6/100 pyrs,

expect 95% of clinic rates to be between 2.9 and 4.3—so used

several larger values for k, and calculated power as a function of

them (next slide).

• Final adjustment— need an average person-years per

cluster for the sample size formula; clinic sizes vary widely, so

use harmonic mean of the anticipated person-years in the

clinics:

instead of the mean 595, used y=346.4 pyrs/cluster

12

Final Brazil Power Calculations

Fixed k (CV), error levels, and control rate, then solved for

intervention rate and hence effectiveness from the

formula:

CV=0.15 CV=0.20 CV=0.25

Control, Rate/100 pyrs 3.65 3.65 3.65

Intervention, Rate/100 pyrs 2.29 2.20 2.10

Effectiveness 37% 40% 42%

22222

2/ )/()](/)[()(1 TCTCTC kyzzN

where N is the number of clusters in each trial arm, y is the pyrs in each cluster

13

Constraining the Brazil Randomization

of Entry Order Into Intervention Status

There are

= 5.4 × 1026

distinct possible orderings of the clinics.

But we want to achieve close balance on clinic-months spent in control and intervention status with respect to a number of covariates: administrative region, existence of a DOTS program, mean CD4, number and proportion of patients on TB therapy, and education.

Use the approach of Raab (Stat Med 2001) and Moulton (Clin Trials 2004) to greatly reduce the above number of orderings to those that meet certain criteria with respect to the above variables.

29 27 25 23 21 19 17 15 13 11 9 7 5 3 1

2 2 2 2 2 2 2 2 2 2 2 2 2 2 1

14

Form of Proportional Constraints

In particular, for each jth covariate xij of the ith clinic, for

a given entry month ti, t = 1, 3, 5, … , 29, and a given

proportional covariate-specific tolerance cj, we

require that a given entry order satisfies:

i.e. the sum of the covariate values weighted by the

number of months in intervention status must be

within cj ×100 percent of that for control status.

Note: in this formulation, the last-entered clinic does not

contribute to trial results.

29 29

1 129 29

(1/(1 )) (28 ( 1)) / ( 1) (1 )

i i

j i ij i ij j

i it t

c t x t x c

15

Clinic-Level Covariates and

Constraints• Variable Meaning cj

• DOTS existence of a DOTS program 0.1

• AP1 dummy variables for AP unit 0.2

• AP2 0.2

• AP3 0.2

• AP4 0.4

• AP5 0.4

• MNCD4 mean CD4 count 0.1

• NPTS number of current patients 0.1

• PCTINF proportion being treated for TB 0.1

• EDLT4 proportion < 4 yrs’ education 0.1

16

City of Rio de Janeiro

Health units - CREATE:

6 units

AP 1.0

AP 2.1

AP 2.2

AP 4.0

AP 5.1AP 5.2AP 5.3

AP 3.3

AP 3.1AP 3.2

< 100 patients

1 units

100 – 500 patients

500 - 1000 patients

> 1000 patients

1 unit

3 units

1 unit

8

4

3, 7, 9

112

10

5

4

2 34

6, 10, 134, 5, 8

3

5 59

6

13

17

4

7

In purple—TB incidence rates

17

Sequence Generation

• We want to select, at random, one sequence from among

all those that satisfy these criteria. If there were a

reasonable number of total sequences (not 10**26) then

we would check each one to see which meet the criteria,

then randomly choose one of the eligible sequences.

• Equivalently, we could randomly generate sequences,

testing each one until we get an acceptable one.

• What we actually did was generate sequences until we

had 1000 that were acceptable; then carried out the final

selection of one of these in a quasi-public randomization

ceremony.

• Method for selection of the final one out of the 1000

eligible:

Specified that we would take the last three digits of the

26 February 2005 Loteria Federal to select it!

18

Design Validity

• Generally: A design is valid if each pair of randomization units has the same probability of being allocated the same treatment (RA Bailey 1983 Biometrika)

• In our one-way crossover design—we want each pair of units to have (about) the same probability of entering intervention status at the same time

• Simulations show it takes a high degree of departure from uniformity of pairing probabilities, coupled with high correlation of affected randomization units, to significantly impact Type I error (Moulton, 2004 Clinical Trials)

19

Tradeoff

Invoke minimization (Pocock & Simon,1975 Bics) in a simultaneous fashion, randomly selecting one of the sequence allocations from among those that have good balance.

* Want to constrain enough so that ensure marginal balance on relevant covariates—like minimization technique for sequential trials except do it all up front

• Don’t want to constrain so much that validity (independence) is lost, setting one up for criticism

20

Validity Check

• Randomly generated sequences until had 5,000 acceptable designs. This took 8.96×10**7 random attempts; that means we estimate that only about 5000/89,600,000, or 0.006% of the total possible designs are acceptable.

• No duplicates among the 5000, although there had been 471 duplicates in a previous run that had the precision for AP4 and AP5 set at 0.3 instead of 0.4

21

Validity Check (continued)

The following matrix gives the number of times that any given pair of clinics entered the trial at the same time point. For example, clinics 3 and 7 entered in the same time point 132 times out of 5,000. All 5,000 x 14 = 70,000 entered pairs are ―folded‖ into the upper diagonal; there are 406 upper diagonal entries, and we would expect about 70,000/406 = 172 in each entry if there were no constraints, with a 95% confidence interval of ± 26.

Although there is an almost three-fold ratio of the highest to the lowest entries, all pairs of clinics had multiple opportunities to enter together, and no pairs never entered together.

22

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 1 0.000 104. 117. 132. 124. 114. 131. 136. 121. 148. 138. 149. 151. 172. 163. 163. 197. 172. 168. 210. 22 0.000 0.000 117. 99.0 127. 110. 128. 137. 120. 139. 153. 150. 178. 151. 177. 166. 198. 181. 203. 183. 23 0.000 0.000 0.000 127. 114. 116. 132. 126. 155. 140. 168. 135. 168. 150. 167. 179. 184. 193. 180. 174. 196. 197. 201. 236. 228. 205. 232. 234 0.000 0.000 0.000 0.000 138. 122. 137. 110. 134. 146. 149. 170. 154. 169. 173. 166. 188. 200. 203. 193. 25 0.000 0.000 0.000 0.000 0.000 140. 120. 140. 147. 146. 165. 159. 191. 187. 183. 177. 169. 178. 188. 203. 16 0.000 0.000 0.000 0.000 0.000 0.000 139. 128. 162. 155. 168. 156. 150. 170. 185. 184. 193. 180. 188. 171. 27 0.000 0.000 0.000 0.000 0.000 0.000 0.000 150. 151. 168. 143. 181. 176. 186. 157. 171. 191. 175. 203. 181. 18 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 168. 156. 171. 141. 172. 182. 179. 190. 162. 220. 191. 200. 19 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 161. 165. 193. 172. 166. 159. 153. 192. 175. 183. 179. 110 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 170. 157. 201. 139. 190. 166. 170. 170. 181. 170. 111 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 168. 177. 170. 155. 154. 165. 158. 160. 191. 112 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 142. 190. 178. 169. 171. 157. 175. 163. 113 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 162. 160. 184. 166. 159. 179. 180. 114 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 176. 165. 168. 166. 172. 175. 115 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 159. 164. 183. 183. 186. 116 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 144. 168. 193. 209. 117 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 178. 164. 218. 118 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 176. 179. 119 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 178. 120 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 121 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.22 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.23 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.24 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.25 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.26 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.27 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.28 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.29 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.

“Validity Matrix”

23

Clinic-Level Covariates and Constraints

• Variable Meaning cj Actual

• DOTS existence of a DOTS program 0.1 0.03

• AP1 dummy variables for AP unit 0.2 0.10

• AP2 0.2 0.04

• AP3 0.2 0.10

• AP4 0.4 0.09

• AP5 0.4 0.21

• MNCD4 mean CD4 count 0.1 0.08

• NPTS number of current patients 0.1 0.07

• PCTINF proportion being treated for TB 0.1 0.07

• EDLT4 proportion < 4 yrs’ education 0.1 0.07

24

Final Comments on Brazil Design

• ―…we make no mockery of honest ad hockery‖ --I.J. Good

• Had to compensate for many factors in the sample size calculation: unequal group sizes, correlation across time, time-varying ratios of trial arm sizes (due to phased intervention)

• Used a rather simple, although greatly restraining, set of constraints for the randomization; other weighting systems might be of interest

• Phased implementation design both suitable and logistically feasible

25

Primary Analysis

• Based on when patients had the

opportunity for the intervention—when

they showed up in a clinic that is in the

intervention phase

• Mimics long-term effectiveness of the

intervention.

Statistical Analysis

• Cox model with clinic-level random effects

• Main covariate of interest:

– For each patient, binary time-varying

covariate tracks switch from control to

intervention phase for that patient

• Simulations showed that this gives same

result as time-varying covariate that is the

proportion of individuals in a clinic who

have shown up during intervention phase

– (as per basic idea in Hussey & Hughes, Contemp Clin Trials 2007)

Slight Problems

• Don’t have dates for all clinic visits—but

will use any information indicating there

was a meaningful clinic contact (e.g. CD4

result, HAART begun,…)

• Will not count a TB case unless the patient

has been in the clinic at least 60 days, so

that prevalent TB is not included—same

for deaths.

Which TB Cases Count for

What?

• Control Phase

– If TB is diagnosed before a patient’s first clinic

visit when the clinic entered intervention

phase

• Intervention Phase

– If TB is diagnosed after a patient’s first clinic

visit during intervention phase

30

Cox Model Looks at Cross-Sectional

Data Slices

1/2

3/4

5/6

7/8

9/1

0

2

9

Clin

ic e

ntr

y to inte

rvention p

eriod

Control

phase

Intervention

phase

Follow-up

period

1 3 5 7 9 29 36 42

Month

Note: still making vertical comparisons during final follow-up period,

as patients are still showing up for their first time in intervention phase

(Perfectly accounts for secular trends)

Analysis at Individual Level

• Clinics are handled as Intent-to-Treat

• Individual patients do not count in the

―treat‖ part until they show up at a clinic

which has already received the

intervention.

• Patients tracked using survival analysis

setup; calendar timeline is used.

• Dynamic cohort—patients entering all the

time, transferring, dying…

Sample Primary Analysis Timeline:

One Patient in One Clinic

32

A. Pacheco

02

46

8

Inte

rven

tion

TB

Incid

en

ce /1

00

pyrs

0 2 4 6 8Control TB Incidence /100 pyrs

Circles below line indicate a clinics with lower rates during intervention phase.

(Area is proportional to the inverse variance of the rate difference)

36

Pure ―Vertical‖ TB Rates Analysis

37

0

1

0

1

0

1

0

1

0

101

01

0

1

01

0

1

0

1

0

1

0

1

0

1

1

0

.02

.04

.06

rate

s

0 5 10 15entryinterval

TB incidence rates per person-year by each two-month interval.

1 is for intervention, 0 is for control phase units. The last interval is

actually from when all units are in intervention until Aug 2009.

Clinic Rate Differences by Order of Entry into Intervention

--another check on secular trend effects

38

Data from

last 5 clinics

to enter

Intervention

phase

39

Courtesy V.Saraceni & K.Lin 40

0

1x10-5

2x10-5

3x10-5

4x10-5

5x10-5

0 200 400 600

Follow-up Time

Hazard

Ratio

Courtesy V.Saraceni & K.Lin 41

42

A. Pacheco

THRio Results

TB cases, total contribution time, incidence per 100pyrs

Control Phase Intervention Phase

Cases 221 254

Person years 16,834 23,126

Rate/100pyrs 1.31 1.10

43

TB/Death cases, total contribution time, incidence per 100pyrs

Control Phase Intervention Phase

Cases 617 696

Person years 16,834 23,126

Rate/100pyrs 3.67 3.01

Conclusions

44

Stepped wedge approach can be fraught

with complications: -- Loss of power

-- Balancing randomization

-- Handling lag in implementation of

intervention: neither vertical nor

horizontal, but diagonal, analysis may be needed.

These all require careful handling and cross-checks.

Other potential problems: -- Secular trends may still linger

-- Aging of cohort confounded with

one-way crossover

THRio had political and logistical reasons for this

design; but better to use a parallel design if

possible!