Upload
dangtram
View
223
Download
0
Embed Size (px)
Citation preview
Designs and Types of Evidence for Dissemination and
Implementation Research C Hendricks Brown
University of Miami Miller School of Medicine
Director, Center for Prevention Implementation Methodology
Director, Prevention Science and Methodology Group Director, Social systems Informatics
chbrown@med miami edu
1
Acknowledgements
• Center for Prevention Implementation Methodology (Ce-PIM) for Drug Abuse and HIV Sexual Risk Behavior (NIDA) PI C Hendricks Brown
• Advanced Center to Improve Pediatric Mental Health Care (P30 MH074678) aka Implementation Methods Research Group (IMRG) PI John Landsverk
• Scaling up MTFC in California NIMH: R01MH076158, PI Patti Chamberlain
• Methods for MH/DA Prevention and Early Intervention, NIMH/NIDA R01-MH040859, PI C Hendricks Brown
• Collaborative Synthesis of Adolescent Depression Trials, NIMH R01-MH040859-
22 PI C Hendricks Brown • NIDA R01 DA019984 A Follow-up of Classroom Services to Prevent Drug Use,
PI Jeanne Poduska,
• NIDA R21DA024370, Scaling-up Prevention Services for Early Drug Abuse Risk in School Systems, PI Jeanne Poduska
2
Acknowledgements Co-Authors John Landsverk, Rady Children’s Hospital Patricia Chamberlain, Oregon Social Learning
Center Lawrence Palinkas, USC Wei Wang, University of South Florida Lisa Saldana, Center for Research to Practice Jeremy Goldhaber-Fiebert, Stanford Sarah Horwitz, Stanford
3 NIH Training Institute for D&I Research Ce-PIM
Outline I. Factors Affecting Design of the D & I Study II. Three Dimensions for Typing Study
Designs for Implementation Research Illustrations - Microsimulation
III. Increasing the “Evidence” in Implementation Research
Roll-Out Randomized Implementation Trials Example of the CAL-OH Randomized
Implementation Trial IV. Summary
4 NIH Training Institute for D&I Research Ce-PIM
Where We Are in the Science of Implementation
• Just beginning • All of this is in flux • Role of Methodology Similar to That for
Prevention Science 20 Years Ago IOM Preventing Mental, Emotional, & Behav Disorders 2009
5
I. Factors Affecting Design of the D & I Study
• Mutually Agreed Upon Research Question
6
System to Support Adoption
1. What designs are used in implementation studies?
Changing Research Questions Effectiveness vs Implementation
7
Intervention Intervention
System to Support Adoption
NIH Training Institute for D&I Research Ce-PIM
General Influences on the Design of a Study
NIH Training Institute for D&I Research Ce-PIM 8
Determine Research Question(s) Research Team
Institutional/Community other Stakeholders & Partners
Intervention Conditions / Implementation Strategies
Population
Measures and Follow-up/ Assessment Procedures
Sampling, Sample Size and Enrolling
Assignment to Intervention Condition / Implementation Strategies
Ethics/Human Subjects/Values Financial/Logistic
s
Policy Makers, Funders
Handy Terminology
• Implementation (or Dissemination) Strategy – system-level program:
Glisson’s ARC Hawkins Communities that Care Community Development Team (CiMH) • Intervention – evidence-based program
9
Illustrative Classes of Research Questions for D&I Research and Common Types of Designs
1. What System Level Factors Impede or Facilitate implementation?
Descriptive Epidemiologic - Cross-Sectional Do More Evidence-Based Organizations
Adopt? 2. What implementation strategies lead to faster uptake, successful adaptation and sustained effects on target population? Interventional (Testing) -- Experimental 3. How does a Strategy Affect Implementation? Etiologic (Mediational) -- Multilevel Longitudinal
10
II. Three Useful Dimensions for Typing of Studies for D & I Research
NIH Training Institute for D&I Research Ce-PIM 11
How “Real World”
How much control over Implementation
Assignment What Levels Are Measured?
II. A Three Useful Dimensions for Typing of Studies for D & I Research
NIH Training Institute for D&I Research Ce-PIM 12
How “Real World” In Vivo -- full implementation as it takes
place In Vitro – implementation facilitated by
researchers In Silico –studies involving simulation
Examples • In Vivo : CMHS/SAMHSA Implementation
of the Good Behavior Game in 24 School Districts
Very Modest Federal Funding Districts Responsible for TA • In Vitro : IES/NIDA Funded Professional
Development Trial in Houston – Poduska PI Training, Supervision Supported through Research
• In Silico: MicroSimulation of Expected Impact of Project KEEP for a State Child Welfare System
13
Microsimulation
• Start with completed efficacy or effectiveness trial data of an intervention
• Model how this intervention would behave in a larger system if implemented broadly
• Design – Needs: 1. Individual level data from trial 2. Longitudinal data from a large system 3. Programming to model and extrapolate
findings
14
KEEP Efficacy Trial - Chamberlain
• Study design: Randomized controlled trial • Population: Children in 700 foster homes in
San Diego county (ages 5-12) • Outcome: Efficacy of a foster parenting
intervention (KEEP) in reducing placement changes over 200 days
• Findings: – Positive exits (reunification with biological families,
adoption, etc) increased – Negative exits (other foster homes, group care, etc)
decreased 15
KEEP Applied to Statewide Data – Jeremy Goldhaber-Fiebert
16
-0.60
-0.50
-0.40
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
Effe
ct o
f KEE
P on
cu
mul
ativ
e in
cide
nce
Positive exits Negative exits Lateral moves
II. B Three Useful Dimensions for Typing of Studies for D & I Research
NIH Training Institute for D&I Research Ce-PIM 17
How much control in implementation
strategy and/or intervention assignment Research Influence Observational - NONE Quasi-Experimental SOME QI-Designs Randomized - Randomizes
implementation strategy to systems (components)
Standard Names for Experimental and Quasi-Experimental Designs (Cochrane,
EPOC)
• Randomized Controlled Trial (RCT) – random allocation to implementation/intervention arms
• Controlled Clinical Trial (CCT) – assigned in some controlled way (even-odd)
• Controlled Before and After Study (CBA) – Pre-post data same time for different arms Comparable controls More than 1 site in each arm • Interrupted Time Series (ITS) – sites
change from non-intervention to intervention 18
Biglan et al., AJCP 2006
NIH Training Institute for D&I Research Ce-PIM 19
Other Communities
NIH Training Institute for D&I Research Ce-PIM 20
Important Research Questions:
NIH Training Institute for D&I Research Ce-PIM 21
Which communities/organizations get invited an
— What percent of those invited participate? — What are the characteristics of
participants? — What are barriers to participation in this context?
Useful Nonrandomized Designs for Dissemination RE-AIM components
NIH Training Institute for D&I Research Ce-PIM 22
1. Broadcast of an Intervention Standardize Invitation and See Who Comes Kellam SG, Branch J, Brown CH, Russell G. Why teenagers come for treatment: A ten-year prospective
study of Woodlawn. Journal of the American Academy of Child Psychiatry 20:477-95, 1981.
2. Narrowcast of an Intervention Social Marketing to a Targeted Audience
3. “Timecast” of an Intervention Multiple Baseline design = Interrupted Time Series o Repeated measures over time of a community outcome o Introduce the intervention to a community midway through o Check whether community outcome differs before and after introduction
Hawkins NG et al., AJPM 2007
NIH Training Institute for D&I Research Ce-PIM 23
II. C Three Useful Dimensions for Typing of Studies for D & I Research
NIH Training Institute for D&I Research Ce-PIM 24
What Levels Are Measured? System Leaders County MH Agency Leader Implementation Agents Agency Hiring Clinicians Intervention Agents Clinicians Delivering Services Intervention Target Families Receiving Services
Example
25
Four Levels of Change for Assessing
Performance Improvement
Assumptions about Change
Larger System / Environment
Organization
Group / Team
Individual
Reimbursement, legal, and
regulatory policies are key
Cooperation, coordination, &
shared knowledge are key
Structure and strategy are key
Knowledge, skill, and
expertise are key
From Shortell, 2004
Types of Evidence for Implementation
NIH Training Institute for D&I Research Ce-PIM 26
• Formal Causal Inference • Hierarchy of Evidence
Generative Modeling
III. Increasing the “Evidence” in Implementation Research
• Evidence is Elusive Concept 1. Formal use of the causality in
philosophy/statistics 2. Intuitive Ideas: Ordering -- Hierarchy of evidence 2 trials with replicated results
better than 1 trial
27
Hierarchies of Evidence
NIH Training Institute for D&I Research Ce-PIM 28
• Preventive Task Force Expert Opinion … Pre-Post Design … Single Randomized Trial Multiple Randomized Trial
Caution: Hierarchy Not Always True
NIH Training Institute for D&I Research Ce-PIM 29
• Date from “Observational Studies” May Be More Valuable than that From Randomized Trials
Example: Do antidepressants cause suicide in
youth? RCT’s – no suicides occurred, few
attempts
Intuitive Idea About Evidence • Study 1: Implementation Strategy A significantly better
than Strategy B Is A truly better than B? Other Explanations -- imbalanced assignment -- differential support staff… • If we can modify Study 1’s Design in such a way to
– reduce the risk of other explanations – Leave all other factors the same
• Then the new Study Design has Stronger Evidence
30
Example
• Study 1: Offer Implementation A to 40 Counties and Compare the 23 Who Select A against the 27 Who Do Not Select A (B).
• Study 1’ : Randomly assign 20 of the 40 Counties to Receive A and 20 to Receive B. And All 40 Counties Do Adhere to this Strategy.
31
Roll-Out Randomized Implementation Trials
• Straightforward way to improve the evidence over interrupted time series or multiple baseline designs
• Generally Acceptable by Communities • Beneficial properties in statistical power
and logistics Brown et al. Clinical Trials 2006 Brown et al. Ann Rev Pub Health 2008
32
Biglan et al., AJCP 2006
NIH Training Institute for D&I Research Ce-PIM 33
Inferential Challenge with this Design
NIH Training Institute for D&I Research Ce-PIM 34
• What if and Exogenous Factor Happens at one of these Times of Transition?
• What if you Select the Most Promising Communities to Work with First?
• What if there are only a small number of communities?
• Harder to Conclude that Intervention Caused Change.
Turning a Multiple Baseline Design into a True Randomized Experiment: Roll-Out Design
NIH Training Institute for D&I Research Ce-PIM 35
ROLL-OUT DESIGN • Divide Available Communities into Comparable
Batches • Start Measuring Outcomes for All Communities • Randomly Assign Each Comparable Batch to
WHEN the Implementation Begins • At the end, ALL Communities Are Exposed
• Analysis Uses All Communities and All Times
• Communities Still Serve as Own Controls • Communities Compared by Exposure Status Across Time
Roll-Out Randomized Trials for Dissemination Research (Brown et al.,
2006 2008)
NIH Training Institute for D&I Research Ce-PIM 36
Population
1
2 3 4
5
Time of Transition in Dissemination Randomly Determined
R
Equivalent Subsets that are Ordered
Randomly
Timing of Dissemination ( 0 to x)
NIH Training Institute for D&I Research Ce-PIM 37
• 1 0 0 x x x x x
• 2 0 0 0 x x x x
• 3 0 0 0 0 x x x
• 4 0 0 0 0 0 x x
• 5 0 0 0 0 0 0 x
Random Assignment Reduces Bias in the Long Run
NIH Training Institute for D&I Research Ce-PIM 38
• Example: mPowerment Community Intervention Yr 1 Yr 2 . . . Yr 10
Treat Control Treat Control
R
R
R
Implication of Roll-Out Designs for Community Research
NIH Training Institute for D&I Research Ce-PIM 39
• Units (communities) that get randomized are Large and often few are available at a time
A Single Trial with a Small Number of Communities is
Nearly Always Underpowered • Randomize small numbers of communities now • Randomize small numbers next year • … • Randomize small numbers in following years • Combine results across the years
• Brown et al., Ann Rev Public Health 2009 • Brown et al., Prev Science 2011
Roll-Out Randomized Trials Are Often Acceptable to Communities and
Researchers • Traditional Control Group is Nonstarter for Many
Communities
Why Communities Are Comfortable Participating • Going Early – get a potentially valuable intervention • Going Later – potentially improved intervention • Randomization is Fair
Statistical and Logistical Reasons for Roll-Out Design • Higher statistical power over traditional wait-listed designs • Robust to External Influences • Higher Training Achieved When Training in Small Groups
40 Family Research Consortium San Juan
July 2011
Examples of Roll-Out Trials Now Being Conducted
• 2 Generations of School-Based Suicide Prevention Trials
• Evaluation of the Training of Suicide Prevention Hot Lines
• Randomized Implementation Trial of 40 Counties in California for an Evidence-Based Foster Care Program (MTFC)
Family Research Consortium San Juan July 2011 41
Randomize 40 Counties in CA Chamberlain et al., Adm Pol MH 2008
Family Research Consortium San Juan July 2011 42
40 CA Counties
26 Wait LIsted
CDT
Stnd
Wait Listed
13 Wait LIsted
COHORT 1 COHORT 2 COHORT 3
Where These Designs Are Likely to Go
• Implementation Trials of Evidence-Based Programs
• Inclusion of Programs in Health Reform through Research Based Practice Organizations
Family Research Consortium San Juan July 2011 43
Summary • Implementation Designs Classified Along How “Real World” How much control over Implementation
Assignment What Levels Are Measured?
• Numerous Opportunities to Improve Evidence
through Design Enhancements – Obtain wider distribution of communities – Strategic use of random assignment – Roll-Out Randomized Trials
• Community Buy-In • Improved Statistical Properties
44
Where are we going in Implementation Research?
NIH Training Institute for D&I Research Ce-PIM 45
Your Role in the Next Stage of Implementation Research: Relying
on What We Now Know
NIH Training Institute for D&I Research Ce-PIM 46
Your Role in the Next Stage of Implementation Research: Your
Ideas in Building This Field
NIH Training Institute for D&I Research Ce-PIM 47