Upload
benjamin-reeves
View
217
Download
2
Tags:
Embed Size (px)
Citation preview
11
Eloise E. KaizarThe Ohio State University
Combining Information From Combining Information From Randomized and Observational Randomized and Observational
Data: A Simulation StudyData: A Simulation Study
June 5, 2008
Joel Greenhouse
Howard SeltmanCarnegie Mellon University
22
OutlineOutline
Motivating ExampleMotivating Example– Association between suicidality and Association between suicidality and
antidepressant use in pediatric antidepressant use in pediatric populationpopulation
Trying to answer the right questionTrying to answer the right question Exploiting strengths of different dataExploiting strengths of different data Simulation StudySimulation Study
33
Pediatric Antidepressant UsePediatric Antidepressant Use
Problem: Antidepressant use may Problem: Antidepressant use may cause suicide for some cause suicide for some children/adolescentschildren/adolescents
Goal: Estimate the average Goal: Estimate the average treatment effect for use in regulatory treatment effect for use in regulatory decision makingdecision making
44
Randomized Controlled Randomized Controlled TrialsTrials
Hammad, et al. (2006) Archives of General Hammad, et al. (2006) Archives of General PsychiatryPsychiatry
55
The Right QuestionThe Right Question
Study population average treatment Study population average treatment effecteffect
Population average treatment effectPopulation average treatment effect
66
HeterogeneityHeterogeneity
Variation due to differences in Variation due to differences in population (“True”)population (“True”)
Variation due to differences in study Variation due to differences in study design (“Artifactual”)design (“Artifactual”)
77
Evidence for Weak External Evidence for Weak External ValidityValidity
Administrative dataAdministrative data– Show no significant association between Show no significant association between
antidepressant use and suicidal actions antidepressant use and suicidal actions (Valuck et al. 2004, Jick et al. 2004) (Valuck et al. 2004, Jick et al. 2004)
Epidemiological dataEpidemiological data– Suggest inverse relationship between Suggest inverse relationship between
antidepressant use and completed suicideantidepressant use and completed suicide Geographically (Gibbons et al. 2006, Isacsson Geographically (Gibbons et al. 2006, Isacsson
2000, Ludwig and Marcotte 2005)2000, Ludwig and Marcotte 2005) Temporally (Gibbons et al. 2007, Olfson, et al Temporally (Gibbons et al. 2007, Olfson, et al
1998)1998)
88
Assessing External ValidityAssessing External Validity
Compare the RCT patients with a Compare the RCT patients with a nationally representative probability nationally representative probability sample of adolescentssample of adolescents– Youth Risk Behavior Survey (YRBS)Youth Risk Behavior Survey (YRBS)– Representative of adolescents attending Representative of adolescents attending
school (aged 12-18)school (aged 12-18)– Basic demographic informationBasic demographic information– Self-report depressionSelf-report depression– Self-report suicidalitySelf-report suicidality
99
Match RCTs and YRBSMatch RCTs and YRBS
Consider only MDD RCTs of ages 12-18Consider only MDD RCTs of ages 12-18 Consider only YRBS respondents reporting Consider only YRBS respondents reporting
depressiondepression
YRBSYRBS RCTsRCTs
Average AgeAverage Age 16.14 (0.04)16.14 (0.04) 14.76 (2.99)14.76 (2.99)
% Female% Female 62.6 (1.8)62.6 (1.8) 63.8 (1.4)63.8 (1.4)
% White% White 54.1 (3.5)54.1 (3.5) 80.1 (1.2)80.1 (1.2)
Poststratify YRBS to match RCTsPoststratify YRBS to match RCTs
1010
Compare OutcomesCompare Outcomes
8-week suicidality8-week suicidality– RCTs 3.6%RCTs 3.6%– YRBS 7.1%YRBS 7.1%
Suicide attemptSuicide attempt– RCTs 5.4% (lifetime)RCTs 5.4% (lifetime)– YRBS 19.9% (12-month)YRBS 19.9% (12-month)
1111
Randomized Controlled Randomized Controlled TrialsTrials
Hammad, et al. (2006) Archives of General Hammad, et al. (2006) Archives of General PsychiatryPsychiatry
1212
Generalizing RCT DataGeneralizing RCT Data
Low RiskLow Risk
High RiskHigh Risk
Reduce the size of Reduce the size of the excluded the excluded populationpopulation– Practical Clinical Practical Clinical
TrialTrial Estimate the effect Estimate the effect
size in the size in the excluded excluded populationpopulation
1313
Current Approaches to Current Approaches to Estimating Average Effect SizeEstimating Average Effect Size
Use meta-analysis to combine RCT Use meta-analysis to combine RCT datadata– Assume effect is not systematically Assume effect is not systematically
heterogeneous by exclusion criteriaheterogeneous by exclusion criteria Use multi-level meta-analysis to Use multi-level meta-analysis to
combine RCT and observational datacombine RCT and observational data– Partial exchangeabilityPartial exchangeability– Assumes the mean is of interestAssumes the mean is of interest
Include bias parametersInclude bias parameters
1414
Proposed ApproachesProposed Approaches
Confidence Profile Method [Eddy, et al., 1988, Confidence Profile Method [Eddy, et al., 1988, 1989]1989]– Model the biases in observational and RCT dataModel the biases in observational and RCT data
Response Surface Approach [Rubin, 1990, Response Surface Approach [Rubin, 1990, 1991]1991]– Create a response surface that incorporates design Create a response surface that incorporates design
variablesvariables– Extrapolate to the ideal designExtrapolate to the ideal design
Cross Design Synthesis [US GAO, 1992]Cross Design Synthesis [US GAO, 1992]– Stratify data based on design variablesStratify data based on design variables– Extrapolate to empty cellsExtrapolate to empty cells
1515
Usefulness of EvidenceUsefulness of Evidence
External Validity
Inte
rnal
Val
idity
RCT
Obs.
IdealS
tron
ger
Wea
ker
StrongerWeaker
1616
FrameworkFrameworkSelf-Selection VariableSelf-Selection Variable
RandomizedRandomized(Strong Internal (Strong Internal
Validity)Validity)
Self-SelectedSelf-Selected(Weak Internal (Weak Internal
Validity)Validity)GeneralizabiliGeneralizabili
ty Variablety Variable
Eligible for Eligible for RandomizatioRandomizatio
nn
Ineligible for Ineligible for RandomizatioRandomizatio
nn
1717
FrameworkFrameworkSelf-Selection VariableSelf-Selection Variable
RandomizedRandomized(Strong Internal (Strong Internal
Validity)Validity)
Self-SelectedSelf-Selected(Weak Internal (Weak Internal
Validity)Validity)GeneralizabiliGeneralizabili
ty Variablety Variable
Eligible for Eligible for RandomizatioRandomizatio
nn
Ineligible for Ineligible for RandomizatioRandomizatio
nn
1818
Self-Selection VariableSelf-Selection Variable
RandomizedRandomized(Strong Internal (Strong Internal
Validity)Validity)
Self-SelectedSelf-Selected(Weak Internal (Weak Internal
Validity)Validity)GeneralizabiliGeneralizabili
ty Variablety Variable
Eligible for Eligible for RandomizatioRandomizatio
nn
Ineligible for Ineligible for RandomizatioRandomizatio
nn
FrameworkFramework
RandomizedExperiments
ObservationalStudies
1919
Self-Selection VariableSelf-Selection Variable
RandomizedRandomized(Strong Internal (Strong Internal
Validity)Validity)
Self-SelectedSelf-Selected(Weak Internal (Weak Internal
Validity)Validity)GeneralizabiliGeneralizabili
ty Variablety Variable
Eligible for Eligible for RandomizatioRandomizatio
nn
Ineligible for Ineligible for RandomizatioRandomizatio
nn
Linear Bias ModelLinear Bias Model
2020
Simulation StudySimulation Study Goal:Goal:
– Use simulation study to investigate effectiveness Use simulation study to investigate effectiveness of different methods for combining information of different methods for combining information from diverse sources in a realistic settingfrom diverse sources in a realistic setting
Key characteristics:Key characteristics:– 24 high-quality experiments with complete 24 high-quality experiments with complete
compliance and uniform randomization eligibilitycompliance and uniform randomization eligibility 200 subjects, individual data unavailable200 subjects, individual data unavailable
– 1 high-quality observational study with no 1 high-quality observational study with no generalizability biasgeneralizability bias 25,000 subjects, individual data available25,000 subjects, individual data available
2121
Simulation Study:Simulation Study:ImplementationImplementation
Generate 1000 data setsGenerate 1000 data sets
Fit models using Bayesian approachFit models using Bayesian approach Compare on MSE, bias and coverageCompare on MSE, bias and coverage
ScenarioScenario 00 11 22 33 44
Effect SizeEffect Size 0.80.8 0.70.7 0.70.7 0.80.8 0.70.7
Generalizability Generalizability BiasBias
00 0.40.4 0.40.4 00 0.40.4
Self-Selection BiasSelf-Selection Bias 00 0.40.4 00 0.40.455
-0.4-0.4
2222
0.0 0.5 1.0 1.5 2.0
Scenario 0 Estimate of Population Effect
RSMIncorrect Coefficients
RSMCorrect Coefficients
Three Level PoolingIG Prior
Random Effects PoolingIG Prior
Fixed Effects Pooling
CDSLinear Formulation
TruePopulation
Effect
0.011
0.011
0.013
0.016
0.016
0.015
MSE
-0.002
-0.002
0.004
0.020
0.020
-0.005
Average Bias
34.8 %
33.6 %
100.0 %
93.6 %
10.8 %
72.5 %
Coverage
2323
0.0 0.5 1.0 1.5 2.0
Scenario 1 Estimate of Population Effect
RSMIncorrect Coefficients
RSMCorrect Coefficients
Three Level PoolingIG Prior
Random Effects PoolingIG Prior
Fixed Effects Pooling
CDSLinear Formulation
TruePopulation
Effect
0.055
0.011
0.162
0.175
0.175
0.014
MSE
0.211
0.002
0.388
0.402
0.401
0.000
Average Bias
5.3 %
38.5 %
95.4 %
9.3 %
0.0 %
73.3 %
Coverage
2424
0.0 0.5 1.0 1.5 2.0
Scenario 2 Estimate of Population Effect
RSMIncorrect Coefficients
RSMCorrect Coefficients
Three Level PoolingIG Prior
Random Effects PoolingIG Prior
Fixed Effects Pooling
CDSLinear Formulation
TruePopulation
Effect
0.011
0.011
0.045
0.175
0.175
0.014
MSE
0.000
0.001
0.169
0.401
0.401
0.001
Average Bias
35.7 %
36.0 %
100.0 %
10.5 %
0.1 %
73.4 %
Coverage
2525
0.0 0.5 1.0 1.5 2.0
Scenario 3 Estimate of Population Effect
RSMIncorrect Coefficients
RSMCorrect Coefficients
Three Level PoolingIG Prior
Random Effects PoolingIG Prior
Fixed Effects Pooling
CDSLinear Formulation
TruePopulation
Effect
0.020
0.011
0.083
0.016
0.016
0.016
MSE
0.092
0.000
0.258
0.026
0.025
-0.025
Average Bias
27.7 %
42.1 %
100.0 %
93.8 %
9.6 %
72.5 %
Coverage
2626
0.0 0.5 1.0 1.5 2.0
Scenario 4 Estimate of Population Effect
RSMIncorrect Coefficients
RSMCorrect Coefficients
Three Level PoolingIG Prior
Random Effects PoolingIG Prior
Fixed Effects Pooling
CDSLinear Formulation
TruePopulation
Effect
0.058
0.012
0.021
0.176
0.176
0.017
MSE
-0.215
-0.002
-0.008
0.400
0.400
0.036
Average Bias
6.4 %
37.2 %
100.0 %
11.0 %
0.0 %
68.7 %
Coverage
2727
Self-Selection VariableSelf-Selection Variable
RandomizedRandomized(Strong Internal (Strong Internal
Validity)Validity)
Self-SelectedSelf-Selected(Weak Internal (Weak Internal
Validity)Validity)GeneralizabiliGeneralizabili
ty Variablety Variable
Eligible for Eligible for RandomizatioRandomizatio
nn
Ineligible for Ineligible for RandomizatioRandomizatio
nn
Linear Bias ModelLinear Bias Model
2828
SummarySummary
Even RCTs that have no heterogeneity may Even RCTs that have no heterogeneity may not be estimating the effect of interest.not be estimating the effect of interest.
Observational data may be used to assess Observational data may be used to assess the extent of the generalizability problemthe extent of the generalizability problem
The Cross Design Synthesis approach can The Cross Design Synthesis approach can potentially be effective for estimating potentially be effective for estimating average effect sizeaverage effect size
Still at the beginning of this workStill at the beginning of this work– More fair comparisons More fair comparisons – Extend to real settingsExtend to real settings