1

Moderators of Treatment Moderators of Treatment Effects in the General Effects in the General Medicine Literature: Medicine Literature:

Looking for ImprovementLooking for Improvement

Nicole Bloser, MHA, MPHNicole Bloser, MHA, MPH

University of California, DavisUniversity of California, Davis

June 5, 2007June 5, 2007

2

BackgroundBackground

Parallel group randomized Parallel group randomized controlled trials (RCTs) are the controlled trials (RCTs) are the cornerstone of evidence based cornerstone of evidence based medicinemedicine

Result: Average of usually Result: Average of usually immeasurable individual effectsimmeasurable individual effects Some benefit, some harmedSome benefit, some harmed Heterogeneity of treatment effectsHeterogeneity of treatment effects

3

BackgroundBackground

Examining treatment impact in Examining treatment impact in similar individuals or subgroupssimilar individuals or subgroups N-of-1 clinical trialN-of-1 clinical trial Prospective stratification with a Prospective stratification with a

multivariable risk indexmultivariable risk index Subgroup analysisSubgroup analysis

4

BackgroundBackground

Subgroup analysis – problemsSubgroup analysis – problems Low powerLow power Multiple testingMultiple testing

Not all subgroup analysis is the sameNot all subgroup analysis is the same P-value within subgroup ≠ interaction P-value within subgroup ≠ interaction

analysisanalysis Interaction analysis is the correct way Interaction analysis is the correct way

to examine moderators of treatment to examine moderators of treatment effects (MTEs)effects (MTEs)

5

BackgroundBackground

MTEs necessary to maximize benefit MTEs necessary to maximize benefit and minimize harm and minimize harm

MTEs often not examined (<50% of MTEs often not examined (<50% of RCTs report interaction analysis)RCTs report interaction analysis)

Studies that examine MTE reporting:Studies that examine MTE reporting: Majority in cardiovascular literatureMajority in cardiovascular literature None since revised CONSORT None since revised CONSORT

statement (2001)statement (2001)

6

Research ObjectiveResearch Objective

We sought to identify current We sought to identify current practice in evaluating moderators of practice in evaluating moderators of treatment effects (MTEs) and to treatment effects (MTEs) and to elucidate trendselucidate trends

Persistent low rate of analyses Persistent low rate of analyses would suggestwould suggest Missed opportunitiesMissed opportunities Slower progress towards personalized Slower progress towards personalized

medicinemedicine

7

Study DesignStudy Design

Systematic reviewSystematic review Annals, BMJ, JAMA, Lancet, NEJMAnnals, BMJ, JAMA, Lancet, NEJM Odd months, 1994, 1999, 2004Odd months, 1994, 1999, 2004

Randomized controlled trials Randomized controlled trials Unit of randomization as the individualUnit of randomization as the individual All independently reviewed and coded All independently reviewed and coded

by two investigatorsby two investigators Adjudication by a third Adjudication by a third

8

Study Selection Study Selection 4863 articles from initial search

Exclude: All articles that were not clinical trials. N=4,322

N=541, Random sample of N=379 selected

Include Exclude

N=303 articles

N=76 articles

319 trials included

9 trials excluded

77 trials excluded

Reasons for exclusion:

Not RCT (N=61 trials)

Unit of randomization not individual patient (N=25 trials)

9

MethodsMethods

Trials were coded as havingTrials were coded as having MTE analysis (utilizing a formal test for MTE analysis (utilizing a formal test for

heterogeneity)heterogeneity) Subgroup analysis only (no formal test)Subgroup analysis only (no formal test) NeitherNeither

Chi-square test used for bivariate Chi-square test used for bivariate comparisonscomparisons

Multiple logistic regression used to Multiple logistic regression used to identify predictors of MTE analysisidentify predictors of MTE analysis

10

Trial CharacteristicsTrial Characteristicsn (%)

Journal of PublicationAnnals 30 (9)BMJ 47 (15)JAMA 50 (16)Lancet 101 (32)NEJM 91 (29)

Year of Publication1994 91 (29)1999 106 (33)2004 122 (38)

Condition StudiedCardiovascular 74 (23)Cancer 42 (13)Infectious Disease 70 (22)Psych / Neuro 25 (8)Other 108 (34)

11

Trial CharacteristicsTrial Characteristics

38% of trials had authors from North 38% of trials had authors from North America (US and Canada)America (US and Canada)

95% utilized a parallel group design95% utilized a parallel group design Study sample size ranged from 6-Study sample size ranged from 6-

41,00041,000

(Median: 262, IQR: 101-708)(Median: 262, IQR: 101-708)

12

MTE and Subgroup MTE and Subgroup ReportingReporting

For those trials reporting MTE analysis:For those trials reporting MTE analysis: 43 (47%) reported on one covariate43 (47%) reported on one covariate 24 (26%) reported on 2-4 covariates24 (26%) reported on 2-4 covariates 17 (18%) reported on 5-10 covariates17 (18%) reported on 5-10 covariates 7 (7%) reported on 11-19 covariates7 (7%) reported on 11-19 covariates

Analysis reported n (%)None 139 (45)Subgroup without statistical comparison 88 (28)MTE 92 (29)

13

Among those trials that reported MTE, Among those trials that reported MTE, majormajor

covariates examined included:covariates examined included:

Only one trial reported MTE analysis using Only one trial reported MTE analysis using a composite multivariable risk index.a composite multivariable risk index.

Covariates examined for Covariates examined for MTEMTE

%Individual Risk Factors 56%Gender 36%Age 29%Study Site 29%Co-occurring treatment 25%Co-morbidities 21%Race/ Ethnicity 15%

14

Bivariate analysisBivariate analysis

Journals published in North America Journals published in North America (Annals, JAMA) and first authors writing (Annals, JAMA) and first authors writing from North America were more likely to from North America were more likely to publish trials with MTE analysispublish trials with MTE analysis

Sample size (p<0.0001 for trend)Sample size (p<0.0001 for trend) Quintile 1: 14%, Quintile 5: 52%Quintile 1: 14%, Quintile 5: 52%

More MTE analysis reported in each of More MTE analysis reported in each of the three successive time periods the three successive time periods (p=0.047)(p=0.047)

15

Logistic RegressionLogistic Regression

Prediction of MTE analysisPrediction of MTE analysis Included in model: study year, journal Included in model: study year, journal

clinical condition, first author’s region, clinical condition, first author’s region, and sample sizeand sample size

Journal and sample size were significantJournal and sample size were significant Reference categories: BMJ, Quintile 1Reference categories: BMJ, Quintile 1 JAMAJAMA: 4.4 (1.4, 13.5), : 4.4 (1.4, 13.5), AnnalsAnnals: 4.2 (1.2, : 4.2 (1.2,

15.1)15.1) Quintile 5Quintile 5: 7.5 (2.9, 19.3): 7.5 (2.9, 19.3)

16

LimitationsLimitations

Limited number of trials reviewedLimited number of trials reviewed MTE results are published MTE results are published

elsewhereelsewhere MTE examined, but not reported due MTE examined, but not reported due

to non-significanceto non-significance

17

ConclusionsConclusions

Missed opportunities for MTE Missed opportunities for MTE analysis aboundanalysis abound Conservative reaction that stifles Conservative reaction that stifles

hypothesis generationhypothesis generation Impairs recognition of patient strataImpairs recognition of patient strata Impedes future researchImpedes future research

When subgroups are reported, the When subgroups are reported, the reporting is not done correctly in reporting is not done correctly in many (~50%) cases. This could lead many (~50%) cases. This could lead to erroneous conclusions.to erroneous conclusions.

18

ConclusionsConclusions

NIH guidelines regarding subgroup NIH guidelines regarding subgroup specific results recommends reporting specific results recommends reporting both significant and non-significant both significant and non-significant results, yet only half of the trials results, yet only half of the trials reported any MTE analysis.reported any MTE analysis.

In the face of broad NIH mandates for In the face of broad NIH mandates for inclusion of subjects by race/ethnicity, inclusion of subjects by race/ethnicity, the low proportion of trials examining the low proportion of trials examining race/ethnicity as a treatment effect race/ethnicity as a treatment effect modifier is puzzling.modifier is puzzling.

19

Policy ImplicationsPolicy Implications

MTE analysis critical to future researchMTE analysis critical to future research The genomic revolution is only going to The genomic revolution is only going to

increase the desire to individualize treatment increase the desire to individualize treatment effectseffects

Kraemer et al. argue that exploratory Kraemer et al. argue that exploratory moderator analysis is critical for designing moderator analysis is critical for designing future confirmatory studiesfuture confirmatory studies Significant exploratory effects are later used as Significant exploratory effects are later used as

guidance for future stratificationguidance for future stratification

Kraemer, HC, E Frank, and DJ Kupfer, Kraemer, HC, E Frank, and DJ Kupfer, Moderators of treatment outcomes: clinical, research, and policy Moderators of treatment outcomes: clinical, research, and policy importance. importance.

Jama, 2006. 296(10): p. 1286-9.Jama, 2006. 296(10): p. 1286-9.

20

Policy ImplicationsPolicy Implications

Rigorous and routine exploratory Rigorous and routine exploratory MTE analysis is necessary and MTE analysis is necessary and should be encouragedshould be encouraged

Standards are essential for Standards are essential for developing practice guidelines that developing practice guidelines that are appropriate to the needs of are appropriate to the needs of complex patientscomplex patients

21

Research TeamResearch Team

University of University of California, DavisCalifornia, Davis

Richard Kravitz, MD, Richard Kravitz, MD, MSPH - PIMSPH - PI

Elizabeth Yakes, MSElizabeth Yakes, MS

University of University of California, Los California, Los AngelesAngeles

Naihua Duan, PhDNaihua Duan, PhD

- PI- PI

Diana Liao, MPHDiana Liao, MPH

Kiavash NikkhouKiavash Nikkhou

22

Thank youThank you