Selective cutoff reporting in studies of diagnostic test accuracy of depression screening tools: Selective cutoff reporting in studies of diagnostic test

Selective cutoff reporting in studies of diagnostic test accuracy of depression screening tools: Selective cutoff reporting in studies of diagnostic test accuracy of depression screening tools:

Comparing traditional meta-analysis to Comparing traditional meta-analysis to individual patient data meta-analysisindividual patient data meta-analysis

Brooke Levis, MSc, PhD CandidateBrooke Levis, MSc, PhD CandidateJewish General Hospital and McGill UniversityJewish General Hospital and McGill University

Montreal, Quebec, CanadaMontreal, Quebec, Canada

Does Selective Reporting of Data-driven Does Selective Reporting of Data-driven Cutoffs Exaggerate Accuracy? Cutoffs Exaggerate Accuracy?

The Hockey AnalogyThe Hockey Analogy

2

What is Screening?What is Screening?

Illustration: This information was originally developed by the UK National Screening Committee/NHS Screening Programmes (www.screening.nhs.uk) and is used under the Open Government Licence v1.0

Purpose to identify Purpose to identify otherwise otherwise unrecognisableunrecognisable diseasedisease

By sorting out apparently By sorting out apparently well persons who well persons who probably have a condition probably have a condition from those who probably from those who probably do notdo not

Not diagnosticNot diagnostic Positive tests require Positive tests require

referral for diagnosis and, referral for diagnosis and, as appropriate, treatmentas appropriate, treatment

A program – of which a A program – of which a test is one componenttest is one component

3

Patient Health Patient Health Questionnaire (Questionnaire (PHQ-9PHQ-9))

Depression screening toolDepression screening tool

Scores range from 0 to 27Scores range from 0 to 27

Higher scores = Higher scores = more severe symptomsmore severe symptoms

The Patient Health Questionnaire (PHQ-9) The Patient Health Questionnaire (PHQ-9) depression screening tooldepression screening tool

4

Extreme scenarios:Extreme scenarios: Cutoff of ≥ 0Cutoff of ≥ 0

All subjects above cutoff All subjects above cutoff

sensitivity = 100% sensitivity = 100%

Cutoff of ≥ 27Cutoff of ≥ 27 All subjects below cutoff All subjects below cutoff

specificity = 100%specificity = 100%

Selective Reporting of Results Using Selective Reporting of Results Using Data-Driven CutoffsData-Driven Cutoffs

5

Does Selecting Reporting of Data-driven Does Selecting Reporting of Data-driven Cutoffs Exaggerate Accuracy?Cutoffs Exaggerate Accuracy?

Sensitivity increases from cutoff of 8 to cutoff of 11

For standard cutoff of 10, missing 897 cases (13%)

For cutoffs of 7-9 and 11, missing 52-58% of data

Manea et al., CMAJ, 20126

QuestionsQuestions

Does selective cutoff reporting lead to exaggerated Does selective cutoff reporting lead to exaggerated estimates of accuracy?estimates of accuracy?

Can we identify predictable patterns of selective cutoff Can we identify predictable patterns of selective cutoff reporting?reporting?

Why does selective cutoff reporting appear to impact Why does selective cutoff reporting appear to impact sensitivity, but not specificity?sensitivity, but not specificity?

Does selective cutoff reporting transfer high Does selective cutoff reporting transfer high heterogeneity in sensitivity due to small numbers of heterogeneity in sensitivity due to small numbers of cases to heterogeneity in cutoff scores, but cases to heterogeneity in cutoff scores, but homogeneous accuracy estimates?homogeneous accuracy estimates?

7

MethodsMethods

Data source: Data source: Studies included in published traditional meta-analysis on Studies included in published traditional meta-analysis on

the diagnostic accuracy of the PHQ-9. (Manea et al, CMAJ the diagnostic accuracy of the PHQ-9. (Manea et al, CMAJ 2012)2012)

Inclusion criteria:Inclusion criteria: Unique patient sampleUnique patient sample Published diagnostic accuracy for MDD for at least one Published diagnostic accuracy for MDD for at least one

PHQ-9 cutoffPHQ-9 cutoff

Data transfer: Data transfer: Invited authors of the eligible studies to contribute their Invited authors of the eligible studies to contribute their

original patient data (de-identified)original patient data (de-identified) Received data from 13 of 16 eligible datasets (80% of Received data from 13 of 16 eligible datasets (80% of

patients, 94% of MDD cases)patients, 94% of MDD cases)8

MethodsMethods

Data preparationData preparation For each dataset, extracted PHQ-9 scores and MDD For each dataset, extracted PHQ-9 scores and MDD

diagnostic status for each patient, and information diagnostic status for each patient, and information pertaining to weightingpertaining to weighting

Statistical analyses (2 sets performed)Statistical analyses (2 sets performed) Traditional meta-analysisTraditional meta-analysis

For each cutoff between 7 and 15, included data from the For each cutoff between 7 and 15, included data from the studies that reported accuracy results for the respective studies that reported accuracy results for the respective cutoff in the original publicationcutoff in the original publication

IPD meta-analysisIPD meta-analysis For each cutoff between 7 and 15, included data from all For each cutoff between 7 and 15, included data from all

studiesstudies

9

Published data

(traditional MA)All data (IPD MA)

Cutoff# of

studies# of

patients# mdd cases

# of studies

# of patients

# mdd cases

7 4 2094 550 13 4589 1037

8 4 2094 550 13 4589 1037

9 4 1579 309 13 4589 1037

10 11 3794 723 13 4589 1037

11 5 1253 216 13 4589 1037

12 6 1388 261 13 4589 1037

13 4 1073 186 13 4589 1037

14 3 977 150 13 4589 1037

15 4 1075 193 13 4589 1037

Comparison of data availability Comparison of data availability

10

MethodsMethods

Model: Model: Bivariate random-effects* Bivariate random-effects* meta-analysis modelsmeta-analysis models

Models sensitivity and specificity at the same timeModels sensitivity and specificity at the same time

Accounts for clustering by studyAccounts for clustering by study

Provides an overall pooled sensitivity and specificity for each Provides an overall pooled sensitivity and specificity for each cutoff, for the 2 sets of analysescutoff, for the 2 sets of analyses

Within each set of analyses, each cutoff requires its own modelWithin each set of analyses, each cutoff requires its own model

Estimates between study heterogeneityEstimates between study heterogeneity

Note:Note: model accounts for correlation between sensitivity and specificity model accounts for correlation between sensitivity and specificity atat each each threshold, but not for correlation of parameters threshold, but not for correlation of parameters acrossacross thresholds thresholds

*Random effects model*Random effects model: sensitivity & specificity assumed to vary across primary studies: sensitivity & specificity assumed to vary across primary studies

11

QuestionsQuestions





12

Comparison of Diagnostic Accuracy Comparison of Diagnostic Accuracy

Published data (traditional MA)

All data (IPD MA)

Cutoff

N studies

Sens SpecCuto

ffN

studiesSen

sSpec

7 4 0.85 0.73 7 130.97

0.73

8 4 0.79 0.78 8 130.93

0.78

9 4 0.78 0.82 9 130.89

0.83

10 11 0.85 0.88 10 130.87

0.88

11 5 0.92 0.90 11 130.83

0.90

12 6 0.82 0.92 12 130.77

0.92

13 4 0.82 0.94 13 130.67

0.94

14 3 0.71 0.97 14 130.59

0.96

15 4 0.61 0.98 15 130.52

0.97

13

Comparison of ROC CurvesComparison of ROC Curves

14

QuestionsQuestions


Can we identify predictable patterns of selective Can we identify predictable patterns of selective cutoff reporting?cutoff reporting?



15

Publishing trends by studyPublishing trends by study

16

Comparison of Sensitivity by Cutoff Comparison of Sensitivity by Cutoff

17

QuestionsQuestions





18

Comparison of Diagnostic Accuracy Comparison of Diagnostic Accuracy

19

Published data (traditional MA)

All data (IPD MA)

Cutoff

N studies

Sens SpecCuto

ffN

studiesSen

sSpec

7 4 0.85 0.73 7 130.97

0.73

8 4 0.79 0.78 8 130.93

0.78

9 4 0.78 0.82 9 130.89

0.83

10 11 0.85 0.88 10 130.87

0.88

11 5 0.92 0.90 11 130.83

0.90

12 6 0.82 0.92 12 130.77

0.92

13 4 0.82 0.94 13 130.67

0.94

14 3 0.71 0.97 14 130.59

0.96

15 4 0.61 0.98 15 130.52

0.97

Why Sensitivity Changes with Why Sensitivity Changes with Moving Cutoffs, but Not SpecificityMoving Cutoffs, but Not Specificity

20

QuestionsQuestions





21

HeterogeneityHeterogeneity

22

SummarySummary

Selective cutoff reporting in depression screening tool DTA Selective cutoff reporting in depression screening tool DTA studies may distort accuracy across cutoffs.studies may distort accuracy across cutoffs.

It will lead to exaggerated estimates of accuracy.It will lead to exaggerated estimates of accuracy.

These distortions were relatively minor in the PHQ, but would These distortions were relatively minor in the PHQ, but would likely be much larger for other measures where standard likely be much larger for other measures where standard cutoffs are less consistently reported and more data-driven cutoffs are less consistently reported and more data-driven reporting seems to occur (e.g., HADS).reporting seems to occur (e.g., HADS).

IPD meta-analysis can address this and will allow subgroup-IPD meta-analysis can address this and will allow subgroup-based accuracy evaluation.based accuracy evaluation.

23

SummarySummary

STARD undergoing revision:STARD undergoing revision:

Needs to require precision-based sample size Needs to require precision-based sample size calculation to avoid very small samples – particularly calculation to avoid very small samples – particularly number of cases – and unstable estimatesnumber of cases – and unstable estimates

Needs to require reporting of spectrum of cutoffs, Needs to require reporting of spectrum of cutoffs, which is easily done with online appendiceswhich is easily done with online appendices

24

AcknowledgementsAcknowledgements

Brett ThombsBrett Thombs Andrea BenedettiAndrea Benedetti Roy ZiegelsteinRoy Ziegelstein Pim CuijpersPim Cuijpers Simon GilbodySimon Gilbody John IoannidisJohn Ioannidis

Alex LevisAlex Levis Danielle RiceDanielle Rice

Scott PattenScott Patten Dean McMillanDean McMillan Ian ShrierIan Shrier Russell SteeleRussell Steele Lorie KlodaLorie Kloda

DEPRESSD Investigators

Other Contributors

25

Documents

Selective cutoff reporting in studies of diagnostic test accuracy of depression screening tools: Selective cutoff reporting in studies of diagnostic test