Applying the GRADE approach to diagnostic technologies
Holger Schünemann
for the GRADE working group
Today’s talk
1) Focus on patient important outcomes
2) Factors influencing the quality of evidence in diagnostic recommendations
GRADE for diagnosis
Shares the fundamental logic of assessment for treatment
However, assessments present unique challenges
Examples and solutions for how to deal with challenges
Focus on importance to patients and consumers!
Summary Quality of evidence reflects our confidence
that estimates of benefits and downsides from a diagnostic strategy generated from research are correct.
Consideration of the directness of evidence is based on how confident we are of the relation between being classified correctly (as a true positive or negative) or incorrectly (as a false positive or negative) and patient-important consequences.
Recommendation depends on the balance between desirable and undesirable effects of the diagnostic test or strategy in terms of patient-important outcomes.
Testing makes a variety of contributions to patient care Clinicians use tests that are usually referred to as
“diagnostic” signs and symptoms, imaging, biochemistry,
pathology, and psychological Some tests naturally report positive and negative
results (pregnancy) Other tests report their results in categories (e.g.
imaging) Today we assume a diagnostic approach that
ultimately categorizes test results as positive or negative
Purpose of a test
Triage to minimize use of an invasive or expensive
test Add-on
to improve diagnosis beyond what is already done
Replacement to replace test that is harmful or costly
Bossuyt et al. BMJ 2006
What is the sensitivity, specificity (accuracy), or likelihood ratios of multislice spiral computed tomography (CT) of coronary arteries compared with conventional invasive angiography?
Test accuracy is a surrogate for patient important outcomes
When clinicians think about diagnostic tests, they focus on their accuracy
Underlying assumption: obtaining a better idea of whether a target condition is present or absent will result in superior patient management and improved outcome.
Test and treatment threshold
Diagnostic tests are used in patients suspected of disease to exclude or confirm a diagnosis.
Purpose of the test
Identify the limitations for which the (new or) alternative test offers a putative remedy eliminating a high proportion of false
positive or negative results, enhancing availability, decreasing invasiveness, or decreasing cost
Identification of sensible clinical questions: patients, diagnostic intervention, comparison, management, and outcomes of interest
Bossuyt et al. BMJ 2006
Sensible clinical questionPopulation: In patients suspected of
coronary artery disease Intervention: does multislice spiral
computed tomography (CT) of coronary arteries
Comparison: compared with conventional invasive coronary
angiography Outcomes: lower complications with
acceptable rates of false negatives (associated with coronary events) and false
positives (leading to unnecessary treatment andcomplications)?
Study designs for diagnosis
If a test fails to improve important outcomes: no reason to use it, whatever its accuracy
Best way to assess diagnostic strategy: randomized controlled trial in which investigators randomize patients to different diagnostic strategies
Study designs
Focus on: mortality, morbidity, symptoms, and quality of life
GRADE approach for treatment or intervention
Patient benefit required Inference from accuracy data that a diagnostic
test or strategy improves patient-important outcome requires availability of effective management strategies
Others: Reduction in test-related adverse effects Exclusion of a disease and reduction in
anxiety Confirming a diagnosis improves patient
well-being from the prognostic information it imparts
Patient benefit required
Genetic testing for Huntington’s chorea reassurance that a patient will not
suffer from the condition ability to plan for future knowing
that patient will sadly fall victim Ability to plan is analogous to an
effective treatment
Almqvist et al Clin Gen 2003
Example of new test and reference test or strategy
Putative benefit of new test
Diagnostic accuracy Patient Outcomes and expected impact on management for the following test outcomes
Sensitivity Specificity
True positives
False positives
True negatives
False negatives
Helical CT for renal calculus compared with intravenous pyeolgram
Detection of more (but smaller) calculi
greater equal Presumed influence on patient important outcomes
Certain benefit for larger stones, for smaller stones the benefit is less clear and unnecessary treatment can result
Likely detriment from unnecessary additional invasive tests
Almost certain benefit from avoiding unnecessary tests
Likely detriment for large stones, less certain for small stones More testing
Directness of the evidence (test results) for patient-important outcomes
Some uncertainty
No uncertainty
No uncertainty
Major uncertainty
Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.
Today’s talk
1) Focus on patient important outcomes
2) Factors influencing the quality of evidence in diagnostic recommendations
Factors that decrease the quality of evidence (and how they differ from treatment approach)
Study designDifferent quality criteria for accuracy studiesValid accuracy studies: Diagnostic uncertainty Consecutive patients Evaluators should be blinded
Factors that decrease the quality of evidence
IndirectnessPopulation, Test and ComparisonOutcomes Similar quality criteria Usual absence of direct evidence about impact
on patient-important outcomes Accuracy studies typically provide low quality
evidence
Factors that decrease the quality of evidenceInconsistency in study results
Similar quality criteria & judgments but: other measures
Imprecise evidenceSimilar quality criteria & judgments, but: Wide confidence intervals for estimates of test accuracy, true and false positives and negatives
Reporting biasSimilar quality criteria & judgments
Balance between presumed patient outcomes, complications and cost: Avoiding the undesirable consequences of more false positives with CT is preferable to avoiding the higher rate of complications (infarction and death) and higher cost with angiography
Example of new test and reference test or strategy
Putative benefit of new test
Diagnostic accuracy Patient Outcomes and expected impact on management for the following test outcomes
Sensitivity Specificity
True positives
False positives
True negatives
False negatives
CT for coronary artery disease compared with coronary angiography
Less invasive testing
equal less Presumed influence on patient important outcomes
Benefit from treatment and fewer complications
Harm from unnecessary treatment
Benefit from reassurance and fewer complications
Detriment from delayed diagnosis or myocardial insult
Directness of the evidence (test results) for patient-important outcomes
No uncertainty
No uncertainty
No uncertainty
Some uncertainty
Evaluating tests – when can
comparative evidence of test
accuracy and other intermediate
outcomes be used as an alternative to randomized trials
(Lord, Irwig, Bossuyt)
Terminology
Directness = PICO Synonyms: Generalizability, Applicability Directness of outcomes: Linkage assumptions
Explicit assumptions about management of TP, TN, FP, FN, complications, indeterminates Treatment effects on:
Mortality, morbidity, HRQL (including psychological well being)
25
Summary of findings
Key findings for diagnostic accuracy studies. Should multislice spiral computed tomography versus conventional coronary angiography be used to diagnose coronary artery disease in a population with a low (20%) pretest probability?
Test findings&
Pooled sensitivity 0.96 (95% CI: 0.94 - 0.98) LR(+) 5.4 (95% CI: 3.4–8.3) Pooled specificity 0.74 (95% CI: 0.065 – 0.84) LR(–) 0.05 (95% CI: 0.03–0.09) Consequences Number per 1000 Importance+ TP 192 8 FP 208 7 TN 592 8 FN 8 9
28
Outcomes
Illustrative Risks(95% CI) Number of
participants (studies)
Quality of the Evidence1 Comments
Assumed outcome with CT – prevalence of 20%
True positives(Patients correctly classified as having coronary artery disease)
192 per 1000 1570(21)
Moderate2
Benefit from treatment and fewer complications.* Some patients will have to undergo angiography.
True negatives (Patients correctly classified as not having coronary artery disease)
592 per 1000 1570(21)
Moderate2
Benefit from reassurance and fewer complications
False positives (Patients incorrectly classified as having coronary artery disease)
208 per 1000 1570(21)
Moderate2
Harm from unnecessary treatment
False negatives (Patients incorrectly classified as not having coronary artery disease)
8 per 1000 1570(21)
Low2, 3
Detriment from delayed diagnosis or myocardial insult
Complications(MI, allergic reactions, renal failure)
99 per 1000 1570(21)
Low2
There is a higher rate of rare complications (infarction and death) and higher cost with angiography – a full profile would be required.
Resource use*(cost of CT and Angiography)
See comment See comment
See comment
Cost are higher for angiography,
1- Quality rated from 1 (very low quality) to 4 (high quality), 2- Cross sectional studies. Indirectness of outcomes in a wide spectrum of patients and indirect comparison of tests, 3– there is greater uncertainty whether these patients will have negative outcomes.*Assumed efficacy of: 1) aspirin daily = 20% RRR; 2) beta-blockage = 18% RRR.
Summary (1) Quality of evidence reflects our confidence
that estimates of benefits and downsides from a diagnostic strategy generated from research are correct.
Summary (2) Consideration of the directness of evidence
is based on how confident we are of the relation between being classified correctly (as a true positive or negative) or incorrectly (as a false positive or negative) and patient-important consequences.
Summary Recommendation depends on the balance
between desirable and undesirable effects of the diagnostic test or strategy in terms of patient-important outcomes.
Example of new test and reference test or strategy
Putative benefit of new test
Diagnostic accuracy Patient Outcomes and expected impact on management for the following test outcomes
Sensitivity Specificity
True positives
False positives
True negatives
False negatives
Helical CT for renal calculus compared with intravenous pyeolgram
Detection of more (but smaller) calculi
greater equal Presumed influence on patient important outcomes
Certain benefit for larger stones, for smaller stones the benefit is less clear and unnecessary treatment can result
Likely detriment from unnecessary additional invasive tests
Almost certain benefit from avoiding unnecessary tests
Likely detriment for large stones, less certain for small stones More testing
Directness of the evidence (test results) for patient-important outcomes
Some uncertainty
No uncertainty
No uncertainty
Major uncertainty
Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.