Upload
phamngoc
View
222
Download
3
Embed Size (px)
Citation preview
1
Methods in epidemiology – Medical statistics – Diagnosis 1
Diagnosis
How to evaluate diagnostic
accuracy
Methods in Epidemiology
At the end of the lecture students should be able
� to discuss the meaning of normality
� to report main biases in the evaluation of diagnostic tests
� to calculate main measures of accuracy of diagnostic
tests (sensitivity, specificity, positive and negative predictive
value)
� to interpret likelihood ratios and ROC curves when
assessing accuracy of a diagnostic test
� to measure reliability of diagnostic tests
Medical statistics
2
Methods in epidemiology – Medical statistics – Diagnosis 1
Methods in epidemiology – Medical statistics – Diagnosis 1
Context Coronary artery disease represents a leading cause of death and health care expenditure in Western countries. Establishing its anatomic diagnosis requires coronary angiography, a procedure that is costly and carries risks and discomfort.
Figure 2. Patient with severe stenosis in the first diagonal (D1) branch of the left anterior descending artery
3
Methods in epidemiology – Medical statistics – Diagnosis 1
Context
Multidetector computed tomography (MDCT) has been
proposed as a noninvasive method to evaluate coronary
anatomy.
Single-center studies have reported sensitivities between 30% and 95% and specificities between 86% and 98% for the detection of obstructive coronary lesions using 16-row MDCT scanners. ?Objective To determine the diagnostic accuracy of 16-row MDCT for the
detection of obstructive coronary disease in a larger
multicenter study.
Methods in epidemiology – Medical statistics – Diagnosis 1
Is this an observational or an experimental study?
In your opinion
4
Methods in epidemiology – Medical statistics – Diagnosis 1
Is this an observational or an experimental study?
In your opinion
Which patients are to be studied?
Methods in epidemiology – Medical statistics – Diagnosis 1
The CATSCAN study was designed to prospectively include
patients between the ages of 30 and 70 years, who were
referred for clinically indicated nonemergency coronary
angiography, for evaluation of chest pain, and for intermediate
or high probability of disease.
5
Methods in epidemiology – Medical statistics – Diagnosis 1
� Diagnostic accuracy depends on the severity spectrum of the disease
Selecting the study subjects
� The study population must be consistent with target population
� Diagnostic accuracy is usually different in outpatients and hospitalized subjects
� The main rule is the ‘Intention to diagnose’ and subjects should be selected according to clinical practice (i.e. when disease si suspected) � consecutive patients
� Characteristics of the study subjects should always be reported (sampled population)
Metodologia clinica 5.1
Actual study
Implementation
Data
Study sample
Planning
Target population
Study population
Study question
Study protocol
Study object Variables to be measured
Structure of clinical research
Inference
Parameter Estimate
iθθθθ̂
Study conclusions
Tθθθθ ( )
iSEθθθθθθθθ ˆ=
Systematic errors
Randomand
Systematic errors
6
Methods in epidemiology – Medical statistics – Diagnosis 1
Selection bias
� If the inclusion criteria do not include a patient spectrum
similar to the population in which the test will be used in
practice, the results of the study may have limited
applicability (e.g advanced or hospitalized subjects)
� Differences in disease severity among studies may
lead to differences in estimates of test performance
Methods in epidemiology – Medical statistics – Diagnosis 1
The CATSCAN study was designed to prospectively include
patients between the ages of 30 and 70 years, who were
referred for clinically indicated nonemergency coronary
angiography, for evaluation of chest pain, and for intermediate
or high probability of disease.
We estimated a sample size of 234 patients to achieve 80% power to reject the null hypotheses.
The primary hypothesis tested was whether MDCT could detect more than 50% luminal narrowing in coronary artery segments larger than 2.0 mm in diameter and have a sensitivity of more than 85% and a specificity of more than 85%.
?
7
Methods in epidemiology – Medical statistics – Diagnosis 1
Is this an observational or an experimental study?
In your opinion
Which patients are to be studied?
Which variables are to be measured to evaluate the
diagnostic accuracy?
Main Outcome Measures Segment-based and patient-based sensitivities and
specificities for the detection of luminal stenosis of more
than 50% (of luminal diameter) based on quantitative
coronary angiography.?
Main Outcome Measures Segment-based and patient-based sensitivities and
specificities for the detection of luminal stenosis of more
than 50% (of luminal diameter) based on quantitative
coronary angiography.?
Methods in epidemiology – Medical statistics – Diagnosis 1
Reference standard
It is the ‘true patient status’. Rather it is the best procedure
to assess diagnosis.
Diagnostic tests
Index test
Is the test whose ability to predict the diagnosis we want
to assess
� Angiography
� MDCT
8
Methods in epidemiology – Medical statistics – Diagnosis 1
Is this an observational or an experimental study?
In your opinion
Which patients are to be studied?
Which variables are to be measured to evaluate the
diagnostic accuracy?
How should the tests be executed?
It was anticipated that 70% of the patients enrolled in the study would complete both MDCT and conventional angiography procedures and have adequate MDCT images for interpretation.
Methods in epidemiology – Medical statistics – Diagnosis 1
Reference standard
It is the ‘true patient status’. Rather it is the best procedure
to assess diagnosis.
Diagnostic tests
Index test
Is the test whose ability to predict the diagnosis we want
to assess
� Angiography
� MDCT
� Execution procedures of the study tests should be clearly pre-defined
� All patients must undergo both tests � cross sectional studies
� Test should be performed in a blind way
9
Methods in epidemiology – Medical statistics – Diagnosis 1
Metodologia clinica 5.1
Actual study
Implementation
Data
Study sample
Planning
Target population
Study population
Study question
Study protocol
Study object Variables to be measured
Structure of clinical research
Inference
Parameter Estimate
iθθθθ̂
Study conclusions
Tθθθθ ( )
iSEθθθθθθθθ ˆ=
Systematic errors
Randomand
Systematic errors
10
Methods in epidemiology – Medical statistics – Diagnosis 1
Information bias
Execution
� The index test is performed long before the reference standard, so
the disease is at a more advanced stage when the last one is performed
� Treatment is started on the basis of the results of the index test,
and the reference standard is applied after treatment has started
� Only a selected sample of patients who underwent the index test
is verified by the reference standard.
Interpretation
� Interpretation of the index test or reference standard is influenced
by knowledge of the results of the other test.
� The availability of information on clinical data may affect the
estimates of the test performance
� the uninterpretable results are simply removed from the analysis
All lead to overestimation of diagnostic accuracy
Furukawa TA, Guyatt GH. Sources of bias in diagnostic accuracy studies and the diagnostic process. CMAJ 2006; 174(4):481-2.
11
Methods in epidemiology – Medical statistics – Diagnosis 1
Conclusions
The results of this study indicate that MDCT coronary
angiography performed with 16-row scanners is limited by a
high number of non evaluable cases and a high false-
positive rate. Thus, its routine implementation in clinical
practice is not justified. Nevertheless, given its high
sensitivity and negative predictive value, 16-row MDCT may
be useful in excluding coronary disease in selected patients
in whom a false-positive or inconclusive stress test result is
suspected.
How much is this conclusion supported by data?
Methods in epidemiology – Medical statistics – Diagnosis 1
Diagnosis
What is diagnosis?
Methods in Epidemiology
12
Methods in epidemiology – Medical statistics – Diagnosis 1
“An effort to recognize the class or group to
which a patient’s illness belongs so that,
based on our previous experience with that
class, the subsequent clinical acts we can
afford to carry out (and the patient is willing to
follow) will maximize that patient’s health”
(Sackett DL et al, Clinical Epidemiology 1985)
Diagnosis
Methods in epidemiology – Medical statistics – Diagnosis 1
(Presumed) healthy
Reference
interval
Diagnosis
13
Methods in epidemiology – Medical statistics – Diagnosis 1
Reference
interval
Who are they?
Healthy people with too
high or too small values
� False positive
Healthy
Diagnosis
Methods in epidemiology – Medical statistics – Diagnosis 1
Sick
Reference
interval
Diagnosis
Healthy
14
Methods in epidemiology – Medical statistics – Diagnosis 1
Reference
interval
Diagnosis
SickHealthy
Methods in epidemiology – Medical statistics – Diagnosis 1
Reference
interval
Who are they?
Sick people with ‘normal’
values � False negative
Diagnosis
SickHealthy
15
Methods in epidemiology – Medical statistics – Diagnosis 1
Reference
interval
Diagnosis
SickHealthy
16
“We cannot know that which we do not already know and are unlikely to find that which we do not suspect”
(Burnum JF, Giving meaning to sign, Ann Intern Med 1993)
Methods in epidemiology – Medical statistics – Diagnosis 1
Diagnosis
How to evaluate diagnostic
accuracy
Methods in Epidemiology
17
At the end of the lecture students should be able
� to discuss the meaning of normality
� to report main biases in the evaluation of diagnostic tests
� to calculate main measures of accuracy of diagnostic
tests (sensitivity, specificity, positive and negative predictive
value)
� to interpret likelihood ratios and ROC curves when
assessing accuracy of a diagnostic test
� to measure reliability of diagnostic tests
Medical statistics
Methods in epidemiology – Medical statistics – Diagnosis 1
How to measure the ability of the test to predict the
correct diagnosis?
� sensitivity
� specificity
� positive predictive value (PPV)
� negative predictive value (NPV)
� likelihood ratio (LR)
� area under the ROC curve
Diagnostic accuracy
18
Methods in epidemiology – Medical statistics – Diagnosis 1
Diagnostic accuracy
Sensitivity and Specificity
Methods in Epidemiology
Methods in epidemiology – Medical statistics – Diagnosis 1
19
Methods in epidemiology – Medical statistics – Diagnosis 1
> 50%
> 50% ≤ 50% Total
≤ 50%
TP FP
FN TN
Please fill in the contingency table with the data of the
previous slide
MDCT
Segment-based analysis (nonevaluable = positive)
Angiography
Methods in epidemiology – Medical statistics – Diagnosis 1
MDCT
> 50%
> 50% ≤ 50%
79 544
10 996
89 1.540
623
1.006
1.629
≤ 50%
TP FP
FN TN
Sensitivity = TP/(TP+FN) = 79/89 = 89%
P (T+|||| D+)
Segment-based analysis (nonevaluable = positive)
Angiography
Total
20
Methods in epidemiology – Medical statistics – Diagnosis 1
> 50%
> 50% ≤ 50%
79 544
10 996
89 1.540
623
1.006
1.629
≤ 50%
FP
FN
Specificity = TN/(TN+FP) = 996/1.540 = 65%
P (T-|||| D-)
Segment-based analysis (nonevaluable = positive)
MDCT
Angiography
TP
TN
Total
Methods in epidemiology – Medical statistics – Diagnosis 1
21
Methods in epidemiology – Medical statistics – Diagnosis 1
> 50%
> 50% ≤ 50%
55 96
10 996
65 1.092
151
1.006
1.157
≤ 50%
FP
FN
Sensitivity = TP/(TP+FN) = 55/65 = 85%
Specificity = TN/(TN+FP) = 996/1.092 = 91%
Segment-based analysis (nonevaluable excluded)
MDCT
Angiography
TP
TN
Total
Methods in epidemiology – Medical statistics – Diagnosis 1
n
ppp
)1(96.1
−±
)06.089.0
89
)11.0)(89.0(96.189.0 ±=±
22
Methods in epidemiology – Medical statistics – Diagnosis 1
Methods in epidemiology – Medical statistics – Diagnosis 1
> 50%
> 50% ≤ 50%
58 58
1 70
59 128
116
71
187
≤ 50%
FP
FN
Sensitivity = TP/(TP+FN) = 58/59 = 98%
Specificity = TN/(TN+FP) = 70/128 = 54%
Patient-based analysis (nonevaluable = positive)
MDCT
Angiography
TP
TN
Total
23
Methods in epidemiology – Medical statistics – Diagnosis 1
> 50%
> 50% ≤ 50%
44 29
15 99
59 128
73
114
187
≤ 50%
FP
FN
Sensitivity = TP/(TP+FN) = 44/59 = 75%
Patient-based analysis (nonevaluable excluded)
Specificity = TN/(TN+FP) = 99/128 = 77%
MDCT
Angiography
TP
TN
Total
Methods in epidemiology – Medical statistics – Diagnosis 1