Assessing the Validity and Reliability of Diagnostic and Screening Tests

Assessing the Assessing the Validity and Validity and Reliability of Reliability of

Diagnostic and Diagnostic and Screening TestsScreening Tests

Adora F. Mendoza – Abat, Adora F. Mendoza – Abat, M.D., CFPM.D., CFP

ScreeningScreening

• The application of a test to people The application of a test to people who are as yet asymptomatic for the who are as yet asymptomatic for the purpose of classifying them with purpose of classifying them with respect to their likelihood of having respect to their likelihood of having a particular diseasea particular disease

• Not diagnostic tests but preliminary Not diagnostic tests but preliminary tests seeking to identify those who tests seeking to identify those who are most likely to have the diseaseare most likely to have the disease

Criteria of Disease Suitable Criteria of Disease Suitable for Screening Program for Screening Program

• 1. The disease should be serious1. The disease should be serious• 2. Early treatment must be 2. Early treatment must be

beneficial in terms of reducing beneficial in terms of reducing mortality and morbidity comparable mortality and morbidity comparable to late treatmentto late treatment

• 3. The prevalence of preclinical 3. The prevalence of preclinical disease must be high in the disease must be high in the population screenedpopulation screened

Biologic Variation of Biologic Variation of Human Populations Human Populations

(Bimodal Curve)(Bimodal Curve)

0

5

10

15

20

25

0 3 6 9 12 15 18 21 24 27

Induration

# of Subj.

Biologic Variation of Biologic Variation of Human Populations Human Populations (Unimodal Curve)(Unimodal Curve)

0

20

40

60

80

100

120

140

<110 120 140 160 >180

S. BP

# of Men

Validity of Screening Validity of Screening TestsTests

• Validity – the ability of the test to Validity – the ability of the test to distinguish between who has a distinguish between who has a disease and who does notdisease and who does not

• 2 Components:2 Components:• 1. Sensitivity – the ability of the test 1. Sensitivity – the ability of the test

to identify correctly those who have to identify correctly those who have the diseasethe disease

• - The proportion of diseased people - The proportion of diseased people who were correctly identified as who were correctly identified as such by the testsuch by the test

Factors that Increase Factors that Increase SensitivitySensitivity

• 1. Lowering the cut-off and making 1. Lowering the cut-off and making criteria for disease definition less criteria for disease definition less stringentstringent

• 2. Parallel Testing – the 2. Parallel Testing – the administration of 2 screening tests administration of 2 screening tests at the same time and persons with a at the same time and persons with a positive result on any of these tests positive result on any of these tests are considered positiveare considered positive

When Should Sensitivity Be When Should Sensitivity Be IncreasedIncreased

• 1. When the disease is serious and a 1. When the disease is serious and a definitive treatment existsdefinitive treatment exists

• 2. When the disease can be spread2. When the disease can be spread• 3. When subsequent diagnostic 3. When subsequent diagnostic

evaluations of positive screening evaluations of positive screening tests are associated with minimal tests are associated with minimal costs and riskscosts and risks

Validity of Screening Validity of Screening TestsTests

• 2. Specificity – the ability of the test 2. Specificity – the ability of the test to identify correctly those who do not to identify correctly those who do not have the diseasehave the disease

• - proportion of non-diseased people - proportion of non-diseased people who are correctly called negative by who are correctly called negative by the test the test

• Note : In order to quantitatively assess Note : In order to quantitatively assess the sensitivity and specificity of a test, the sensitivity and specificity of a test, we must have another source of truth we must have another source of truth with which to compare the test resultswith which to compare the test results

Factors that Increase Factors that Increase SpecificitySpecificity

• 1. Making the criteria for disease 1. Making the criteria for disease definition more stringent or definition more stringent or increasing the cut-off levelincreasing the cut-off level

• 2. Series Testing – an initial 2. Series Testing – an initial screening test is administered and screening test is administered and then, only persons who are positive then, only persons who are positive on this preliminary test will undergo on this preliminary test will undergo a second additional screening a second additional screening procedureprocedure

When Should Specificity Be When Should Specificity Be IncreasedIncreased

• 1. When the costs or risks 1. When the costs or risks associated with further diagnostic associated with further diagnostic techniques are substantialtechniques are substantial

• 2. When the stigma associated with 2. When the stigma associated with having the disease is severehaving the disease is severe

Concept of the Sensitivity Concept of the Sensitivity and Specificity of Screening and Specificity of Screening

ExamsExams• Sensitivity = 80 x Sensitivity = 80 x

100100• 100100• = 80%= 80%

• Specificity = 800 x Specificity = 800 x 100100

• 900900• = 89%= 89%

TestTest Dse.Dse. NoNo

Dse.Dse.TotaTotall

++ 8080 100100 180180

-- 2020 800800 820820

TotaTotall

100100 900900 10010000

Comparison of the Results of a Comparison of the Results of a Dichotomous Test with Actual Dichotomous Test with Actual

Disease StatusDisease StatusSensitivity = TP Sensitivity = TP

x 100 x 100

TP + FNTP + FN

Specificity = TN Specificity = TN x 100 x 100

TN + FPTN + FP

Test Test ResultsResults

W/ Dse.W/ Dse. W/OW/O

Dse.Dse.

++ Have Have dse. & dse. & have (+) have (+) test = test = true (+) true (+) or (TP)or (TP)

No dse. No dse. & have & have (+) test (+) test = false = false negative negative (FP)(FP)

-- Have Have dse. but dse. but have (-) have (-) test = test = false (-) false (-) or (FN)or (FN)

No dse. No dse. and and have (-) have (-) test = test = true (-) true (-) (TN)(TN)

Issues of False PositivesIssues of False Positives

• All people who screened positive are All people who screened positive are brought back to a more brought back to a more sophisticated and more expensive sophisticated and more expensive teststests

• The anxiety and worry induced in The anxiety and worry induced in persons who have been told that persons who have been told that they have tested positivethey have tested positive

Importance of Problems in Importance of Problems in False NegativesFalse Negatives

• Depends on the nature and severity Depends on the nature and severity of the disease being screened for, of the disease being screened for, the effectiveness of available the effectiveness of available intervention measures, and whether intervention measures, and whether the effectiveness is greater if the the effectiveness is greater if the intervention is administered early in intervention is administered early in the natural history of the diseasethe natural history of the disease

Tests of Continuous Tests of Continuous VariablesVariables

Non-DiabeticsDiabetics

High

Low

Blood Sugar

TestTest

Screening Using a Screening Using a Dichotomous TestDichotomous Test

Test Disease Total

Positive Negative

(+) a (True positives)

b (FalsePositives)

a + b (all people with positive tests)

(-) c (False negatives)

d ( True negatives)

c + d ( all people with negative tests)

Two-Stage ScreeningTwo-Stage Screening

• First Stage – less expensive, less First Stage – less expensive, less invasive, or less uncomfortable testinvasive, or less uncomfortable test

• - screened positive are recalled for - screened positive are recalled for further testingfurther testing

• Second Stage – more expensive, more Second Stage – more expensive, more invasive which may have greater invasive which may have greater sensitivity and specificitysensitivity and specificity

• - reduces the problem of false - reduces the problem of false positivespositives

Two Stage Screening Two Stage Screening ProgramProgram

• Test 1 (Blood Sugar)Test 1 (Blood Sugar)• Assume: Disease Prevalence = 5%Assume: Disease Prevalence = 5%• Population = 10, 000Population = 10, 000• Test 1 (Blood Sugar) : Sensitivity = 70%Test 1 (Blood Sugar) : Sensitivity = 70%• Specificity = 80%Specificity = 80%


DiseaseDisease TotalTotal

(+)(+) (-)(-)

(+)(+) 350350 1,9001,900 2,2502,250

(-)(-) 150150 7,6007,600 7,7507,750

TotalTotal 500500 9,5009,500 10,00010,000

Two Stage Screening Two Stage Screening • Test 2 (OGTT)Test 2 (OGTT)• Given:Given:

Sensitivity = 90%Sensitivity = 90%

Specificity = 90%Specificity = 90%• Solution :Solution :

Net SensitivityNet Sensitivity

= (315/500)(100)= 63%= (315/500)(100)= 63%

Net Specificity Net Specificity

= [ (7600+1710) / = [ (7600+1710) / 9500] [100]9500] [100]

= 98%= 98%

TestTest Dse.Dse. -Dse-Dse TotalTotal

++ 350350 19019000

22502250

-- 150150 76076000

77507750

TotaTotall

500500 95095000

1000010000TestTest Dse.Dse. -Dse-Dse TotaTota

ll

++ 315315 190190 505505

-- 3535 17117100

17417455

TotaTotall

350350 19019000

22522500

Predictive Value of a TestPredictive Value of a Test

• The proportion of people correctly The proportion of people correctly identified by the test to have a diseaseidentified by the test to have a disease

• Positive Predictive Value – the Positive Predictive Value – the proportion of patients who test proportion of patients who test positive and who have actually the positive and who have actually the disease in questiondisease in question

• - divide the number of true positives by - divide the number of true positives by the total number who tested positive the total number who tested positive (true positives + False positives)(true positives + False positives)

• Negative Predictive Value – the Negative Predictive Value – the probability that those who tested probability that those who tested negative do not have the diseasenegative do not have the disease

• - divide the number of true negatives - divide the number of true negatives by all those who tested negative by all those who tested negative (true negatives + false negatives)(true negatives + false negatives)

Predictive Value of a TestPredictive Value of a Test

• Positive Predictive Value = (80/180) x 100 = Positive Predictive Value = (80/180) x 100 = 44%44%

• Negative Predictive Value = (800/820) x 100= Negative Predictive Value = (800/820) x 100= 98%98%


DiseaseDisease No Dse.No Dse. TotalTotal

PositivePositive 8080 100100 180180

NegativeNegative 2020 800800 820820

TotalTotal 100100 900900 1,0001,000

Relation of Disease Relation of Disease Prevalence to Predictive Prevalence to Predictive

ValueValueEx.; Sensitivity = 99% Specificity = 95%Ex.; Sensitivity = 99% Specificity = 95%

Dse. Test Sick Not Totals Dse. Test Sick Not Totals Predictive Predictive

Prevalence Results Sick Prevalence Results Sick Value Value

1% + 99 495 5941% + 99 495 594

- 1 9405 9406- 1 9405 9406

Totals 100 9900 10000Totals 100 9900 10000

5% + 495 475 9705% + 495 475 970

- 5 9025 9303- 5 9025 9303

Totals 500 9500 10000Totals 500 9500 10000

Relation of Disease Relation of Disease Prevalence to Predictive Prevalence to Predictive

ValueValueEx.; Sensitivity = 99% Specificity = 95%Ex.; Sensitivity = 99% Specificity = 95%

Dse. Test Sick Not Totals Dse. Test Sick Not Totals Predictive Predictive

Prevalence Results Sick ValuePrevalence Results Sick Value

1% + 99 495 594 (99/594) x 1% + 99 495 594 (99/594) x 100 = 17%100 = 17%

- 1 9405 9406 - 1 9405 9406 (9405/9406) x 100=100% (9405/9406) x 100=100%

Totals 100 9900 10000 Totals 100 9900 10000

5% + 495 475 970 (495/970) 5% + 495 475 970 (495/970) x 100 = 51%x 100 = 51%

- 5 9025 9303 - 5 9025 9303 (9025/9303) x 100 = 97% (9025/9303) x 100 = 97%

Totals 500 9500 10000 Totals 500 9500 10000

Relation of Predictive Value Relation of Predictive Value to the Specificity of the Testto the Specificity of the Test

• Prevalence = 50%Prevalence = 50%• Sensitivity = 50%Sensitivity = 50%• Specificity = 50%Specificity = 50%• Predictive Value?Predictive Value?


TestTest ++ -- TotaTotall

++ 250250 250250 500500-- 250250 250250 500500

TotaTotall

500500 500500 10010000TestTest ++ -- TotaTotall

++ 100100 400400 500500

-- 100100 400400 500500

TotaTotall

200200 800800 10010000





++ 180180 400400 580580

400400 420420 -- 2020


++

100100 8080 180180

720720 870870 --

100100


InterpretationInterpretation• Sensitivity = 90%Sensitivity = 90%

-90% of those with the disease will be labeled -90% of those with the disease will be labeled as positive by the testas positive by the test

• Specificity = 90%Specificity = 90%- 90% of those without the disease will be 90% of those without the disease will be

labeled as negative by the testlabeled as negative by the test

Positive Predictive Value = 75%Positive Predictive Value = 75%

75% of those who test positive will really 75% of those who test positive will really have the diseasehave the disease

Negative Predictive Value = 75%Negative Predictive Value = 75%

75% of those who test negative will really not 75% of those who test negative will really not have the diseasehave the disease

Reliability (Repeatability of Reliability (Repeatability of Tests)Tests)

• Regardless of the specificity and Regardless of the specificity and sensitivity, if the test results cannot sensitivity, if the test results cannot be reproduced, the value and be reproduced, the value and usefulness of the test are minimal.usefulness of the test are minimal.

Factors that Contribute to Factors that Contribute to Variation in Test ResultsVariation in Test Results

• 1. Intrasubject Variation – variation 1. Intrasubject Variation – variation between individual subjectsbetween individual subjects

• - In evaluating any test result, it is - In evaluating any test result, it is important to take into consideration the important to take into consideration the conditions under which the test was conditions under which the test was carried out, including the time of day.carried out, including the time of day.

• 2. Interobserver Variation – variation 2. Interobserver Variation – variation between those reading the test resultsbetween those reading the test results

Relation Between Validity Relation Between Validity and Reliabilityand Reliability

Exercise 1Exercise 1

• Calculate for the sensitivity, specificity, Calculate for the sensitivity, specificity, positive predictive value, and negative positive predictive value, and negative predictive valuepredictive value

• Interpret your answersInterpret your answers

THROAT CULTURETHROAT CULTURE


With With DiseaseDisease

W/O Dse.W/O Dse. TotalTotal

++ 2727 3535 6262-- 1010 7777 8787

TotalTotal 3737 112112 149149

Exercise 2Exercise 2

• Complete the table with needed dataComplete the table with needed data• Solve for the positive predictive Solve for the positive predictive

value and interpret your answervalue and interpret your answer• State the relationship of predictive State the relationship of predictive

value and specificity using the value and specificity using the positive predictive valuepositive predictive value

• Solve for the negative predictive Solve for the negative predictive value and interpret your answervalue and interpret your answer

Relationship of Specificity Relationship of Specificity to Predictive Valueto Predictive Value

Ex. Prevalence = 10% ; Sensitivity = Ex. Prevalence = 10% ; Sensitivity = 100%100%

Specificity Test Sick Not Totals Specificity Test Sick Not Totals Predictive Predictive

Results Sick Results Sick ValueValue

70% +70% +

- -

Totals 10000Totals 10000

95% +95% +

--

Totals 10000 Totals 10000

Documents

Assessing the Validity and Reliability of Diagnostic and Screening Tests