10-11Sensitivity, Specificity Biostatistics and Research Design, #6350 Screening, Diagnostic Accuracy (sensitivity & specificity)

Sensitivity, Specificity10-11

Biostatistics and Research Design, #6350

Screening, Diagnostic Accuracy(sensitivity & specificity)


Thought for the Day:

“…The arts and sciences, and a thousand appliances…, but the

wind that blows is all that anybody knows”

Henry David Thoreau

Sensitivity, Specificity

Learning Concepts: Screening

• Understand sensitivity and specificity of a diagnostic test

• Be able to calculate sensitivity and specificity

• Be able to use the concepts of sensitivity and specificity in clinical decision making

NB: from Latin: nota bene; means “good note”

10-11


Why Screen for Disease?

• Early detection --> Early treatment• Access into health care system• Not everyone gets routine care• Community service• Practice builder

10-11


Why Screen? (early diagnosis)

Biologic onset

of disease

Disease detectable by screening test

Detection by screening test

Disease detectable by routine methods

Morbidity (death)

10-11


What a screening is / What a screening is not

• IS: An indication of a problem:– Cost-effective– Rapid

• IS NOT: Completely diagnostic:– Over-referrals and under-referrals– Not 100% accurate– Not a substitute for regular health care

10-11


Special Notes:

For a screening to be effective:

1. Need a system in place to handle referrals

2. The condition being screened for must be treatable

10-11


Screening Programs at SCCO:School Screening

• CA state law since 1947• 1st, 3rd, and 6th grades• 1971 minimum intervals for conducting a

screening• Only legally mandated tests:

– Snellen visual acuity – Color vision testing for boys

10-11


Screening Programs at SCCO:Special Events

• Save Your Vision Week• Back to School Open House• Community screening programs:

–Regular school screenings–Senior centers (IOP)

• Special Olympics

10-11


The Orinda Study (overview; more later)

• Screened children in grades 1 - 8• Ages 5 - 13• Total of 1,163 children screened• Goals: To design the least expensive, least

technical and most effective screening program

10-11


The Orinda Study

Modified Clinical Technique (MCT) consists of:

• visual acuity• retinoscopy• cover test• color vision• ophthalmoscopy

10-11


The Orinda Study

• Results:The Modified Clinical Technique (MCT) is effective in identifying more than 90% of those with vision problems

• Test Positive, Disease Positive; sensitivity = 90%

10-11


The Orinda Study

• What about inclusion of other tests?• Why or why not:

– visual field– tonometry– subjective refraction– blood pressure

10-11


Efficacy of Diagnostic Tests or Methods: General

• As clinicians (or researchers), we need to use tests that detect the disease or condition well, while properly classifying those without the condition– NB: can’t use the test under consideration to

assign as affected or normal • Concepts: > 3:

– Sensitivity– Specificity– Receiver Operating Characteristic (ROC)

10-11


Sensitivity

• Accuracy of the screening procedure to correctly identify all individuals in a population who have a particular disorder

• NB: Newer terminology = Detection Rate

10-11



Sensitivity• Out of all of the people who have the

disorder, how many does your screening test correctly identify?

• True positives• Mnemonic?

– “Test positive, disease positive”– “Sensitive to disease”

10-11


Basic Setup for a 2 2 Contingency Table*(sensitivity and specificity)

Test Disease

Positive (D+) Negative (D-)

Positive Test TP(true positive)

FP (false positive)

Negative Test FN(false negative)

TN(true negative)

(total truly affected)

(total truly unaffected)

* Also called a “confusion matrix” (Wikipedia, 2010 Onward)10-11


Sensitivity: Example

• Van Bjisterveld OP, Diagnostic Tests in the Sicca Syndrome. Arch Ophthalmol; 82:10-14, 1969

• Rose bengal staining • 550 normals• 43 dry eye patients• NB: Both eyes included, but this artificially

inflates the statistical significance since the eyes are not independent for this condition– Also, limited information as to how the “drys” were

classified

10-11


Rose Bengal Staining

10-11


Sensitivity: Example, Rose Bengal*

Test Disease Totals

Positive (D+)

Negative (D-)

Positive Test

82 40 122

Negative Test

4 1060 1064

Totals 86 1100 1186

* Using cut-off value of 3.5 out of 9 possible Sensitivity = 0.95 (82/86)

10-11


Specificity

• Accuracy of the screening procedure to correctly identify those who do not have the disorder

• Mnemonics?– “Test negative, disease negative”– “Specific to health”

10-11


Specificity

• Out of all of those who do not have the disorder, how many does your screening correctly identify?

• True negatives• Implication:

– if specificity = 0.90, 10% of normals will be referred for care

– if specificity = 0.94, 6% of normals will be referred for care, etc.

10-11


Specificity: Example, Rose Bengal*

Test Disease Totals

Positive (D+)

Negative (D-)

Positive Test

82 40 122

Negative Test

4 1060 1064

Totals 86 1100 1186

* Using cut-off value of 3.5 out of 9 possible

Specificity = 0.96 (1060/1100)10-11


Sensitivity and Specificity

• Generally inversely related• Cannot usually have 100% for both (but

you CAN maximize both, as we have just observed)

10-11


Comparison of Specific and Sensitive tests

Sensitive tests:

• few false-negatives• for serious but treatable conditions• test people without complaints

Specific tests:

• few false-positives• for conditions with

serious misdiagnosis consequences

• confirm a suspected diagnosis

10-11


Example*: Test Needs to be Specific and Sensitive

• HIV/AIDS: screening test = detect antibodies to virus (ELISA assay)

• Sensitivity: 72/74 who were HIV positive; (97% sensitivity)– Inappropriate reassurance to an infected person:– Delays treatment, increases spread of virus?

• Specificity: 257/261 healthy persons (98% specificity)– News of infection could be devastating to a healthy

individual

*Greenberg, RS, et al. Medical Epidemiology, 3rd Ed., pp. 7-8, 2110, McGraw Hill, New York.10-11


ROC Curves

• Background: developed during WW II for radar: how to best detect enemy aircraft

• Plot: true positives and false positives (1 – specificity)

• Can use differing tests or combinations of several tests to provide the largest AUC– Close to 1.00 is best

• Bottom Line: Another test metric

10-11


ROC Curves: Example: Tear Film Thickness to Dx Dry eye

Maximum Sensitivity for DE (0.86) and Specificity (0.94)if tear thickness < 2.75 micrometers10-11

Diagnostic Tests for MGD: Paugh’s Pearls Clinical Test Scale Range Cut Point Sensitivity Specificity Area

Under Curve

Surface Regularity Index

0 – 1.5+ 0.57 75% 73% 0.796

Lid Margin Evaluation

0 - 4 1.5 83% 84% 0.908

Tear Break Up Time 0 – 10+ 6.2 seconds 85% 85% 0.908

NaFl Staining (Oxford)

0 - 20 6.0 73% 68% 0.813

Meibomian Gland Expression Lower Lid

0 - 3 1.1 74% 70% 0.786

Meibomian Gland Expression Upper Lid

0 - 3 1.2 76% 65% 0.774

Meiboscopy 0 - 4 0.6 72% 73% 0.778

MGD Score 0 - 11 3.0 87% 83% 0.929


Guidelines for Selecting a Diagnostic Test

• Has there been an independent masked comparison with a gold standard of diagnosis? (e.g., an autorefractor vs. subjective)

• Has the diagnostic test been evaluated in a patient sample that included an appropriate spectrum of mild and severe, treated and untreated disease, plus individuals with different but similar disorders?

10-11


Concepts in Action: The Vision in Preschoolers (VIP) Study

• Preschool screening is a major policy issue at the state and national level

• Screenings mandated in most states, some even comprehensive eye exams for kids – e.g., Kentucky

Determine the best methods to screen for major eye conditions by nurses and lay personnel

10-11


VIP: Details, Phase I*• Phase I: ODs and OMDs screened

Headstart children vs. comprehensive eye exam (over represent vision problems)

• 4 major conditions: amblyopia, strabismus, sig. ref. error and unexplained VA loss

• Goal: compare 11 screening tests vs. exam: which are most sensitive?

• Strategy: set specificity at 90% (10% over-referral), what is sensitivity of screeners?

* VIP Study Group, Ophthalmol 2004;111:637-65010-11


VIP (LEPs): Phase I Results

• Best overall sensitivity: • Ref Error: Non-cycloplegic retinoscopy = 63%• Ref Error: SureSight screener = 63%• Ref Error: Retinomax screener = 63%• VA: Lea symbols test = 61%• Sensitivity of most important to detect:

– refractive error: severe anisom., hyperopia > 5D, astig. > 2.5D, myopia > 6D:

80-90%

10-11


VIP Phase II: Nurses vs. Lay People*

• n = 1452 total Headstart preschoolers: – Age: 3 to < 5 yrs– n = 990 normals – n = 462 with vision conditions

• All preschoolers had gold std. exams• Used best automated refractors from Phase I:

(Retinomax, Suresight) plus Lea Symbols (VA) and Stereo Smile II (stereo acuity)

• Specificity set at 0.90 (10% over-referral)

* VIP Study Group, IOVS 2005;46:2639-264810-11


VIP Phase II: Results

• Overall, nurse screeners had slightly higher sensitivities, but not statistically significant– Also: both groups similar to licensed docs

• E.g., for Group I (most severe conditions):• Autorefractors:

– Nurses: sensitivity = 0.83 - 0.88– Lay: sensitivity = 0.82 – 0.85

• Stereo Smile II:– Nurses: sensitivity = 0.58– Lay: sensitivity = 0.56

10-11


“Whether a test should be used or not…”

10-11

Documents

10-11Sensitivity, Specificity Biostatistics and Research Design, #6350 Screening, Diagnostic Accuracy (sensitivity & specificity)