Upload
john-doyle
View
219
Download
1
Tags:
Embed Size (px)
Citation preview
Developing a Validated Tool For Evaluating Whole Slide Images
University of PittsburghDepartment of Biomedical Informatics
Authors
Dana Grzybicki, MD, PhDEvaluation Team – Principal Investigator
Russell Silowash, BSEvaluation Team – Research Analyst
Robb Wilson, MAEvaluation Team – Project Manager
Leslie Anthony, MAUPMC IMITS Telepathology Project – Project Manager
University of PittsburghDepartment of Biomedical Informatics
Background• Through appropriations in the
defense-spending bills for 2002 and 2004, the University of Pittsburgh Medical Center (UPMC) and the United State Air Force Medical Service (AFMS) created a partnership called the Integrated Medical Information Technology System (IMITS) Program
• Telepathology is a branch of the IMITS program that implements and validates digital pathology practices
University of PittsburghDepartment of Biomedical Informatics
Introduction
• Purpose of evaluation research– Independent examination of questions related to:
• Technical validity• Feasibility• Effectiveness
• To our knowledge, there are no generally available validated tools for evaluating WSI cases
• The UPMC Digital Pathology Imaging Group is working on the validation of a unique evaluation tool
University of PittsburghDepartment of Biomedical Informatics
Participants
• The validation of this tool was part of our telepathology evaluation project involving 5 UPMC pathologists– 2 pathology fellows– 3 staff pathologists with training in GU pathology
University of PittsburghDepartment of Biomedical Informatics
Case Selection
• 30 difficult prostate biopsy foci
Diagnosis Number of Cases
Adenocarcinoma 12
Atypical 6
Atypical PIN (ATYPIN) 1
High Grade PIN 3
Benign 8
University of PittsburghDepartment of Biomedical Informatics
University of PittsburghDepartment of Biomedical Informatics
University of PittsburghDepartment of Biomedical Informatics
Hypotheses
• Content Validity– The assessment is asking the proper questions
for the study at hand
• Internal Validity– There will be a positive correlation between the
number of slides/images in a case and the time needed to complete the case
University of PittsburghDepartment of Biomedical Informatics
Hypotheses (continued)
• External Validity– Whole slide image quality will be positively
correlated with glass slide quality
• Construct Validity– There will be a negative correlation between the
diagnostic confidence of a participant and the case complexity rating
University of PittsburghDepartment of Biomedical Informatics
Content Validity Results
• Content validity has been obtained by gaining feedback from pathologists that are part of the Digital Pathology Imaging Group (DPIG)
University of PittsburghDepartment of Biomedical Informatics
Internal Validity Results
• Statistically significant for only one participant (r2=0.327, p<0.01)
• Our inability to demonstrate internal validity for most of the participants was most likely due to time categorical variables that were too broad
University of PittsburghDepartment of Biomedical Informatics
External Validity Results
Average Image vs Slide Quality Correlation
y = 0.5505x + 0.6673
0
0.5
1
1.5
2
2.5
3
3.5
0 0.5 1 1.5 2 2.5 3 3.5
Image Quality Rating
Slid
e Q
ual
ity
Rat
ing
• A positive correlation exists between WSI quality and glass slide quality
•There were statistically significant positive correlations for 3 of the 5 subjects
University of PittsburghDepartment of Biomedical Informatics
Construct Validity ResultsWSI Average Case Complexity vs Diagnostic Confidence
y = -0.6775x + 3.7803
1
1.5
2
2.5
3
3.5
0.9 1.4 1.9 2.4 2.9
Case Complexity
Co
nfi
de
nc
e In
Dia
gn
os
is
• Negative correlation exists in WSI phase of study between confidence in diagnosis and case complexity
• 3 of 5 pathologists had statistically significant positive correlations in the WSI Phase
University of PittsburghDepartment of Biomedical Informatics
Construct Validity Results (continued)
Glass Average Case Complextiy vs Diagnostic Confidence
y = -0.4087x + 3.3959
0.9
1.4
1.9
2.4
2.9
3.4
0.9 1.4 1.9 2.4 2.9
Case Complexity
Co
nfi
de
nc
e in
Dia
gn
os
is
• Negative correlation exists in the Glass phase between case complexity and diagnostic confidence
• 3 of 5 pathologists had statistically significant positive correlations in the Glass Phase
University of PittsburghDepartment of Biomedical Informatics
Summary
• We are able to establish content, external and construct validity, however internal validity at this point has not been definitively established.– Low internal validity could be due to time
categories being too broad
University of PittsburghDepartment of Biomedical Informatics
Next Steps
• Change the categories for the time variableTime to Complete (Minus
Interruptions)
•Less Than 15 Minutes
•15 – 30 Minutes
•30 – 45 Minutes
•45 – 60 Minutes
•Over 60 Minutes
Time to Complete (Minus Interruptions)
•Less Than 30 Seconds •7 – 10 Minutes
•30 Seconds – 1 Minute •10 – 12 Minutes
•1 – 2 Minutes •12 – 15 Minutes
•2 – 5 Minutes •Over 15 Minutes
•5 – 7 Minutes
• Implement an automatic timing solution
University of PittsburghDepartment of Biomedical Informatics
Next Steps (continued)
• Continue the validation of this tool by obtaining additional data and testing internal validity utilizing the modified time variables
University of PittsburghDepartment of Biomedical Informatics
Funding
• This work was supported by funding from the U.S. Air Force administered by the U.S. Army Medical Research Acquisition Activity, Fort Detrick, Maryland (Award No. W81XWH-04-2-0030 and Contract No. DAMD 17-0302-0017). The content of the information does not imply U.S. Air Force or Government endorsement of factual accuracy or opinion