#CMIMI18#CMIMI18

Joint Classification and Localization of Critical Findings in Chest X-ray using Deep Multi-Instance Transfer Learning

Evan Schwab, PhDPhilips Research North America

#CMIMI18

Goal: Use AI to Assist Radiologists in Diagnostics

#CMIMI183 CISS

1. CXR lack annotations (e.g. local bounding boxes).

2. Unlike pictures of dogs, difficult to validate correctness without experts.

3. Deep learning networks commonly non-interpretable

Objectives Constraints

1. Automatically classify CXR (No Finding vs Pneumothorax).

2. Automatically localize abnormality in CXR (if present).

3. Classify and localize jointly. (Want to end-to-end solution).

Medical images lack local annotations

#CMIMI184

Wang, Xiaosong, et al. "Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases." CVPR, 2017.

State-of-the-Art: Grad-CAM CXR Saliency Maps

#CMIMI186

1024 x 1024 224 x 224

Pre-Trained Network (VGG16)

224 x 224

7x7

** Images Drawn to Scale

DownSample

FeedInto

Grad-Cam Heatmap

UpSample

#CMIMI187

1024 x 1024 224 x 224

Pre-Trained Network (VGG16)

224 x 224

7x7

** Images Drawn to Scale

Extract FeedInto

UpSample

Grad-Cam Heatmap

#CMIMI188

Which key opens the door?

Multi-Instance Learning (MIL)

#CMIMI189

Which patch contains Pneumothorax?

Multi-Instance Learning (MIL)

No Finding Pneumothorax Pneumothorax

#CMIMI1810

1024 x 1024

N x 224 x 224N Patches

Pre-Trained Network (VGG16)

.001 .92 .24

Sigmoid Patch Scores

Max ScoreOutput:

1. Predicted Bag Label: Pneumothorax with 92%2. Max Patch Location

Proposed Deep MIL with Transfer Learning

#CMIMI1811

Observed (+) Label

(+) (+) (+)

Observed (-) Label

(-) (-) (-)N Patches form 1 CXR fed as 1 training batch

N Patches form 1 CXR fed as 1 training batch

Sigmoid Layer Nx1

Max Layer 1x1

Nx224x224

Training Deep MIL with Transfer Learning

Pre-Trained Network (VGG16)

#CMIMI18

Data: – NIH CXR dataset– Subset of data has ground truth bounding box of Pneumothorax (PTX)– Binary Classification: PTX vs No Finding– 60 Subjects: 30 Subjects PTX, 30 Subjects No Finding– Ground truth bounding boxes only used for visual verification at the end– Divide data classes evenly into 2/3 Training 1/3 Validation

Setup:– Accuracy/AUC given by correct image classification– Stochastic Gradient Descent, Decay, Momentum– Transfer Learning: Freeze first 15 layers of VGG16– Add Sigmoid Patch Score Layer– Add Final Layer:

• Max

• Max Sum over Neighborhood

• Log Sum Exponential (LSE), (parameter r)

12

Preliminary Results

Final Layer AUC

Max 0.67

Max Sum 0.75

LSE (r=1) 0.89

LSE (r=2) 0.89

LSE (r=5) 0.94

LSE (r=10) 0.67

#CMIMI1813CISS

Epoch 1

Epoch 10

Patch Scores/Locations for 10 Epochs

#CMIMI1814 CISS/Neuro

Epoch 1

Epoch 10

Patch Scores/Locations for 10 Epochs

#CMIMI1815 CISS

Epoch 1

Epoch 10

Patch Scores/Locations for 10 Epochs

#CMIMI18

Thank you!

Questions/Comments Contact: evan.schwab@philips.com

#CMIMI18