Prof Yong Man Ro

Deep Learning- Automatic Malignancy

Detection

Yong Man Ro (ymro@kaist.ac.kr)

Image and Video sYstems lab. (http://ivylab.kaist.ac.kr)

School of Electrical Engineering, KAIST

2016.09.12

Prof Yong Man Ro

Contents

� Deep learning based breast mass classification

� Latent feature representation with depth directional long-term recurrent

learning for single-view analysis

� Latent feature representation with 3D multi-view deep CNN for bilateral

analysis

� Recent publications related to medical deep learning in Prof. Ro Lab.

� Latent feature representation with depth directional long-term recurrent learning for breast masses in

digital breast tomosynthesis, Medical Physics, vol.62, no.3, pp.1009–1031 , 2017

� Latent feature representation with 3-D Multi-view Convolutional Neural Network for Bilateral Analysis in

Digital Breast Tomosynthesis ,”, IEEE ICASSP, 2016

� Region matching based on local structure information in ipsilateral digital breast tomosynthesis

views,”,IEEE ICIP 2015

� Detection of masses in digital breast tomosynthesis using complementary information of simulated

projection," Medical Physics, vol. 42, pp. 7043 2015

� Breast mass detection using slice conspicuity in 3D reconstructed digital breast volumes," Physics in

Medicine and Biology, vol. 59, pp. 5003 2014

Prof Yong Man Ro

Automatic mass detection via visual recognition

Feature space

Maximum

margin

Mass

Normal

Support

vectors

Example of SVM classification

Positive class Negative class

Sp

ars

e

coef

fici

ent

Example of sparse representation based classification

A test mass image

Sparse representation

Apparent spiculated patternsSubtle spiculated patterns

A general framework

of automatic mass

detection

Prof Yong Man Ro

Mass detection via deep learning

Deep learning do this tasks

Prof Yong Man Ro

Breast cancer screening :

2D DM and 3D DBT

� 3D DBT clearly shows breast cancers

2D DM 3D DBT

Mass

2D DM

Microcalcification

3D DBT

Prof Yong Man Ro

Latent feature representation

with depth directional long-term recurrent learning

Related publication:

Latent feature representation with depth directional long-term recurrent

learning for breast masses in digital breast tomosynthesis, Yong Man Ro

etal. Medical Physics, revising, 2016

Prof Yong Man Ro

Overview of the proposed method

� Variation of depth directional texture patterns

� Slices of masses show similar texture patterns, while FPs show different

texture patterns among slices (as a medical doctor’s diagnosis)

�Because FPs are occurred when tissues in different depth are overlapped

Example of breast cancer

screening using DBT [2]Mass

Prof Yong Man Ro

� Encoding scheme for masses via the proposed latent feature

representation

Slices of

the VOI

Encoding spatial structures

3D DBT volume

……

Top

Bottom

Central

…

CNN

(2D spatial)

CNN

(2D spatial)

LSTM

(Depth direction)

LSTM

(Depth direction)

LSTM

(Depth direction)

…

Shared

weights

Top-to-central slices Bottom-to-central slices

Latent feature

representation

VOI

Encoding texture variations in

top-to-central slice direction and

bottom-to-central slice direction

Encoding central slice directional

symmetric patterns

Depth

direction

Latent feature representation with depth directional long-term recurrent learning for breast masses in

digital breast tomosynthesis, Yong Man Ro etal. Medical Physics, revising, 2016

Prof Yong Man Ro

� Central slice directional learning using depth directional long-term

recurrent learning

� Modelling the symmetric pattern of slice feature representations of masses

with respect to the central slice

9/29

Structure of LSTM layers

Latent feature representation with depth directional long-term recurrent learning for breast masses in

digital breast tomosynthesis, Yong Man Ro etal. Medical Physics, revising, 2016

Prof Yong Man Ro

Visualization of activations on the learned LSTM layer10/29

Masses

FPs

Evolution of states of three gates and a memory cell and an output (i.e.,

latent feature representation) for 4 LSTM cells at last LSTM layer

Slice index

Prof Yong Man Ro

Experiment 1

� Classification performance comparisons with existing methods

11/29

Feature group AUC

Hand-crafted features [1] 0.847

Slice feature representation 0.871

Proposed latent feature representation 0.919

[1] Kim D H, Kim S T and Ro Y M 2015 Improving mass detection using combined feature representations from projection views and reconstructed volume of DBT and boosting based

classification with feature selection Phys. Med. Biol. 60 8809

ROC curve

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

False positive fraction (FPF)

Tru

e p

osi

tiv

e fr

acti

on

(T

PF

)

Hand-crafted features

Slice feature representation

Proposed latent feature representation

Prof Yong Man Ro

Bilateral analysis:

Latent feature representation with 3D multi-view CNN

Related publication:

Latent feature representation with 3-D Multi-view Convolutional Neural Network for

Bilateral Analysis in Digital Breast Tomosynthesis ,”, Yong Man Ro IEEE ICASSP,

2016

Prof Yong Man Ro

Clinical practice: Bilateral analysis

� Bilateral analysis for breast cancer screening and diagnosis

� Mass is an asymmetric density which is visible on two projections (CC and MLO)

� Asymmetry between the left and right breast of a given subject is an important sign

used by radiologists to diagnose breast cancer

Example of reconstructed slices including

a mass (white circled)

RMLO LMLO

Asymmetric density

Prof Yong Man Ro

Conventional: hand-crafted features in DBT

� Measuring the bilateral dissimilarity between two VOIs in left

breast and right breast [1]

� Texture dissimilarity (i.e., Sum of squared differences (SSD) measure)

� Dissimilarity of intensity distribution (i.e., Histogram correlation)

� Dissimilarity of mass characteristics between VOIs (i.e., Absolute difference of single

features)

� Limitation

� Bilateral characteristics are abstract

� Due to the subtle characteristics

of masses in bilateral analysis,

it is hard to design effective

hand-crafted features

T

(transform)VOI in source

image (i.e. left)

r Registered

VOI

LMLO RMLO�� ��

T(r)

Estimating dissimilarity

between two VOIs

Illustration of hand-crafted

bilateral feature extraction

[1] “Feature extraction from bilateral dissimilarity in digital breast tomosynthesis reconstructed volume,” IEEE international conference on image processing, 2015.

Prof Yong Man Ro

Bilateral analysis with multi-view 3D CNN

� 3D CNN learns tissue structure in volume

� Multi-view fusion network learns different representation of VOIs as input

and learns features individually

“Latent feature representation with 3-d multi-view deep convolutional neural network for bilateral analysis in digital breast tomosynthesis,” IEEE International Conference on Acoustics, Speech and

Signal Processing (ICASSP), 2016.

Prof Yong Man Ro

Experimental results

Comparisons of ROC curves of FP reduction using hand-crafted

features and proposed latent bilateral feature representation

“Feature extraction from bilateral dissimilarity in digital breast tomosynthesis reconstructed volume,” IEEE international conference on image processing, 2015.

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

False positive fraction (FPF)

Tru

e p

osi

tiv

e fr

acti

on

(T

PF

)

Hand-crafted features [1] (AUC=0.826 ± 0.013)

Latent bilateral features (AUC=0.847 ± 0.012) t-SNE feature visualization

Original input

data

FPs

Masses