View
224
Download
4
Category
Preview:
Citation preview
Prof Yong Man Ro
Deep Learning- Automatic Malignancy
Detection
Yong Man Ro (ymro@kaist.ac.kr)
Image and Video sYstems lab. (http://ivylab.kaist.ac.kr)
School of Electrical Engineering, KAIST
2016.09.12
Prof Yong Man Ro
Contents
� Deep learning based breast mass classification
� Latent feature representation with depth directional long-term recurrent
learning for single-view analysis
� Latent feature representation with 3D multi-view deep CNN for bilateral
analysis
� Recent publications related to medical deep learning in Prof. Ro Lab.
� Latent feature representation with depth directional long-term recurrent learning for breast masses in
digital breast tomosynthesis, Medical Physics, vol.62, no.3, pp.1009–1031 , 2017
� Latent feature representation with 3-D Multi-view Convolutional Neural Network for Bilateral Analysis in
Digital Breast Tomosynthesis ,”, IEEE ICASSP, 2016
� Region matching based on local structure information in ipsilateral digital breast tomosynthesis
views,”,IEEE ICIP 2015
� Detection of masses in digital breast tomosynthesis using complementary information of simulated
projection," Medical Physics, vol. 42, pp. 7043 2015
� Breast mass detection using slice conspicuity in 3D reconstructed digital breast volumes," Physics in
Medicine and Biology, vol. 59, pp. 5003 2014
Prof Yong Man Ro
Automatic mass detection via visual recognition
Feature space
Maximum
margin
Mass
Normal
Support
vectors
Example of SVM classification
Positive class Negative class
Sp
ars
e
coef
fici
ent
Example of sparse representation based classification
A test mass image
Sparse representation
Apparent spiculated patternsSubtle spiculated patterns
A general framework
of automatic mass
detection
Prof Yong Man Ro
Mass detection via deep learning
Deep learning do this tasks
Prof Yong Man Ro
Breast cancer screening :
2D DM and 3D DBT
� 3D DBT clearly shows breast cancers
2D DM 3D DBT
Mass
2D DM
Microcalcification
3D DBT
Prof Yong Man Ro
Latent feature representation
with depth directional long-term recurrent learning
Related publication:
Latent feature representation with depth directional long-term recurrent
learning for breast masses in digital breast tomosynthesis, Yong Man Ro
etal. Medical Physics, revising, 2016
Prof Yong Man Ro
Overview of the proposed method
� Variation of depth directional texture patterns
� Slices of masses show similar texture patterns, while FPs show different
texture patterns among slices (as a medical doctor’s diagnosis)
�Because FPs are occurred when tissues in different depth are overlapped
Example of breast cancer
screening using DBT [2]Mass
Prof Yong Man Ro
� Encoding scheme for masses via the proposed latent feature
representation
Slices of
the VOI
Encoding spatial structures
3D DBT volume
……
Top
Bottom
Central
…
CNN
(2D spatial)
CNN
(2D spatial)
LSTM
(Depth direction)
LSTM
(Depth direction)
LSTM
(Depth direction)
…
Shared
weights
Top-to-central slices Bottom-to-central slices
Latent feature
representation
VOI
Encoding texture variations in
top-to-central slice direction and
bottom-to-central slice direction
Encoding central slice directional
symmetric patterns
Depth
direction
Latent feature representation with depth directional long-term recurrent learning for breast masses in
digital breast tomosynthesis, Yong Man Ro etal. Medical Physics, revising, 2016
Prof Yong Man Ro
� Central slice directional learning using depth directional long-term
recurrent learning
� Modelling the symmetric pattern of slice feature representations of masses
with respect to the central slice
9/29
Structure of LSTM layers
Latent feature representation with depth directional long-term recurrent learning for breast masses in
digital breast tomosynthesis, Yong Man Ro etal. Medical Physics, revising, 2016
Prof Yong Man Ro
Visualization of activations on the learned LSTM layer10/29
Masses
FPs
Evolution of states of three gates and a memory cell and an output (i.e.,
latent feature representation) for 4 LSTM cells at last LSTM layer
Slice index
Prof Yong Man Ro
Experiment 1
� Classification performance comparisons with existing methods
11/29
Feature group AUC
Hand-crafted features [1] 0.847
Slice feature representation 0.871
Proposed latent feature representation 0.919
[1] Kim D H, Kim S T and Ro Y M 2015 Improving mass detection using combined feature representations from projection views and reconstructed volume of DBT and boosting based
classification with feature selection Phys. Med. Biol. 60 8809
ROC curve
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
False positive fraction (FPF)
Tru
e p
osi
tiv
e fr
acti
on
(T
PF
)
Hand-crafted features
Slice feature representation
Proposed latent feature representation
Prof Yong Man Ro
Bilateral analysis:
Latent feature representation with 3D multi-view CNN
Related publication:
Latent feature representation with 3-D Multi-view Convolutional Neural Network for
Bilateral Analysis in Digital Breast Tomosynthesis ,”, Yong Man Ro IEEE ICASSP,
2016
Prof Yong Man Ro
Clinical practice: Bilateral analysis
� Bilateral analysis for breast cancer screening and diagnosis
� Mass is an asymmetric density which is visible on two projections (CC and MLO)
� Asymmetry between the left and right breast of a given subject is an important sign
used by radiologists to diagnose breast cancer
Example of reconstructed slices including
a mass (white circled)
RMLO LMLO
Asymmetric density
Prof Yong Man Ro
Conventional: hand-crafted features in DBT
� Measuring the bilateral dissimilarity between two VOIs in left
breast and right breast [1]
� Texture dissimilarity (i.e., Sum of squared differences (SSD) measure)
� Dissimilarity of intensity distribution (i.e., Histogram correlation)
� Dissimilarity of mass characteristics between VOIs (i.e., Absolute difference of single
features)
� Limitation
� Bilateral characteristics are abstract
� Due to the subtle characteristics
of masses in bilateral analysis,
it is hard to design effective
hand-crafted features
T
(transform)VOI in source
image (i.e. left)
r Registered
VOI
LMLO RMLO�� ��
T(r)
Estimating dissimilarity
between two VOIs
Illustration of hand-crafted
bilateral feature extraction
[1] “Feature extraction from bilateral dissimilarity in digital breast tomosynthesis reconstructed volume,” IEEE international conference on image processing, 2015.
Prof Yong Man Ro
Bilateral analysis with multi-view 3D CNN
� 3D CNN learns tissue structure in volume
� Multi-view fusion network learns different representation of VOIs as input
and learns features individually
“Latent feature representation with 3-d multi-view deep convolutional neural network for bilateral analysis in digital breast tomosynthesis,” IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP), 2016.
Prof Yong Man Ro
Experimental results
Comparisons of ROC curves of FP reduction using hand-crafted
features and proposed latent bilateral feature representation
“Feature extraction from bilateral dissimilarity in digital breast tomosynthesis reconstructed volume,” IEEE international conference on image processing, 2015.
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
False positive fraction (FPF)
Tru
e p
osi
tiv
e fr
acti
on
(T
PF
)
Hand-crafted features [1] (AUC=0.826 ± 0.013)
Latent bilateral features (AUC=0.847 ± 0.012) t-SNE feature visualization
Original input
data
FPs
Masses
Recommended