SKIN LESION DETECTION FROM DERMOSCOPIC IMAGES USING
CONVOLUTIONAL NEURAL NETWORKS
Adrià Romero López Oge Marques Xavier Giró-i.Nieto
AUTHOR ADVISORS
Acknowledgments
2
MIDDLE Research Group
Víctor Campos Albert Gil
Jack Burdick Janet Weinthal Adam Lovett
Oge Marques Borko Furht Xavier Giró-i.Nieto Albert Jiménez
‘’Outline
3
1. Motivation2. State of the art3. Methodology4. Experimental Results5. Conclusions
1.Motivation
4
Background of the problem
▣ Skin cancer: most predominant type of cancer ▣ The frequency of melanoma doubles every 20 years ▣ Each year (in USA):
□ 76,380 new cases of melanoma □ 6,750 deaths
▣ Melanoma is a deadly form of skin cancer, but survival rates are high if detected and diagnosed early
▣ Melanoma detection: rely on hand-crafted features □ ABCDE rule (Asymmetry, Border, Color, Dermoscopic
structure, and Evolving)□ CASH rule (Color, Architecture, Symmetry, and
Homogeneity)
5
Background of the problem
▣ Discriminating between benign and malignant skin lesions is challenging
▣ Without computer-based assistance: 60~80% detection accuracy
6
Scope and goals
▣ Scope:□ Assist physicians in classifying skin lesions (especially in
melanoma detection: 2-class classifier problem) ▣ Goal:
□ Use state-of-the-art techniques, called Deep Learning, to design an intelligent medical imaging-based skin lesion diagnosis system
□ Achieve (or improve upon) state-of-the-art results for:■ skin lesion segmentation, and■ skin lesion classification
□ Evaluate the impact of skin lesion segmentation on the accuracy of the classifier
7
Hypothesis
Previous segmentation of an image containing a skin lesion (i.e., isolating the lesion from the background) improves the accuracy and sensitivity of a Deep Learning classification model approach.
Challenges
▣ Dermoscopic images may:■ Contain artifacts, such as: moles, freckles, hair,
patches, shading and noise.■ Present low contrast images between lesion and
background■ Contain multiple skin lesions
9
Related work
•Typical block diagram (Non-Deep Learning approach from [Glaister2013])
10
2.State of the art
11
State-of-the-art hierarchy
12
CNNs
Deep learning motivation
▣ Image representations to:□ Image classification□ Object detection and recognition□ Semantic Segmentation
13
Self-driving cars[Goodfellow et al. 2014]
[Ciresan et al. 2013]
[Turaga et al 2010]
Slide credit: Bay Area Deep Learning School Presentation by A. Karpathy
Supervised learning
14[Car] [Dog]
Parameters
Slide credit: “Artificial Intelligence, revealed” by Facebook Research
Why deep learning now?
15
Large datasets GPUs (Graphics Processing Unit)
* Not applicable to medical imaging
[Deng et al. Russakovsky et al.]
[NVIDIA et al.]
Framework
Convolutional Neural Networks
16
Some input vector (our images).
Also known as ConvNets or CNNs
Our class label
▣ Convolutional Layers▣ Activation Layers▣ Pooling Layers
Convolution layer
17
32
32
3
5x5x3 filter
32x32x3 image
Convolve the filter with the imagei.e. “slide over the image spatially, computing dot products”
Filters always extend the full depth of the input volume
Slide credit: Bay Area Deep Learning School Presentation by A. Karpathy
Convolution layer
18
32
32
3
32x32x3 image
1 number: the result of taking a dot product between the filter and a small 5x5x3 chunk of the image(i.e. 5*5*3 = 75-dimensional dot product + bias)
Slide credit: Bay Area Deep Learning School Presentation by A. Karpathy
Linear function
5x5x3 filter → weights (Learnt by Backpropagation algorithms)
Activation layer
19
32
32
3
32x32x3 image5x5x3 filter
Convolve (slide) over all spatial locations
ReLU (Rectified Linear Units)
1
28
28
Slide credit: Bay Area Deep Learning School Presentation by A. Karpathy
activation map
Pooling layer
▣ Undersampling task□ Makes the representation smaller and more
manageable□ Operates over each activation map independently
20Slide credit: Bay Area Deep Learning School Presentation by A. Karpathy
Fully-Connected (FC) layer
21
Main scheme
22
Input image[Yann LeCun et al.]
Main scheme
23
1. Convolutional Layers 2. Activation Layer 3. Pooling Layers
[Yann LeCun et al.]
Main scheme
24
[Yann LeCun et al.]
Fully-Connected Layer
Main scheme
25
[Yann LeCun et al.]
Output label
ConvNets for classification
▣ Classification → Scoring:□ The CNN computes a class score {float} to each
image □ This score will be related to a class label {integer}
26
[224x224x3]
f Class scores, indicating class labels
training
Slide credit: Bay Area Deep Learning School Presentation by A. Karpathy
ConvNets for segmentation
▣ Segmentation → Localization:□ The CNN assigns a class label to each pixel (classify
all pixels)■ {0,1} → {absence of object, presence of object}
□
27Slide credit: CS231n
ConvNets for segmentation
28Slide credit: CS231n
▣ Upsampling□ From labels {1x1} to Segmented Image {224x224} px
Transfer learning
29
1. Train on Imagenet
3. Medium dataset:finetuning
more data = retrain more of the network (or all of it)
2. Small dataset:feature extractor
Freeze these
Train this
Freeze these
Train this
Slide credit: Bay Area Deep Learning School Presentation by A. Karpathy
Medical Imaging case
3.Methodology
30
Framework
▣ Python environment:□ Keras - Deep Learning Library for Theano or TensorFlow□ OpenCV / PIL (Python Imaging Library)□ SciPy (Library for Mathematics, Science and Engineering) □ Scikit-learn (Machine Learning Library)□ CUDA library for the GPUs
31
+ =
ISIC Archive dataset
▣ ISBI 2016 Challenge dataset□ Skin Lesion Analysis towards melanoma detection□ 1279 RGB images□ Labeled as either benign or malignant□ Includes the binary mask for each image
32
Class
Benign Malignant Total Images
Training subset 727 173 900
Validation subset 304 75 379
0 → outside lesion area255 → inside lesion area
Binary mask
Method scheme
33
Data augmentation
▣ Enlarge our few training examples:□ Re-scaling□ 40 degrees rotations □ Horizontal shifts□ Zooming□ Horizontal flips
34
Original image Random transformations
Preprocessing
▣ Mean subtraction: X -= np.mean(X, axis = 0)▣ Image Normalization: X /= np.std(X, axis = 0)
▣ Image cropping & resizing□ Segmentation model: 64 x 80 px□ Classification model: 224 x 224 px
35
Segmentation model: U-Net architecture
36
▣ Convolutional Networks for Biomedical Image Segmentation by Olaf Ronneberger et al.
Binary Mask
Segmentation model: training parameters
37
▣ U-Net trained from scratch (small image size)▣ Weights randomly initialized▣ Loss function:
□ Dice coefficient▣ Adam optimizer (Stochastic gradient-based
optimization):□ Learning rate: 10e-5
▣ Batch size: 32▣ Training epochs: 500 epochs▣ 13 sec / epoch on NVidia GeForce GTX TITAN X GPU
Objective
To verify our hypothesis:1. Unaltered lesion classification2. Perfectly segmented lesion classification3. Automatically segmented lesion classification
38
Logical AND operation
Logical AND operation
Original Binary Mask (perfect)
Binary Mask obtained with the U-Net
Previous segmentation of the skin lesion improves the accuracy and sensitivity of a Deep Learning classification model.
(1)
(2)
(3)
Method Scheme (reminder)
39
Classification Model: VGG-16 Architecture
40
▣ Five Convolutional Blocks (2D conv.)
▣ 3 x 3 receptive field▣ ReLU as Activation
Functions▣ Max-Pooling▣ Classifier block:
□ 3 FC Layers at the top of the network
Fine-tuning the VGG-16 Architecture
41
▣ Weights initialized with the VGG-16 pretrained on Imagenet dataset
▣ Freeze bottom of the network
▣ Just train the top of the VGG-16 Train this
41
Freeze these
Classification Model: Loss function
▣ Problem: ISIC dataset classes not balanced□ Validation subset:
■ 304 benign images■ 75 malignant images
▣ Weighted Loss function:
where ρ is defined as 1−frequency appearance (minor class)
42
Classification Model: Training parameters
43
▣ VGG-16 fine-tuned▣ Weights initialized with the VGG-16 pretrained on
Imagenet dataset▣ Loss function:
□ Weighted Loss function▣ SGD optimizer (Stochastic gradient-based
optimization):□ Learning rate: 10e-5
▣ Batch size: 32▣ Training epochs: 50 epochs▣ 35 sec / epoch on NVidia GeForce GTX TITAN X GPU
Overfitting
▣ When a model fits the training data too well□ Noise in the training data is learned by the model
▣ How to prevent it?□ Dropout□ Choosing a reduced network (VGG-16 with 138M
param. rather than VGG-19 with 144M param.)
44
4.Experimental
Results
45
Segmentation Evaluation
47
Participant Accuracy Dice Coef. Jaccard Index
Sensitivity Specificity
MIDDLE group
0.9176 0.8689 0.9176 0.9301 0.9544
▣ Comparing pixel by pixel of each masks:
Ground truth Mask obtained
JACCARD INDEX:
Segmentation Examples
50
▣ Satisfactory segmentation examples
▣ Poor segmentation examples
Classification Evaluation
51
Model Accuracy Loss Sensitivity Precision
Unaltered lesion clas.
0.8469 0.4723 0.8243 0.9523
Perfectly segmented lesion clas.
0.8390 0.4958 0.8648 0.9621
Automatically segmented lesion clas.
0.8174 0.5144 0.8918 0.9681
Classification Evaluation
52
Model Accuracy Loss Sensitivity Precision
Unaltered lesion clas.
0.8469 0.4723 0.8243 0.9523
Perfectly segmented lesion clas.
0.8390 0.4958 0.8648 0.9621
Automatically segmented lesion clas.
0.8174 0.5144 0.8918 0.9681
▣ With segmentation □ Accuracy decreases□ Loss increases
Classification Evaluation
53
Model Accuracy Loss Sensitivity Precision
Unaltered lesion clas.
0.8469 0.4723 0.8243 0.9523
Perfectly segmented lesion clas.
0.8390 0.4958 0.8648 0.9621
Automatically segmented lesion clas.
0.8174 0.5144 0.8918 0.9681
▣ But...with segmentation □ Sensitivity increases !□ Precision increases !
Classification Evaluation
54
Model Accuracy Loss Sensitivity Precision
Unaltered lesion clas.
0.8469 0.4723 0.8243 0.9523
Perfectly segmented lesion clas.
0.8390 0.4958 0.8648 0.9621
Automatically segmented lesion clas.
0.8174 0.5144 0.8918 0.9681
▣ But...with segmentation: □ Sensitivity increases !□ Precision increases !
SENSITIVITY = TP / (TP + FN)
PRECISION = TP / (TP + FP)
Sensitivity in Medical Settings
▣ Sensitivity is often considered the most important metric in the medical setting
▣ For early diagnosis□ By missing a False Negatives (true melanoma case)
the model would fail in the early diagnosis□ It is better to raise a False Positive than to create a
False Negative
55
Classification evaluation
56
Model Accuracy Loss Sensitivity Precision
Unaltered lesion clas.
0.8469 0.4723 0.8243 0.9523
Perfectly segmented lesion clas.
0.8390 0.4958 0.8648 0.9621
Automatically segmented lesion clas.
0.8174 0.5144 0.8918 0.9681
▣ And the Automatically Segmented Model is even BETTER than the Perfectly Segmented□ Physicians can avoid Manual Segmentation tasks
Confusion Matrices
57
False Negatives descending
Unaltered Classifier Perfectly Classifier Segmented Classifier
Classification Examples
58
5.Conclusions
59
Conclusions
▣ DL solution for assisting dermatologists with the diagnosis of skin lesions□ Specifically, for early melanoma detection
▣ Does a previous semantic segmentation improve the performance of a fine-tuned CNN for a 2-class classifier?□ Hypothesis verified
▣ Perfect Segmentation was not needed to obtain the best classification result of the model□ DL Segmentation approach obtained the best
sensitivity classification result
60
Conclusions
▣ BioMed 2017 Conference → Paper Accepted□ Title: “Skin Lesion Classification from Dermoscopic
Images Using Deep Learning Techniques”▣ SIIM 2017 Meeting → Paper Accepted
□ Title: “The Impact of Segmentation on the Accuracy and Sensitivity of a Melanoma Classifier Based on Skin Lesion Images”
▣ MICCAI 2017 Conference → Intention of Paper ▣ MIUA 2017 Conference → Intention of Paper▣ ISBI 2017 Challenge → Intention of Participation
□ Skin Lesion Analysis Towards Melanoma Detection
61
Thanks!Any questions?
62
You can find me at: