Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Milan Šulc Fine-grained Recognition of Plant Species from Images
Fine-grained Recognitionof Plant Species from Images
Milan Šulc
PhD candidate Advisor Jiřiacute Matas
Visual Recognition GroupCenter for Machine PerceptionCzech Technical University in Prague
Milan Šulc Fine-grained Recognition of Plant Species from Images
Many formulations exist depending on
bull Type of observationbull from an image (or images) of a known plant organbull from images of multiple organsbull from images of a larger part of the plant
bull Acquisition conditions bull controlled background viewpoint occlusion vs unconstrained
bull Granularity of the decision Typically fine-grained classification required the classes have
bull Small inter-class differencesbull High intra-class variability
Plant Species Recognition
236
Milan Šulc Fine-grained Recognition of Plant Species from Images
The observation depends on many factors
bull Genotypebull Agebull Seasonbull Local environment
Climate Altitude Illumination
bull Clutter (other plants in the foreground or background)bull Acquisition conditions
bull Device
Plant Species Recognition
336
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Constrained recognition of plants from photos of tree bark photos or scans of leaves
ldquoOld-school handcrafted features achieve excellent recognition ratesrdquo
bull Plant recognition ldquoin the wildrdquo for up to 10K plant species ldquoAlmost unconstrained scenario Outperforms 59 human experts best results in the LifeCLEF 2018 plant identification challengerdquo
bull Re-weighting predictions for different training- and test-time class prior probabilities
bull Test-time class prior estimation
bull Future work Knowledge distillation from Ensembles
Presentation Outline
436
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Bark recognition [1] using texture descriptors based on Local Binary Patterns
bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]
[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011
Our Early Work on Plant Recognition
536
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Fast description Histograms of Completed LBP (Local Binary Patterns)
2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)
3) Improved scale space for multi-scale description and scale invariance
4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]
Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)
Fast Features Invariant to Rotationand Scale of Texture
636
184 129 140
159 156 150
168 80 130
0
rarr00011100
0
0
0
0
1
1
1
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Leaf recognition texture of the leaf interior and border [1][2]
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Our Early Work on Plant Recognition
736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our Early Work on Plant Recognition
836
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Leaf Recognition in the Era of CNNs
936
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Many formulations exist depending on
bull Type of observationbull from an image (or images) of a known plant organbull from images of multiple organsbull from images of a larger part of the plant
bull Acquisition conditions bull controlled background viewpoint occlusion vs unconstrained
bull Granularity of the decision Typically fine-grained classification required the classes have
bull Small inter-class differencesbull High intra-class variability
Plant Species Recognition
236
Milan Šulc Fine-grained Recognition of Plant Species from Images
The observation depends on many factors
bull Genotypebull Agebull Seasonbull Local environment
Climate Altitude Illumination
bull Clutter (other plants in the foreground or background)bull Acquisition conditions
bull Device
Plant Species Recognition
336
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Constrained recognition of plants from photos of tree bark photos or scans of leaves
ldquoOld-school handcrafted features achieve excellent recognition ratesrdquo
bull Plant recognition ldquoin the wildrdquo for up to 10K plant species ldquoAlmost unconstrained scenario Outperforms 59 human experts best results in the LifeCLEF 2018 plant identification challengerdquo
bull Re-weighting predictions for different training- and test-time class prior probabilities
bull Test-time class prior estimation
bull Future work Knowledge distillation from Ensembles
Presentation Outline
436
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Bark recognition [1] using texture descriptors based on Local Binary Patterns
bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]
[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011
Our Early Work on Plant Recognition
536
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Fast description Histograms of Completed LBP (Local Binary Patterns)
2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)
3) Improved scale space for multi-scale description and scale invariance
4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]
Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)
Fast Features Invariant to Rotationand Scale of Texture
636
184 129 140
159 156 150
168 80 130
0
rarr00011100
0
0
0
0
1
1
1
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Leaf recognition texture of the leaf interior and border [1][2]
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Our Early Work on Plant Recognition
736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our Early Work on Plant Recognition
836
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Leaf Recognition in the Era of CNNs
936
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
The observation depends on many factors
bull Genotypebull Agebull Seasonbull Local environment
Climate Altitude Illumination
bull Clutter (other plants in the foreground or background)bull Acquisition conditions
bull Device
Plant Species Recognition
336
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Constrained recognition of plants from photos of tree bark photos or scans of leaves
ldquoOld-school handcrafted features achieve excellent recognition ratesrdquo
bull Plant recognition ldquoin the wildrdquo for up to 10K plant species ldquoAlmost unconstrained scenario Outperforms 59 human experts best results in the LifeCLEF 2018 plant identification challengerdquo
bull Re-weighting predictions for different training- and test-time class prior probabilities
bull Test-time class prior estimation
bull Future work Knowledge distillation from Ensembles
Presentation Outline
436
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Bark recognition [1] using texture descriptors based on Local Binary Patterns
bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]
[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011
Our Early Work on Plant Recognition
536
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Fast description Histograms of Completed LBP (Local Binary Patterns)
2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)
3) Improved scale space for multi-scale description and scale invariance
4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]
Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)
Fast Features Invariant to Rotationand Scale of Texture
636
184 129 140
159 156 150
168 80 130
0
rarr00011100
0
0
0
0
1
1
1
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Leaf recognition texture of the leaf interior and border [1][2]
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Our Early Work on Plant Recognition
736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our Early Work on Plant Recognition
836
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Leaf Recognition in the Era of CNNs
936
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Constrained recognition of plants from photos of tree bark photos or scans of leaves
ldquoOld-school handcrafted features achieve excellent recognition ratesrdquo
bull Plant recognition ldquoin the wildrdquo for up to 10K plant species ldquoAlmost unconstrained scenario Outperforms 59 human experts best results in the LifeCLEF 2018 plant identification challengerdquo
bull Re-weighting predictions for different training- and test-time class prior probabilities
bull Test-time class prior estimation
bull Future work Knowledge distillation from Ensembles
Presentation Outline
436
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Bark recognition [1] using texture descriptors based on Local Binary Patterns
bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]
[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011
Our Early Work on Plant Recognition
536
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Fast description Histograms of Completed LBP (Local Binary Patterns)
2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)
3) Improved scale space for multi-scale description and scale invariance
4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]
Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)
Fast Features Invariant to Rotationand Scale of Texture
636
184 129 140
159 156 150
168 80 130
0
rarr00011100
0
0
0
0
1
1
1
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Leaf recognition texture of the leaf interior and border [1][2]
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Our Early Work on Plant Recognition
736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our Early Work on Plant Recognition
836
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Leaf Recognition in the Era of CNNs
936
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Bark recognition [1] using texture descriptors based on Local Binary Patterns
bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]
[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011
Our Early Work on Plant Recognition
536
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Fast description Histograms of Completed LBP (Local Binary Patterns)
2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)
3) Improved scale space for multi-scale description and scale invariance
4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]
Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)
Fast Features Invariant to Rotationand Scale of Texture
636
184 129 140
159 156 150
168 80 130
0
rarr00011100
0
0
0
0
1
1
1
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Leaf recognition texture of the leaf interior and border [1][2]
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Our Early Work on Plant Recognition
736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our Early Work on Plant Recognition
836
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Leaf Recognition in the Era of CNNs
936
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Fast description Histograms of Completed LBP (Local Binary Patterns)
2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)
3) Improved scale space for multi-scale description and scale invariance
4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]
Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)
Fast Features Invariant to Rotationand Scale of Texture
636
184 129 140
159 156 150
168 80 130
0
rarr00011100
0
0
0
0
1
1
1
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Leaf recognition texture of the leaf interior and border [1][2]
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Our Early Work on Plant Recognition
736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our Early Work on Plant Recognition
836
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Leaf Recognition in the Era of CNNs
936
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Leaf recognition texture of the leaf interior and border [1][2]
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Our Early Work on Plant Recognition
736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our Early Work on Plant Recognition
836
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Leaf Recognition in the Era of CNNs
936
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our Early Work on Plant Recognition
836
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Leaf Recognition in the Era of CNNs
936
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Leaf Recognition in the Era of CNNs
936
[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo
Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available
bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates
bull Contained ldquodistractorsrdquo in the test set
[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016
LifeCLEF 2016 Plant Identification Task
1036
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)
bull Maxout [3]
bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification
VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs
[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327
LifeCLEF 2016 Our Approach [1]
1136
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonbull Combining test images with the same observation ID was allowed
in the competition and had a significant effect on the final scores
[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016
LifeCLEF 2016 Plant Identification Task
Team Single-image recognition [ mAP]
Sum per observation[ mAP]
Bluefield [1] 611 742
SabanciUGebzeTU [2] 738 793
CMP (ours) 710 788
1236
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images
obtained from Encyclopedia Of Life (EoL)
bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search
bull test set of 25K images no distractors
bull 80 participants registered only 8 participants submitted results
LifeCLEF 2017 Plant Identification Task
1336
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Random examples from the ldquotrustedrdquo training set
bull Random examples from the ldquonoisyrdquo training set
LifeCLEF 2017 Plant Identification Task
1436
[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Our approach [1]
bull Inception-ResNet-v2 [2]
bull Maxout [3]
bull Bootstrapping [4] consistency objectives for training on noisy labels
[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich
LifeCLEF 2017 Plant Identification Task
1536
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data
additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates
[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017
LifeCLEF 2017 Plant Identification Task
1636
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts
1736
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experts vs Machines (after LifeCLEFrsquo17)
Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts
1836
[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull Scenario similar to the 2017 challenge
bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training
2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small
subset of the 10k species
bull Test set of 6892 images
bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)
LifeCLEF 2018 Plant Identification Task
1936
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Preliminary experiments (using the 2017 test set for validation)
1 Running averages (exponential decay) of trainable variables
684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)
Interpretation Noisy samples (from web majority of data) in mini-batches
may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo
Learning rate was probably too high
2 Assuming changes in class prior distribution is very important
LifeCLEF 2018 Plant Identification Task
2036
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
0 10 20Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Label Distributions
2136
0 2000 4000 6000 8000 10000Class (sorted by Ntrain)
0
500
1000
1500
2000
2500
im
ag
es
Test set
Training set
Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance
Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities
where
Then
CNN Outputs as Posterior Estimates
2236
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples
CNN Outputs as Posterior Estimates
2336
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Assuming that the probability density function remains unchanged
The mutual relation of the posteriors is
Adjusting Estimates to New Priors
2436
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
When Test Set Priors Are Unknown
2536
How to estimate the test-set priors
Saerens et al proposed a simple EM procedure to maximize the likelihood L(x
0x1x2)
This procedure is equivalent [2] to fixed-point-iteration minimization of
the KL divergence between and
[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Test Set Prior Estimation in LifeCLEF
2636
Preliminary experiments (using the 2017 test set for validation)
When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863
On-line [1] after each new test image
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Plant Ident Task Results
2736
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
LifeCLEF 2018 Experts vs Machines
2836
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
2936
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3036
0 200 400 600 800 1000 1200Class (sorted)
00
01
02
03
04
05
s
am
ple
s
Training set
Validation set
0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)
000
005
010
015
020
s
am
ple
s
Training set
Validation set
iNaturalist 2018 FGVCx Fungi 2018
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
35
40
45
50
55
60
65
70
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
0 50000 100000 150000 200000 250000 300000 350000 400000Training steps
350
375
400
425
450
475
500
525
Acc
ura
cy [
]
CNN output accuracy
Known (flat) test distr
[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
When New Priors Are Known
3136
Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows
ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo
[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3236
So far we approached the estimation of new class priorsby Maximum Likelihood (ML)
It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P
k
α ge 1 - favours dense distributions rarr avoids Pkasymp0
- makes Dirichlet log-concave rarr suitable for SGD
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3336
CIFAR-100 experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
Hyper-prior on Class Priors
3436
Fine-grained experiments
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
1) Knowledge distillation from Ensembles
Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble
have the same architecture are trained on the same training set
How to distill knowledge from ensembles to improve single-model accuracy
2) Test set prior estimation
In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)
3) Learning embeddings Metric learning
As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical
3536
Future Work
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion
Milan Šulc Fine-grained Recognition of Plant Species from Images
bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition
bull Constrained tasks can be solved by faster features with 99+ accuracy
bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem
bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear
bull Human expert performance reached
bull Q amp A sulcmilacmpfelkcvutcz
3636
Discussion