View
13
Download
0
Category
Preview:
Citation preview
Identifying Anomalous Objects in SAS Imagery using Uncertainty
Calum BlairJohn ThompsonUniversity of Edinburgh
8th June 2015c.blair@ed.ac.uk
Neil Robertson
Heriot-Watt University
Outline
• Motivation
– Reliable detectors and anomalies
– Challenging environments: SAS applications
• Method
– Classification approaches
– Uncertainty & Reliability
• Results
• Conclusion
15/07/2015 2
Related work
• Reliable object classification in complex environments:– Probabilistic classification with
`well-calibrated’ uncertainty (indicate lack of confidence)
– Classifier algorithms:accuracy != reliability
– Gaussian Process Classifiers (GPCs) suitable in previous work
15/07/2015 3
[1]Blair et al, 2014
Motivation
• Mine countermeasures in SAS imagery: – Goal: detect multiple object classes
• Challenging environment:– Data collection harder than video!
– Application to new environments?
– Cost of false positive & false negatives unequal
• Same goal: reliable detectors, capable of identifying anomalies?
15/07/2015 4
15/07/2015 5
Goals
15/07/2015 6
• Train on cone and wedge shapes representing different mine types
• Accurate & reliable detection of two classes
• Detect unknown/ anomalous objects at test time? Use third object shape: cylinder
Method
• Standard sliding window-based classifiers
• Normalise each window & pass to classifier algorithm
• During training, skip windows with cylinders
15/07/2015 7
Window Normalisation
Probabilistic Classification
Uncertainty Detection
Classification: SVM
15/07/2015 8
Obtain SVM score:
Using linear / RBF kernel:
Learn a,b to obtain probabilistic classification; squash score into [0,1]:
Gaussian Process Classifier
15/07/2015 9
Probabilistic classification of
Training: find best-fitting distribution of latent variable f*, with Gaussian distribution:
covariance kij between train and test is:
squash with sigmoid:
probabilistic measure:
Entropy/Uncertainty
15/07/2015 10
Given multiple classifications:
No extra classification.Maximised when multiple detectors uncertain.Assumes reliable detectors!Threshold at H>0.9 & discard overlaps with existing detections.
Results
• 2 datasets (Colossus2, Catharsis2)
• Image classification & uncertainty
15/07/2015 11
2-class classification
Object TP FN FP Precision % Recall %
SVM-RBF cone 35 19 91 28 65
SVM-RBF wedge 44 16 4655 0.93 73
SVM-Linear cone 39 15 18879 0.21 72
SVM-Linear wedge 0 60 203 0 0
GPC-Linear cone 40 14 15569 0.26 74
GPC-Linear wedge 54 6 15330 0.015 90
15/07/2015 12
Colossus 2 test set:
2-class classification
Object TP FN FP Precision % Recall %
SVM-RBF cone 7 1 149 4.9 100
SVM-RBF wedge 1 0 836 0.11 100
SVM-Linear cone 8 0 3916 0.2 100
SVM-Linear wedge 0 1 5 0 0
GPC-Linear cone 8 0 3267 0.24 100
GPC-Linear wedge 1 0 15330 0.007 100
15/07/2015 13
Catharsis 2: More distinct from training data, fewer objects
2-class classification:
15/07/2015 14
Cones: Colossus Catharsis
15/07/2015 15
Wedges: Colossus Catharsis
2-class failures: Ripples
15/07/2015 16
Ripple size & orientation ≅ object highlight & shadowFuture work: incorporate region context classifiers for rippled seabeds.
2-class classification: reliability
15/07/2015 17
Cones: Colossus Catharsis
‘Well-calibrated’ line: given predicted values, what fraction at a given value evaluate as positive?
Results: Uncertainty
15/07/2015 18
Colossus2 “TP” “FN” “FP” Precision % Recall %
SVM-RBF 12 15 17 41.4 44.4
SVM-Linear 24 3 2804 0.84 88.9
Catharsis2
SVM-RBF 1 0 57 1.72 100
SVM-Linear 1 0 2640 0.03 100
Limited data makes evaluation difficult; suitable for presentation to operator?
Results: Failures
15/07/2015 19
False positive from uncertainty detector
Wedge: missed by own detector but found by uncertainty detector
Conclusions
• SVM and GPC classifiers for objects in SAS
• GPCs perform less well when unsophisticated features used
• Uncertainty info allows detection of ‘unknown’ classes; SVM-RBF performs best here.
15/07/2015 20
QUESTIONS?
15/07/2015 21
Recommended