Identifying Anomalous Objects in SAS Imagery using Uncertainty

Calum BlairJohn ThompsonUniversity of Edinburgh

8th June 2015c.blair@ed.ac.uk

Neil Robertson

Heriot-Watt University

Outline

• Motivation

– Reliable detectors and anomalies

– Challenging environments: SAS applications

• Method

– Classification approaches

– Uncertainty & Reliability

• Results

• Conclusion

15/07/2015 2

Related work

• Reliable object classification in complex environments:– Probabilistic classification with

`well-calibrated’ uncertainty (indicate lack of confidence)

– Classifier algorithms:accuracy != reliability

– Gaussian Process Classifiers (GPCs) suitable in previous work

15/07/2015 3

[1]Blair et al, 2014

Motivation

• Mine countermeasures in SAS imagery: – Goal: detect multiple object classes

• Challenging environment:– Data collection harder than video!

– Application to new environments?

– Cost of false positive & false negatives unequal

• Same goal: reliable detectors, capable of identifying anomalies?

15/07/2015 4

15/07/2015 5

Goals

15/07/2015 6

• Train on cone and wedge shapes representing different mine types

• Accurate & reliable detection of two classes

• Detect unknown/ anomalous objects at test time? Use third object shape: cylinder

Method

• Standard sliding window-based classifiers

• Normalise each window & pass to classifier algorithm

• During training, skip windows with cylinders

15/07/2015 7

Window Normalisation

Probabilistic Classification

Uncertainty Detection

Classification: SVM

15/07/2015 8

Obtain SVM score:

Using linear / RBF kernel:

Learn a,b to obtain probabilistic classification; squash score into [0,1]:

Gaussian Process Classifier

15/07/2015 9

Probabilistic classification of

Training: find best-fitting distribution of latent variable f*, with Gaussian distribution:

covariance kij between train and test is:

squash with sigmoid:

probabilistic measure:

Entropy/Uncertainty

15/07/2015 10

Given multiple classifications:

No extra classification.Maximised when multiple detectors uncertain.Assumes reliable detectors!Threshold at H>0.9 & discard overlaps with existing detections.

Results

• 2 datasets (Colossus2, Catharsis2)

• Image classification & uncertainty

15/07/2015 11

2-class classification

Object TP FN FP Precision % Recall %

SVM-RBF cone 35 19 91 28 65

SVM-RBF wedge 44 16 4655 0.93 73

SVM-Linear cone 39 15 18879 0.21 72

SVM-Linear wedge 0 60 203 0 0

GPC-Linear cone 40 14 15569 0.26 74

GPC-Linear wedge 54 6 15330 0.015 90

15/07/2015 12

Colossus 2 test set:

2-class classification

Object TP FN FP Precision % Recall %

SVM-RBF cone 7 1 149 4.9 100

SVM-RBF wedge 1 0 836 0.11 100

SVM-Linear cone 8 0 3916 0.2 100

SVM-Linear wedge 0 1 5 0 0

GPC-Linear cone 8 0 3267 0.24 100

GPC-Linear wedge 1 0 15330 0.007 100

15/07/2015 13

Catharsis 2: More distinct from training data, fewer objects

2-class classification:

15/07/2015 14

Cones: Colossus Catharsis

15/07/2015 15

Wedges: Colossus Catharsis

2-class failures: Ripples

15/07/2015 16

Ripple size & orientation ≅ object highlight & shadowFuture work: incorporate region context classifiers for rippled seabeds.

2-class classification: reliability

15/07/2015 17

Cones: Colossus Catharsis

‘Well-calibrated’ line: given predicted values, what fraction at a given value evaluate as positive?

Results: Uncertainty

15/07/2015 18

Colossus2 “TP” “FN” “FP” Precision % Recall %

SVM-RBF 12 15 17 41.4 44.4

SVM-Linear 24 3 2804 0.84 88.9

Catharsis2

SVM-RBF 1 0 57 1.72 100

SVM-Linear 1 0 2640 0.03 100

Limited data makes evaluation difficult; suitable for presentation to operator?

Results: Failures

15/07/2015 19

False positive from uncertainty detector

Wedge: missed by own detector but found by uncertainty detector

Conclusions

• SVM and GPC classifiers for objects in SAS

• GPCs perform less well when unsophisticated features used

• Uncertainty info allows detection of ‘unknown’ classes; SVM-RBF performs best here.

15/07/2015 20

QUESTIONS?

15/07/2015 21