IMAGE ARCHIVE AND LEAF CLASSIFIER SPECIFIC ENABLERS

Stuart E. Middleton, Banafshe Arbab-Zavar, Stefano Modafferi, Ken Meacham and Zoheir SabeurUniversity of Southampton IT Innovation CentreENVIROFI specific enabler17th January 2013

“ENVIROfying” the Future Internet

• WP1 pilot use case• Image archive

• Architecture• User interface

• Leaf classifier• Architecture• Algorithms• User interface

Overview

Image archive and leaf classifier specific enablers

2

• WP1 pilot: Citizens in Tuscany• Data sources

• Proof of concept• CROWD SOURCING FROM SIR HAROLD HILLIER GARDENS, UK• HTTP://WWW3.HANTS.GOV.UK/HILLIERGARDENS

• User trial• CROWD SOURCING VIA WP1 PILOT IN THE TUSCANY REGION

• Image archive to record crowd-sourced leaf images• Web portal & backend service (Italian & English)• Integrated mobile phone platform• Support for general public and botanical experts

• Leaf image + auxiliary images + geo-tag + metadata

WP1 pilot use case

Image archive and leaf classifier specific enablers

3

• Leaf classifier to label unknown images• Web portal & backend service (Italian & English)• Integrated mobile phone platform

• Biodiversity ontology support• Scientific names (Latin)• Common names (Italian, English)• Domain ontology URI’s (e.g. TaxMeOn)• Natura 2000 habitat codes

• Value proposition• Supporting crowd sourced leaf observations allows image data

collection by volunteers at a scale beyond traditional methods

WP1 pilot use case

Image archive and leaf classifier specific enablers

4

Image archive architecture

Image archive and leaf classifier specific enablers

5

Image, Geotag, User Metadata

Web browser

Users (crowd sourcing)

User

Mobile Data Acquisition Framework (MDAF)

Mobile observation server

HTTP RESTfulImage archive service

OWLIM (metadata)mySQL (data)

Image archive service

Domain experts

Expert

Mobile device

Image archive UI

Image(s), Geotag(s),User metadata

Database syncronization

Expert metadata

Image records

Imagerecords

Crowd sourcing(web upload and mobile support)

Expert review of labels

Image archive user interface

Image archive and leaf classifier specific enablers

6

Leaf classifier architecture

Image archive and leaf classifier specific enablers

7

Unlabelled image

Web browser

Users (general public)

User

Mobile Data Acquisition Framework (MDAF)

Mobile observation server

HTTP RESTful SPSLeaf classifier process

OWLIM (metadata)mySQL (data)

Image classifier service

Mobile device

Leaf classifier UI

Unlabelledimage(s)

SPS request- image URI's

Classification label set(s)

Training setsignatures

SPS request- image URI's

Classificationlabel set(s)

Expert

Expert reviewed training set

Training set

Classification label set(s)

Label set

Labelset(s)

Users request classifications(unlabelled images)

Top N matches returned(leaf classifier algorithm)

• Classic benchmark datasets• e.g. Swedish leaf: 1,125 images, 15 species

• NO SHADOWS• LIMITED ROTATION

• Crowd-sourced datasets challenging!• e.g. Hillier Gardens (IT Innovation): 1400 images, 54 species

• SHADOWS• NATURAL OUTDOOR LIGHTING• ARBITRARY ROTATION

Leaf classifier algorithms

Image archive and leaf classifier specific enablers

8

• Segmentation - Colour-based Expectation-Maximization• HSV colour space; discard hue due to the high level of noise• Colour-based EM algorithm for pixel classification using k-means

clustering to initialize the EM algorithm (Belhumeur 2008)• Three clusters are considered representing: leaf; shadow and

background.

Leaf classifier algorithms

Image archive and leaf classifier specific enablers

9

P. Belhumeur, et al."Searching the World’s Herbaria: A System for Visual

Identification of Plant Species." ECCV. 2008. 116-129.

Leaf classifier algorithms

Image archive and leaf classifier specific enablers

10

Belhumeur 2008 tried segmentation with two clusters- problems handling shadows

Leaf

Shadow

Background

We use three clusters forleaf, shadow, background

- shadows eliminated

• Segmentation - Colour-based Expectation-Maximization

Leaf classifier algorithms

Image archive and leaf classifier specific enablers

11

← The 3 clusters are re-classified based on cluster’s properties. Here, both leaf and shadow clusters were subsequently classified as leaf.

• Segmentation - Examples

• Feature extraction - Inner Distance Shape Context (Ling, 2007)

• Matching - fusion of two matching methods based on confidence levels:• Point-based IDSC matching• Contour matching

Leaf classifier algorithms

Image archive and leaf classifier specific enablers

12

Inner-distance connections between sampled points

Inner-distance shape contextPoint correspondence between two images of the same class

H. Ling, D. W. Jacobs. Shape Classification Using the Inner-Distance. 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, pp. 286 - 299.

• Distinctive classes

Leaf classifier algorithms

Image archive and leaf classifier specific enablers

13

Vitex Agnus-Castus

P(Best match) = 100%Confidence = 100%

Quercus Polycarpa

P(Best match) = 100%Confidence = 99.82%

Alnus Glutinosa 'Pyramidalis‘P(Best match) = 100%Confidence = 99.66%

Platanus ’Pyramidalis’

P(Best match) = 100%Confidence = 97.60%

Acer MonspessulanumP(Best match) = 100%Confidence = 97.5%

Tilia Tomentosa'Petiolaris'P(Best match) = 100%Confidence = 81.85%

Populus Nigra

P(Best match)=93.33%Confidence = 76.67%

Rhamnus Alpina

P(Best match)=92.86%Confidence = 82.28%

Cornus Sanguinea

P(Best match)=90.32%Confidence = 74.91%

Fagus Sylvatica 'Grandidentata'P(Best match)=90.00%Confidence = 77.78%

Ulmus

P(Best match)=90.00%Confidence = 66.48%

• Erroneous results can be caused by:• Similarity between the leaf shape of different species• Error in segmentation• Insufficient number of training samples

Leaf classifier algorithms

Image archive and leaf classifier specific enablers

14

Species name P(Best match)%

Confidence%

Error in classification caused by:

Shape similarity Error in segmentation

Insufficient training samples

Carpinus Betulus 84.09 67.68 x - - Acer Saccharum subsp. Leucoderme

83.33 81.48 x - -

Sorbus Degenii 80.65 61.65 x - - Ostrya Carpinifolia 78.57 44.05 x - - Crataegus_Crus-Galli 75.86 63.22 x x - Magnolia x Soulangeana

74.19 49.64 x x -

Acer Platanoides 'Globosum'

66.67 52.96 x - -

Quercus Robur 64.29 51.19 - x x Pyrus x Michauxii 63.64 51.01 x x x Magnolia x Loebneri 63.33 57.04 x x - Fraxinus 14.29 8.73 - x x

Examples ofsimilar shapes

Acer Platanoides 'Globosum'

Acer Saccharum subsp Leucoderme

Platanus ’Pyramidalis’

Magnolia x Loebneri

Magnolia x Soulangeana

Carpinus Betulus

Ostrya Carpinifolia

Rhamnus Alpina

Ulmus

• Hillier Gardens dataset results• Current dataset: 1400 images, 54 species• Mean probability of correct first match: 85.18%• Mean confidence in correct classification: 73.88%

Leaf classifier algorithms

Image archive and leaf classifier specific enablers

15

Leaf classifier user interface

Image archive and leaf classifier specific enablers

16

Thank you for your attentionStuart E. Middleton

{sem}@it-innovation.soton.ac.ukwww.ENVIROFI.eu

twitter.com/ENVIROFI

The research leading to these results has received funding from the European Community's Seventh

Framework Programme (FP7/2007-2013) under Grant Agreement Number 284898

17