Graduate School of Geography, Clark University

Rahul Rakshitrrakshit@clarku.eduGraduate School of GeographyClark University

hero.clarku.edu/holmes

1

Accuracy assessment measures in OBIA

A survey of

Background: Pixel Vs Object based accuracy estimates

Object 1 Object 1Object 1

InaccurateGeometry

Object 1 Object 1Object 1

InaccurateClassification

Forest NonForestLegend

Pixel 1 Pixel 1Pixel 1 Inaccurate

Classification

Classification Reference Accuracy Assessment

Graduate School of Geography, Clark University 2

1. Impervious2. Grass3. Bare Soil4. Coniferous5. Deciduous

Land-cover

Scale 1:600Scale 1:150

Background: Selecting the sampling unit - Point

Graduate School of Geography, Clark University 3

1. Impervious2. Grass3. Bare Soil4. Coniferous5. Deciduous

Land-cover

Background: Selecting the sampling unit - Polygon

Graduate School of Geography, Clark University 4

1. Impervious2. Grass3. Bare Soil4. Coniferous5. Deciduous

Land-cover

WAASWide Area Augmentation System 5 m accuracy

Background: Positional Accuracy of a GPS

Graduate School of Geography, Clark University 5

Selection Criteria: 100 papers

2002 2003 2004 2005 2006 2007 2008 2009 2010 20110

10

20

30

1

7 8 8

15

4

30

14

10

3

Year

No.

of P

aper

s

Conference Proceedings Book Chapters Peer Reviewed Articles0

10

20

30

40

50

60

70

24

7

69

Source

No.

of P

aper

s •IJRS•PE&RS•RS of Env•Sensors•CJRS•Landscape & Planning

Studies that have used OBIA as a classification tool to create thematic maps

Graduate School of Geography, Clark University 6

Remote sensing accuracy assessment

Sampling DesignMethod for choosing locations at which reference class will be determined

Response Design

Analysis

Method for determining the reference class

Agreement between classified Vs reference datasets, results

Stehman (1998), Congalton and Green (2008)

Graduate School of Geography, Clark University 7

Data Description●Paper ●Authors ●Year of Publication ●Accuracy Assessment Performed

Classification Design●Theme ●Year of Data ●Sensor ●Resolution ●No. of Classes ●Software

Segmentation Properties●Segmentation Algorithm ●Segmentation Scale

Sampling Design●Sampling Method ●Total samples ●Sampling Units ●No. of samples per class

Response Design

●Reference Data Source ●Independent validation source●Difference in years between reference and training data

Analysis●Error Matrix ●Alternate error estimation ●Overall Accuracy

Survey Attributes

Graduate School of Geography, Clark University 8

AA = accuracy assessment Blank Space = no data available

Sample Data

Graduate School of Geography, Clark University 9

88%

12%

Accuracy assessment permformed Accuracy assessment not permformed

Results:

• Accuracy estimated as sufficient/reasonable.

• Accuracy measured by visually comparing the OBIA outputs with the imagery.

• Accuracy assessment to be conducted in the future.

Graduate School of Geography, Clark University 10

5050

Total no. of samples No Information

No. of SamplesNo

sam

plin

g met

hod

men

tione

d

60%

Random17%

StratifiedRandom

19%

Stratified 3%Accessible

1%

13

No. of samples per class

Sampling Design

Sampling Method

39

24

37

Points Polygons No Information

Sampling Unit

Graduate School of Geography, Clark University 11

Response Design

No Information

Field Data (GPS)

35%

Image Interpretation

28%

Other ThematicMaps 9%

28%

28

72

Year of Reference Data No Information

56

10

34

Yes No No Information

Independent source of validation

Source of reference data

Graduate School of Geography, Clark University 12

Analysis

Error Matrix61%

Kappa 3%

No Matrix

Visual Estimate10%

5%

No Information12%

Only Overall Accuracy

9%

Graduate School of Geography, Clark University 13

Conclusions

Graduate School of Geography, Clark University 14

1. Most of the studies do not mention the sampling method2. Both points and polygons are used as sampling units3. Almost half of the studies do not provide information on total

number of samples

4. Most of the studies use field data (GPS) as reference datasets5. Majority of the studies separate validation data from

classification data6. Majority of the studies do not provide information on year of

reference datasets

7. Majority of the studies use the error matrix to show accuracy assessment results

8. Some studies have visually estimated the agreement between classified and reference data

Recommendations:

Graduate School of Geography, Clark University 15

• Temporal difference between classification and reference datasets should be kept to minimum• Avoid misregistration (low positional accuracy)• Take spatially well distributed samples (spatial autocorrelation)• Use stratified random sampling• Specify sample size for each stratum• Use thematic maps as reference datasets with caution• Mask out training data from sampling design• Hybrid point and polygon sampling approach (Albert Decatur: Upcoming)

• Object geometry accuracy can be quantified by tools such as LIST (Landscape Interpretation Support Tool) and CI (Comparison Index) that use polygon overlay

• Data quality: Jarlath O’Neil Dunne http://letters-sal.blogspot.com/2010/08/is-peer-reviewed-literaturethe-best.html

Graduate School of Geography, Clark University

• Prof. Robert Gilmore Pontius, Jr.• Prof. Colin Polsky• Prof. John Rogan• Albert Decatur• Shitij Mehta• Jarlath O’ Neil Dunne

More Information: rrakshit@clarku.edu http://hero.clarku.edu/holmes

Acknowledgements

This material is based upon work supported by the National Science Foundation (NSF) under grant Nos. BCS-0709685 (Coupled Natural-Human Systems), OCE-0423565 (Long-Term Ecological Research), SES-0849985 (REU Site), and BCS-0948984 (ULTRA-ex), and by the Clark University O'Connor '78 Endowment.  Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funders.

16