Fundamentals of accuracy_assessment_session2_czaplewski

Session 2

Fundamentals of Accuracy Assessment

Raymond L Czaplewski

United States Forest Service

Rocky Mountain Research Station

Fort Collins, Colorado USA

1

Session 2 Topics

• Different sample designs– Simple Random Sampling (Systematic Sampling)

– Stratified Random Sampling

• Different sample survey estimators

• Different sample sizes, n=30, 60, 150

• How close are estimates to true value?

• Example of a 30×30 = 900 pixel world

2

N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ NN N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N NN N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N NN N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ NN N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N NN N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ NN N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^N N N N N N N ^ N N N N N N N N N N N N N N ^ ^ ^ ^ N ^ N NN N N N N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N ^ ^ ^ ^

True (reference) population

Hypothetical “real world”

3

Nat

ural

Urb

an

Crop

N ^

Reference class30×30 = 900 pixels

Hypothetical remotely sensed thematic map model for this “real world”

Map #1N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ NN ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ NN N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N

N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N NN N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^N N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ NN ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ NN ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^^ N N N N N ^ ^ N N N N N ^ N N N N N N N ^ N N ^ N N ^N N N N ^ N ^ N ^ N N N N ^ N N N N ^ ^ ^ ^ ^ ^ N N ^ N ^ 4

Natural NUrbanCrop ^M

ap c

lass

30×30 = 900 pixels

N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ NN ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ NN N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N

N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N NN N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^N N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ NN ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ NN ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^^ N N N N N ^ ^ N N N N N ^ N N N N N N N ^ N N ^ N N ^N N N N ^ N ^ N ^ N N N N ^ N N N N ^ ^ ^ ^ ^ ^ N N ^ N ^

N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ NN N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N NN N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N NN N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ NN N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N NN N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ NN N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^N N N N N N N ^ N N N N N N N N N N N N N N ^ ^ ^ ^ N ^ N NN N N N N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N ^ ^ ^ ^

Remotely sensed thematic Map #1

True (reference) population Map #1

Error matrix presented by Steve Stehman

Traditional Analysis: Error (Confusion) Matrix

Reference Land CoverMapped Natural Urban Crop TotalNatural 0.25 0.03 0.08 0.36Urban 0.02 0.12 0.04 0.18Crop 0.10 0.04 0.32 0.46Total 0.37 0.19 0.44

Natural Urban Crop TotalNatural 226 27 74 327

Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900

69% kappa 51%

Natural Urban Crop TotalNatural 25% 3% 8% 36%

Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%

Overall accuracy

Reference class

Map

cla

ss

Reference class

Map

cla

ss

5

30×30 = 900 pixels

True Error Matrix

True error matrix parameters


Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900

69% kappa 51%Overall accuracy

Reference class

Map

cla

ss


Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%

Reference class

Map

cla

ss

6

True error matrix parameters, graphical presentation

0% 50% 100%

True Reference Land Cover area

True Map Land Cover areaNatural

Urban

Crop


Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900


Reference class

Map

cla

ss


Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%

Reference class

Map

cla

ss

7

44%37% 19%


0% 50% 100%

True Reference Land Cover area

True Map Land Cover areaNatural

Urban

Crop


Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900


Reference class

Map

cla

ss


Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%

Reference class

Map

cla

ss

8

46%36% 18%

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappa

True

Producer’s AccuracyUser’s Accuracy



Urban 11% 67% 22% 100%Crop 22% 9% 70% 100%

User's AccuracyReference class

Map

cla

ss

Natural Urban CropNatural 68% 16% 19%

Urban 5% 63% 9%Crop 27% 21% 72%Total 100% 100% 100%

Producer's AccuracyReference class

Map

cla

ss

Overall accuracy 69% kappa 51%

9

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappa

True




Urban 11% 67% 22% 100%Crop 22% 9% 70% 100%


Map

cla

ss


Urban 5% 63% 9%Crop 27% 21% 72%Total 100% 100% 100%


Map

cla

ss


10


0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappa

True


Urban 11% 67% 22% 100%Crop 22% 9% 70% 100%


Map

cla

ss


Urban 5% 63% 9%Crop 27% 21% 72%Total 100% 100% 100%


Map

cla

ss



11


In the real world, we do not know the true classification for all 900 pixels

True error matrix

12

? ? 900


ap c

lass

Nat

ural

Urb

an

Crop

N ^

Reference class


In the real world, we do know the true classification for 30 sampled pixels

Sample of true (reference) population True error matrix

13

? 900


ap c

lass

Nat

ural

Urb

an

Crop

N ^

Reference class

In the real world, we do know the true classification for 30 sampled pixels

Sample of true (reference) population


Urban 0 4 2 6Crop 2 2 10 14Total 10 6 14 30

73% kappa 58%


Urban 0% 13% 7% 20%Crop 7% 7% 33% 47%Total 33% 20% 47% 100%

Map

cla

ss

Overall accuracyReference class

Map

cla

ss

Reference class

Error matrix estimate from sample

True error matrix

14

? 900

In the real world, we do not know the true classification for all 900 pixels

• Let us leave the real world for the next 30 minutes to compare– Known estimate of an error matrix with a sample

of 30 pixels

– Unknown true error matrix for all 900 pixels

15

Comparison of true (unknown) error matrix with (known) sample estimate

16


Urban 0 4 2 6Crop 2 2 10 14Total 10 6 14 30


Reference class

Map

cla

ss


Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900

69% kappa 51%

Map

cla

ss

Overall accuracy

Reference classError matrix estimate from sampleTrue (unknown) error matrix

Examples of random sampling error, simple random sample #1, sample size n=30

Area of each Land Cover Type

User's Accuracy Producer's Accuracy

0% 50% 100%

True Reference Land Cover areaSample Reference Land Cover area

True Map Land Cover areaSample Map Land Cover area

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

17




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

18




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

19




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

20




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

21

But how good is the sample estimate? Example, Producer’s Accuracy Urban

22



0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True



0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

Example: Producers accuracy for urban

23

Estimated Producer's Accuracy = 80%

`

0% 50% 100%

Natural

Urban

Crop



0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

Sample #1, n=60

24


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples

0 20 40 60 80 100% Producers Accuracy

Truth = 63%

Sample #2, n=60

25


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #3, n=60

26


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #4, n=60

27


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #5, n=60

28


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #6, n=60

29


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #7, n=60

30


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #8, n=60

31


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #9, n=60

32


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #10, n=60

33


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #11, n=60

34


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #12, n=60

35


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #13, n=60

36


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #14, n=60

37


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #15, n=60

38


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #16, n=60

39


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #17, n=60

40


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #18, n=60

41


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #19, n=60

42


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #20, n=60

43


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #21, n=60

44


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #22, n=60

45


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #23, n=60

46


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #24, n=60

47


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #25, n=60

48


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #26, n=60

49


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #27, n=60

50


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #28, n=60

51


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #29, n=60

52


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #30, n=60

53


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #31, n=60

54


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #32, n=60

55


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #33, n=60

56


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #34, n=60

57


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #35, n=60

58


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #36, n=60

59


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #37, n=60

60


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #38, n=60

61


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #39, n=60

62


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #40, n=60

63


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #41, n=60

64


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #42, n=60

65


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #43, n=60

66


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #44, n=60

67


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #45, n=60

68


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #46, n=60

69


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #47, n=60

70


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #48, n=60

71


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #49, n=60

72


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

Sample #50, n=60

73


`

0% 50% 100%

Natural

Urban

Crop

0

5

10

Num

ber o

f sam

ples


Truth = 63%

0

100

200

300

Num

ber o

f sam

ples



`

0% 50% 100%

Natural

Urban

Crop

74

True accuracy = 63%

Truth = 63%

0

100

200

300

Num

ber o

f sam

ples



`

0% 50% 100%

Natural

Urban

Crop

In the real world, we do not know the true value

75

True accuracy = 63%

0

5

10

Num

ber o

f sam

ples


In the real world, we do not know the true value, and we have only 1 sample

76


`

0% 50% 100%

Natural

Urban

Crop

Examples of random sampling error, simple random sample

• Any single sample estimate can differ from true error matrix from random sampling error

• Given our only sample with n=60, the estimated urban producers accuracy = 78% even though the true value is 63%

• However, the sample estimate is expected to equal the true value over all possible samples

77

Examples of random sampling error, simple random sample

• How can we improve reliability of estimate?

• What if sample size increased from n=60 to n=150?

78




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

79




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

80




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

81




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

82




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

83

As sample size increases from n=60 to n=150,more potential sample estimates are closer to the unknown true value

84

0

100

200

300

400

Num

ber o

f sam

ples


63%

0

100

200

300

400

Num

ber o

f sam

ples


63%

n=60 n=150

• When we do not know the true value and we have only 1 sample,• Chances that our 1 estimate is closer to the true value increases as

the sample size increases

Examples of random sampling error, simple random sample, sample size n=150

• Effects of random sampling error become smaller as the sample size becomes larger

• Sampling error affected by sample size, not proportion of population sampled

• Sample size n=1 000 very accurate estimate of true error matrix

• 1 000 pixels = 0.01% of 1 000 000 pixel area

• 10% of a 1 000 000 pixel area = 100 000 pixels

85


86

0

100

200

300

400

Num

ber o

f sam

ples


63%

0

100

200

300

400

Num

ber o

f sam

ples


63%

n=60 n=150

• When we do not know the true value and we have only 1 sample,


87

63% 63%

n=60 n=150

• When we do not know the true value and we have only 1 sample,• Variance estimate gives an idea of spread among potential sample

estiamtes

0

100

200

300

400

Num

ber o

f sam

ples


0

100

200

300

400

Num

ber o

f sam

ples



88

n=60 n=150

• When we do not know the true value and we have only 1 sample,• Variance estimate gives an idea of spread among potential sample

estiamtes

0

100

200

300

400

Num

ber o

f sam

ples


0

100

200

300

400

Num

ber o

f sam

ples


Alternative Estimators

• Sample selection and sample estimator are separate, but related, components

• Following example– Simple Random Sample, n=30

– Unbiased Simple Random Sample Estimator

– Unbiased Stratification Estimator

89

Estimator = Simple Random Sample n=30 Sample #51

Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%

Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%


Map

cla

ss

Reference class

Map

cla

ss

Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327

Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900

Map

cla

ss

Pixel mapSample estimateReference class

Map

cla

ss

90




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss

Estimator = Stratification n=30 Sample #51



Map

cla

ss


Map

cla

ss

18%2% = 3%27%

91




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

18%9% = 13%27%

92




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

18%7% = 10%27%

93




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

94




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

46%8% = 7%37%

95




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

96




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

46%37% = 30%37%

97




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

98




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

36%26% = 27%37%

99




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

100




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

36%10% = 10%37%

101




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss

Reference class

Map

cla

ss




Map

cla

ss


Map

cla

ss

Stratification Estimator

Simple Random Sampling Estimator

102

Estimator = Simple Random Sample n=30 Sample #51



0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

103




0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

104

Alternative Estimators

• Stratification and Simple Random Sampling Estimators are both unbiased applied to the same Simple Random Sample

• Stratification Estimator is expected to be more precise than Simple Random Sampling Estimator when applied to the same Simple Random Sample

105

Alternative Sampling Designs

• Simple Random Sampling– Equal probability sampling regardless of map– Sample selection at any time– n = 150

• Pre-Stratified Sampling– More intense sampling of pixels in more rare map

categories– Sample selection after map categories known– nNatural = nUrban = nCrop = 50– n = nNatural + nUrban + nCrop = 150

106

True Sample 0 Total TotalNatural 38 4 8 50 Natural 15%

Urban 5 34 11 50 Urban 31%Crop 12 6 32 50 Crop 12%Total 55 44 51 150

Map

cla

ss

Sampling IntensityReference class

Map

cla

ss

Pre-Stratified Sample

True Sample 0 Total TotalNatural 45 6 11 62 Natural 19%

Urban 3 20 4 27 Urban 17%Crop 14 7 40 61 Crop 15%Total 62 33 55 150

Map

cla

ss

Sampling IntensityReference class

Map

cla

ss

Simple Random Sample

Sample Design = Simple Random Sample Sample #52 n=150 Sample Design = Pre-Stratified Sample #53 n=150

All map pixelsTrue Sample 0 Total

Natural 226 27 74 327Urban 18 108 36 162

Crop 89 36 286 411Total 333 171 396 900

Map

cla

ss

True value from all pixelsReference class

Pre-Stratified Sample #13

107

Simple Random Sample #12

Sample Design = Simple Random Sample Sample #52 n=150

Estimator = Simple Random Sampling

108



0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

Sample Design = Pre-Stratified Sample #53 n=150Estimator = Simple Random Sampling

109



0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

Sample Design = Pre-Stratified Sample #53 n=150

110


Urban 2% 12% 4% 18%Crop 11% 5% 29% 46%Total 40% 21% 39% 100%


Reference class

Map

cla

ss


Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%


Reference class

Map

cla

ss


Urban 3% 23% 7% 33%Crop 8% 4% 21% 33%Total 37% 29% 34% 100%


Reference class

Map

cla

ss

Simple Random Sampling Estimator

True error matrix Stratification Estimator

Sample Design = Pre-Stratified Sample #53 n=150Estimator = Pre-Stratified Design

111



0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

Sample Design = Pre-Stratified Sample #53 n=150Estimator = Simple Random Sampling

112



0% 50% 100%



Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Natural

Urban

Crop

0% 50% 100%

Overall accuracy

kappaSample

True

Summary

• Estimator must be compatible with sampling design to be unbiased

• Simple Random Sampling– Simple Random Sample Estimator Unbiased

– Stratification Estimator Unbiased

• Pre-Stratified Sampling – Simple Random Sample Estimator Biased

– Stratification Estimator Unbiased

113

Comments

• Stratification estimator improves estimates for Simple Random Samples and Pre-Stratified Samples

• Variance formulae differ among estimators– The variance estimator for Simple Random

Sampling will be biased for other estimators

– Congalton and Green(2009)

114

Lessons Learned

• Unbiased sample estimates requires compatibility between sampling design and sample survey estimator

• Chances of a sample estimate being close to true (unknown) value improves with larger sample sizes

115

116

Natural Urban Crop Total Natural Urban Crop TotalNatural 226 27 74 327 Natural 206 35 60 300

Urban 18 108 36 162 Urban 27 226 47 300Crop 89 36 286 411 Crop 68 39 193 300Total 333 171 396 900 Total 300 300 300 900

Natural Urban Crop Total Natural Urban Crop TotalNatural 25% 3% 8% 36% Natural 23% 4% 7% 33%

Urban 2% 12% 4% 18% Urban 3% 25% 5% 33%Crop 10% 4% 32% 46% Crop 8% 4% 21% 33%Total 37% 19% 44% 100% Total 33% 33% 33% 100%

Overall accuracy 0.69 kappa 51% Overall accuracy 0.69 kappa 54%

True value from all pixels

Reference class Reference class

Normalized value from all pixelsReference class Reference class

Map

cla

ssM

ap c

lass

Map

cla

ssM

ap c

lass

Normalization (MARGFIT) modifies error matrix so that margins are equal

117

Which error matrix is more accurate?

Education

Fundamentals of accuracy_assessment_session2_czaplewski