LCMAP Validation: An annual time-series land cover reference dataset

Methods and QA/QC results

Bruce Pengra, SGT, Contractor to the USGS

U.S. Geological Survey / EROS

November 7, 2018

USGS / USFS Collaboration

LCMAP / EROS (USGS)LCMS / Utah State University (USFS)

Reference Data needs:

• Characterize and validate the LCMAP annual land cover map

• Use with map layer to estimate of land cover and land cover

change

• Meet Forest Service / LCMS requirements (validation, model

training, etc.)

LCMAP Annual Land CoverCONUS/National Coverage

Landsat resolutionAnnual land cover 1985 to current

Ongoing annual updates

Conus LCMAP annual land cover simulated from NLCD data

LCMAP Annual Land CoverCONUS/National Coverage

Landsat resolutionAnnual land cover 1985 to current

Ongoing annual updates

Conus LCMAP annual land cover simulated from NLCD data

Reference Data Requirements

per LCMAP land cover validation

▪ Temporal coverage 85 to current

▪ Spatial coverage, CONUS▪ Landsat resolution (and grid)▪ Classified to match Land Cover

Trends legend▪ Can be updated annually ▪ Within LCMAP budget

▪ Delivered at the same time LCMAP products are

Methods for Meeting those Reference Data Requirements

➢ The Data & Tools

➢ The Response Design

➢ The Sample Design

➢ The QA/QC process

➢ Application

Base Sample of 25,000 plots

Temporal Coverage 1985 -current

Using sate l l i te data and/or aer ia l imagery for large area land cover accuracy assessment is wel l establ ished

Landsat time series data

1, 2, 3, 4, 5, 6, 7, 8, 9, 10

Google Earth high resolution imagery

Tools to make the process efficient

TimeSync, developed at OSU for US Forest Service.

Displays Landsat anniversary date image

Plots values through time series

Records and manages interpretations

The TimeSync Application manages Landsat data display and recording of interpretations for pre-defined sample locations

Plotted values for the sample pixel, showing values for all clear pixels

Anniversary date Landsat images, selected by the user from the usable images in each year

Google Earth

Efficient access to a time-series of high resolution images

FS = Forest Sample PlotLabel

GS = Wetland

Forest Service

Crosswalk

LCMAP Crosswalk

All Variables

Labeling Protocol

Land Use / Land Cover / Change

Methods for Meeting those Reference Data Requirements

➢ The Data & Tools

➢ The Response Design

➢ The Sample Design

➢ The QA/QC process

➢ Application

Methods for Meeting those Reference Data Requirements

➢ The Data & Tools

➢ The Response Design

➢ The Sample Design

➢ The QA/QC process

➢ Application

Joint Response DesignTodd Schroeder – USFS Warren Cohen – USFS/OSU

Interpretation Protocol • land use variables• land cover variable• land change processes

Joint Response Design

Land Use:(primary and secondary)

DevelopedForestAgricultureNon-forest WetlandRangeland/PastureOther

Land Cover:(primary and secondary)

TreesShrubsGrass/Forb/HerbImperviousBarrenSnow/Ice Water

Change Processes:FireHarvestMechanicalStructural DeclineSpectral DeclineWindHydrologyDebrisGrowth/RecoveryStableOther

Methods for Meeting those Reference Data Requirements

➢ The Data & Tools

➢ The Response Design

➢ The Sample Design

➢ The QA/QC process

➢ Application

Methods for Meeting those Reference Data Requirements

➢ The Data & Tools

➢ The Response Design

➢ The Sample Design

➢ The QA/QC process

➢ Application

The Sample Design

The Sample Design

The Sample Design

The Sample Design

The Sample Design

Methods for Meeting those Reference Data Requirements

➢ The Data & Tools

➢ The Response Design

➢ The Sample Design

➢ The QA/QC process

➢ Application

Methods for Meeting those Reference Data Requirements

➢ The Data & Tools

➢ The Response Design

➢ The Sample Design

➢ The QA/QC process

➢ Application

The QA/QC process

Validation of the Validation

The literature repeatedly points out that no perfect ground truth data exists with which to compare validation data 13,14,15,16,17,18,19

With no absolute “gold standard” available, Olofsson (2014)13 suggests that, “in practice, only assessing interpreter variability may be feasible.”

The QA/QC process

Our primary QA/QC activity is based on comparison of two independent interpretations (at ~60% of all plots)

• Agreement between these independent interpretations provides a measure of interpretation consistency that was used for quality control and for inferring the consistency of the full data set.

Assessment of the agreement (prior to review-and-revision) is calculated from a random sample of approximately 25% of all plots

• The assumption is that this is a baseline agreement number that will be improved by review and revision

The QA/QC process

Three goals:

1. Improve the interpreter consistency by providing feedback

2. Improve the data by correcting interpretation mistakes

3. Quantify the variability of the reference data – pre review

The QA/QC process

Following collection of each set of plots (~1000 plots each), several forms of feedback are provided to the interpreters

An individual contingency matrix showing where that interpreter had disagreed with interpretations by the other members of the interpretation team was provided.

Inte

rpre

ter

13

7

Interpreter 137 versus all other interpreters (set 10)

Set 10 Other Interpreters

Water Developed Disturbed Barren Forest Grass-shrub Ag Wetland Total Agreement %

Water 151 1 13 165 92

Developed 262 5 267 98

Disturbed 11 14 12 3 2 1 43 33

Barren 1 1 0

Forest 28 1706 8 97 1839 93

Grass-shrub 68 17 65 156 2204 96 94 2700 82

Ag 2 32 328 362 91

Wetland 1 33 64 98 65

Total 151 341 68 65 1907 2248 426 269 5475

4729agreement pixels

Agreement % 100 77 21 0 89 98 77 24 86.4overall agreement %

The QA/QC process

Following collection of each set of plots (~1000 plots each), several forms of feedback are provided to the interpreters

Review Notes

Plot 19497: Illogical call of primary use/cover Rangeland/Trees by 105. Tree line is less than 120 ft wide and therefore does not meet Forest criteria as 146 has. Use should be Other with dominant Tree cover. NHAP 1985 high resolution seems to show same (majority) tree line as later in time series.

Plot 19732: Both call GFH Rangeland but 109 includes a Fire in 2008 while 105 calls Stable. Review of NBR and chips does support a Fire in 2008 that was missed by 105.

Plot 20000: Illogical primary use/cover of Rangeland/Trees by 105.

Answers to Questions submitted by interpreters

The QA/QC process

Following collection of each set of plots (~1000 plots each), several forms of feedback are provided to the interpreters

Topical reviews of best practices and interpretation tips for classes of concern

The QA/QC process

All QA/QC duplicate plots (60% of all plots)

Overall between interpreter agreement for each of the first 22 sets

The QA/QC process

Some classes showed especially good improvement over time

The QA/QC process

Three goals:

1. Improve the interpreter consistency by providing feedback

2. Improve the data by correcting interpretation mistakes

3. Quantify the consistency of the reference data – pre review

Three goals:

1. Improve the interpreter consistency by providing feedback

2. Improve the data by correcting interpretation mistakes

3. Quantify the consistency of the reference data – pre review

The QA/QC process Improving the data by correcting interpretation mistakes

60% of all plots get a second interpretation

These plots are candidates for review based on a variety of agreement criteria• Overall agreement• Any disagreement related to change process• Belong to leading categories of disagreement

34% of duplicate plots are flagged for review by senior interpreters based on these criteria • Reviewers can identify the better interpretation of the two or suggest other revisions when it is clear an error was made• Instructions are recorded to revise the original interpretation where needed

Corrections are made to approximately 12% of all reference data plots to arrive at the final data• Corrections range from minor changes in a single year to major changes for all 33 years • They tend to be minor, such as recording a fire or harvest in the wrong year

The QA/QC process

Three goals:

1. Improve the interpreter consistency by providing feedback

2. Improve the data by correcting interpretation mistakes

3. Quantify the consistency of the reference data – pre review

The QA/QC process

Three goals:

1. Improve the interpreter consistency by providing feedback

2. Improve the data by correcting interpretation mistakes

3. Quantify the consistency of the reference data – pre review

The QA/QC process

The Random sampleBetween analyst agreement for duplicate interpretations at 5085 plots (x33 years) selected randomly for estimating initial agreement among interpreters over the full data set.

B Interpretations

Develope Disturbe Grass-Water d d Barren Forest shrub Ag Wetland Ice-Snow Total A agree

Water 8579 58 51 3 33 117 123 8964 95.71%

s Developed 76 6204 87 6 382 493 182 18 7448 83.30%

n Disturbed 15 81 839 33 375 189 65 113 1710 49.06%

atio

Barren 3 1 14 972 31 1032 1 2054 47.32%

et Forest 421 312 42130 2774 166 692 46495 90.61%

Inte

rpr

Grass-shrub 91 622 210 956 2425 55353 2467 380 62504 88.56%

A

Ag 46 292 94 14 182 2283 27740 87 30738 90.25%

Wetland 280 86 911 489 99 5975 7840 76.21%

Ice-Snow 33 33

147825

88.10%

100.00%

agreement pixels

overall agreement

Total 9090 7679 1693 1984 46469 62730 30720 7388 33 167786

B agree 94.38% 80.79% 49.56% 48.99% 90.66% 88.24% 90.30% 80.87% 100.00%

Methods for Meeting those Reference Data Requirements

➢ The Data & Tools

➢ The Response Design

➢ The Sample Design

➢ The QA/QC process

➢ Application

Application

➢ Validation and Characterization of LCMAP annual land cover

➢ Validation and characterization of LCMAP land cover change

➢ Estimation of national land cover composition

➢ Estimation of national land cover composition change

Application

Validation and Characterization of LCMAP annual land cover Comparison of the Map and Reference data land cover labels at each of the 25,000 sample points for each of 33 years

~825,000 points of comparison

Application

Estimation of national land cover composition & composition change

Class 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

Water 1.7% 1.8% 1.8% 1.8% 1.8% 1.7% 1.7% 1.7% 1.7% 1.8% 1.8% 1.8% 1.8% 1.8% 1.9% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.8% 1.9% 1.9%

Developed 4.2% 4.2% 4.2% 4.3% 4.3% 4.4% 4.4% 4.4% 4.5% 4.5% 4.6% 4.6% 4.7% 4.7% 4.8% 4.8% 4.9% 5.0% 5.1% 5.2% 5.3% 5.4% 5.5% 5.6% 5.5% 5.6% 5.6% 5.6% 5.7% 5.7% 5.8% 5.8% 5.8%

Disturbed 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%

Barren 0.6% 0.7% 0.6% 0.6% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7% 0.7%

Forest 29.7% 29.7% 29.6% 29.5% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.3% 29.4% 29.4% 29.4% 29.4% 29.4% 29.3% 29.3% 29.2% 29.3% 29.4% 29.3% 29.4% 29.4% 29.3% 29.2% 29.1% 29.0%

GrassShrub 38.4% 38.5% 38.6% 38.7% 39.0% 39.1% 39.3% 39.4% 39.5% 39.4% 39.4% 39.4% 39.3% 39.3% 39.2% 39.2% 39.1% 39.1% 39.2% 39.2% 39.2% 39.1% 39.1% 39.2% 39.1% 39.0% 39.0% 38.8% 38.7% 38.7% 38.6% 38.7% 38.7%

Agriculture 20.2% 20.1% 20.0% 20.0% 19.9% 19.7% 19.5% 19.4% 19.3% 19.3% 19.2% 19.2% 19.2% 19.2% 19.2% 19.1% 19.0% 18.9% 18.7% 18.6% 18.6% 18.5% 18.5% 18.4% 18.4% 18.4% 18.4% 18.5% 18.7% 18.7% 18.8% 18.8% 18.9%

Wetland 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.0% 5.0% 5.1% 5.1% 5.0% 5.0% 5.0% 5.0% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1% 5.1%

Ice&Snow 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%

Application

Regional versions of the validation and area estimate applications