29
NR 422 Quality Control Jim Graham Spring 2009

NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Embed Size (px)

Citation preview

Page 1: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

NR 422Quality Control

Jim Graham

Spring 2009

Page 2: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Staircase of Knowledge

Increasing Subjectivity

Hum

an v

alue

add

ed

ObservationAnd

Measurement

Data

Information

Knowledge

Understanding

Wisdom

OrganizationInterpretation

Verification

SelectionTesting

ComprehensionIntegration

Judgment

Environmental Monitoring and Characterization, Aritola, Pepper, and Brusseau

Page 3: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Error

• Data does not match reality (ever)

• Gross errors

• Accuracy (bias): distance from truth– | Measurement mean – Truth |

• Precision: variance within the data– Standard Deviation (stddev)

• Measurement Limits

Page 4: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Accuracy and Precision

High AccuracyLow Precision

http://en.wikipedia.org/wiki/Accuracy_and_precision

Low AccuracyHigh Precision

Page 5: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Bias (Accuracy)

• Bias = Distance from truth

Truth Mean

Bias

Page 6: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Standard Deviation (Precision)

Each band represents one standard deviationSource: Wikipedia

Page 7: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Other Approaches

• Confidence Intervals

• +- Some range (suspect)

Page 8: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Sources of Error• Measurement Error

– Protocol– User– Instrument

• Processing Errors– Procedure– User – Instrument

• Data Errors– Age– Metadata/Documentation

Page 9: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Protocol

• Rule #1: Have one!

• Step by step instructions on how to collect the data– Calibration– Equipment required– Training required– Steps– QAQC

• See Globe Protocols:– http://www.globe.gov/sda/tg00/aerosol.pdf

Page 10: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Protocol Error

• Is there a protocol?

• What is being measured?

• Is it complete: How large? How small?

• Unexpected circumstances (illness, weather, accidents, equipment failures, changing ecosystems)

Page 11: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

User Measurement Errors

• Wrong Datum

• Data in wrong field/attribute

• Missing data

• Gross errors

• Precision and Accuracy

• Observer error: expertise and “drift”

Page 12: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Instrument Errors

• Calibration

• Drift

• Humans as instruments:– DBH– Weight– Humans are almost always involved!– Fortunately we can be calibrated and have

our drift measured

Page 13: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Calibration

• Sample a portion of the study area repeatedly and/or with higher precision– GPS: benchmarks, higher resolution– Measurements: lasers, known distances– Identifications: experts, known samples

• Use bias and stddev throughout study

• Also provides an estimate for min/max

Page 14: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Flow of error

• Capture error during data collection

• Determine error of other datasets– If unavailable, estimate the error

• Maintain error throughout processing– Error will increase

• Document final error in reports and metadata

Page 15: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Processing Error

• Error changes with processing

• The change depends on the operation and the type of error:– Min/Max– Average Error– Standard Error of the Mean– Standard Deviation– Confidence Intervals

Page 16: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Combing Bias

• Add/Subtraction:– Bias (Bias1+Bias2)=

• T- (Mean1*Num1+Mean2*Num2)/(Num1*Num2)

• Simplified: (|Bias1|+|Bias2|)/2

• Multiply Divide:– Bias (Bias1*Bias2)=

• T- (Mean1*Mean2)• Simplified: |Bias1|*|Bias2|

Derived by Jim Graham

Page 17: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Combining Standard Deviation

• Add/Subtract:– StdDev=sqrt(StdDev1^2+StdDev2^2)

• Multiply/Divide:– StdDev=

• sqrt((StdDev1/Mean1)^2+(StdDev2/Mean2)^2)

http://www.rit.edu/cos/uphysics/uncertainties/Uncertaintiespart2.html

Page 18: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Exact numbers

• Adding/Subtracting:– Error does not change

• Multiplying:– Multiply the error by the same number

Page 19: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Significant Digits (Figures)

• How many significant digits are in:– 12– 12.00– 12.001– 12000– 0.0001– 0.00012– 123456789

• Only applies to measured values, not exact values (i.e. 2 oranges)

Page 20: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Significant Digits

• Cannot create precision:– 1.0 * 2.0 = 2.0– 12 * 11 = 130 (not 131)– 12.0 * 11 = 130 (still not 131)– 12.0 * 11.0 = 131

• Can keep digits for calculations, report with appropriate significant digits

Page 21: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Rounding

• If you have 2 significant digits:– 1.11 -> ?– 1.19 -> ?– 1.14 -> ?– 1.16 -> ?– 1.15 -> ?– 1.99 -> ?– 1.155 -> ?

Page 22: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Quality Control/Assurance

• Calibrate “Instruments”

• Perform random checks on data

• Watch for “drift”

• Document all errors in Metadata!

Page 23: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Design of Sampling

• Random

• Stratified random

• Clustered

• Systematic

• Iterative

Page 24: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Number of Samples

• 30?

• Figure 2.7 from Environmental Monitoring and Characterization

Page 25: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Statistical Studies

• Is the sampling really random or uniform?– Bias– “Most data is collect near a road, a porta-

poty, and a restaurant!” – Tom Stohlgren

Page 28: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Spatial Autocorrelation

• Used to determine type of sampling

Page 29: NR 422 Quality Control Jim Graham Spring 2009. Staircase of Knowledge Increasing Subjectivity Human value added Observation And Measurement Data Information

Rounding

• If you have 2 significant digits:– 1.11 -> 1.1– 1.19 -> 1.2– 1.14 -> 1.1– 1.16 -> 1.2– 1.15 -> 1.1– 1.99 -> 2.0– 1.155 -> 1.5