Modeling Quality Assurance Project Plans (QAPPs)

Sandra Arismendez

TCEQ Laboratory and Quality Assurance Section

July 8, 2015

QAPPs: Why Write Them?

• Because they are required – EPA Order CIO 2105.0 (formerly 5360.1 A2) sets forth

requirement for all organizations conducting environmental data operations on EPA’s behalf

– State agencies (including TCEQ and TSSWCB) and

TCEQ contractors follow suit

• So that projects can be planned and implemented effectively

Importance of Modeling QAPPs

• Helps ensure that the model(s) (and underlying data) are sufficient for achieving project objectives

• Provides documentation of specifications and activities

QAPP Requirements

• Graded Approach – Level of detail commensurate with nature of the work and intended use of data

Tools for QAPP Development

• EPA Requirements for Quality Assurance Project Plans (EPA QA/R-5) is governing requirement (http://www.epa.gov/quality/qs-docs/r5-final.pdf)

• EPA Guidance for Modeling QAPPs (EPA QA/G-5M) (http://www.epa.gov/QUALITY/qa_docs.html)

• QAPP Program Shells

• Other QAPPs

QAPP Content

• Section A – Project Management

• Section B – Measurement and Data Acquisition

• Section C – Assessment and Oversight

• Section D – Data Validation and Usability

Section A – Project Management

• A1, A2, A3 – Title Page, Signature Page, TOC, Distribution List

• A4 – Project/Task Organization • A5 – Problem Definition/Background • A6 – Project/Task Description and Schedule • A7 – Quality Objectives and Criteria • A8 – Special Training/Certification • A9 – Documents and Records

A5 – Problem Definition/Background

• Define problem, place modeling activities into overall context with the project, and explain why modeling is being conducted

• Provide justification for model(s) to be used • Summarize what questions will be answered

and/or what decisions will be made

A6 – Project/Task Description and Schedule

• Summarize work to be done

• Detail project tasks, schedule, and milestones

– Development, verification, and validation of code or software is a task (if applicable)

• Provide enough detail so it is clear as to what versions and

implementations of packaged models will be used • If multiple models are being used, any linkages should be

discussed (preferably diagrammed)

A7 – Quality Objectives and Criteria for Model Inputs/Outputs

• Inputs: Quantity, quality, and data type are relevant

– Quantity and quality: Precision, Bias, Accuracy, Sensitivity, Representativeness, Comparability, Completeness

– Type: data collection technique • Outputs: Qualitative and quantitative criteria and

measures – Quantitative: “Goodness of fit,” error and other forms of

uncertainty, sensitivity, graphical analysis – Qualitative: Reasonableness (what is reasonable should be

defined)

Sensitivity and Uncertainty Analyses

• Sensitivity analysis evaluates the effect of changes in input values or assumptions on a model’s results

• Uncertainty analysis investigates the effects of a lack

of knowledge or other potential sources of error in the model

• When used in combination, sensitivity and uncertainty analysis allow model users to be more informed about the confidence that can be placed in model results

* Text above copied directly from Guidance on the Development, Evaluation, and Application of Environmental Models, Council for Regulatory Environmental Modeling (EPA/100/K-09), USEPA, 2009.

A9 – Documents and Records

• Specify records, their contents, storage location, retention period, formats, and back-up protocols

• Examples: QAPPs, User’s Manual(s), various

reports, raw files, code, calibration records, verification/validation records

Section B – Measurement and Data Acquisition

• B7 – Calibration

• B9 – Non-direct Measurements (Data Acquisition Requirements)

• B10 – Data Management and Hardware/Software Configuration

B7 – Calibration • Identify calibration parameters (including literature

values)

• If calibration process is iterative and/or stepwise, spell out calibration steps

• Specify criteria/guidelines used to determine whether calibration is successful – Also, state what will be done if minimum requirements are

not met (e.g., recalibration with additional data, consideration of different modeling approach, etc.)

Section B9 – Non-Direct Measurements (Secondary Data)

• This section is critical as primary data collection is rarely described in modeling QAPPs

• Tables can be extremely helpful • Don’t forget the graded approach!

• If some data will be acquired that are not yet known,

this may be built into the QAPP – In this case, it is especially important to specify criteria for

acceptance and use of those data

B10 – Data Management • Details about data storage, production,

management, storage, and archival should be provided – If SOPs already exist, these may be referenced instead – If data coding or software development is taking place,

some reference to these procedures should be provided

• Hardware and software requirements/specifications should be included

• Flow diagrams can be very instructive

Section C – Assessment and Oversight

• C1– Assessments and Response Actions

• C2 – Reports to Management

C1 – Assessments and Response Actions

• Describe surveillance activities, audits, and the process for identifying, documenting, and correcting problems

• Also, describe assessments of the model performance, and the schedule for these activities

Section D – Data Validation and Usability

• D1 – Departures from Validation Criteria

• D2 –Validation Methods

• D3 – Reconciliation with User Requirements

Section D

• D1: Describe how results are evaluated and departures from procedures presented

• D2: Describe steps in the validation process

– Performance criteria from A7 should be referenced

• D3: Describe uses of data and how limitations

will be reported

Questions?