Level 1 Water Audit Validation

North American Water Loss Conference 2017

Lucy Andrews

What Is Typical Water Loss in the US?

Collecting Water Loss Data

CA DRBC GA TN TX

total audits 300 517 452 629 2,646

# of unrealistic audits 100 130 74 122 1,065

% of unrealistic audits 33% 25% 16% 19% 40%

R. Sturm, K. Gasner, and L. Andrews. 2015. Water Audits in the United States: A Review of Water Losses and Data Validity (Project 4372B). Denver, CO: Water Research Foundation.

Something Is Amiss…

What’s the problem?

How do I fix it?Water Research Foundation 4639: Level 1 Water Audit Validation

Data Quality Matters!

inaccuracy & uncertainty in

inputs

inaccuracy & uncertainty in

results

• Instruments

• Databases

• People

• Missing information

Sources of error:

What Does Validation Look Like Abroad?

― IWA – validation is recommendedconsider reliability (qualitative) and accuracy (quantitative)

― Australia – third-party validation every three yearsprompted by drought (2004)

― Quebec – minimum night flow analysisannual Ministry validation interview with all agencies

Audit collection programs also exist in Austria, England, and Denmark.

What Does Validation Look Like in the US?

― Georgia – level 1 validation is required

started in 2012; first three years of data were validated by a state-funded third party

― California – level 1 validation is requiredstarted in 2017; first year of data was validated by state-funded third party

― AWWA WADI – third-party validation of annual national dataset

started before validation methodology had been standardized

Additional states are considering validation programs to improve the water audit data informing resource decisions.

Programmatic Recommendations

― Standardize water audit data collection and validation.

― Insist on transparent and neutral data validity grades.

― Approach regulating data validity grades and scores with caution.

― Focus on the role of data validity in water loss control and utility management.

― Establish a legislative mandate and funding.

What Does Validation Accomplish?

Validation aims to:

1. Identify and appropriately correct errors in data and application of methodology.

2. Evaluate and communicate uncertainty in water audit data inputs.

How Much Effort?

Level 1: data validity and methodology interview

Level 2: desktop analysis and raw data work

Level 3: field data collection

Validation doesn’t necessarily fix all errors –

auditing and validation are retrospective.

Level 1 Validation

Goals: Confirm interpretation of methodologyIdentify evident errorsAssign appropriate data validity grades

Outcomes: Appropriate data validity gradesRecommendations for further validation

Limitations: Does not correct errors in raw dataDoes not study instrument performance

Level 1 Validation

The Validator:

― Knowledgeable – understands what can go wrong

― Objective – keeps the goal in mind

― Diplomatic – avoids blaming

― Systematic – doesn’t miss anything

Not the auditor!

Level 1 Validation

The Method:

1. Collect the audit and required supporting documents.

2. Examine initial performance indicators.

3. Validate audit inputs (volumes, values, validity grades).

4. Re-examine final performance indicators.

5. Document results and propose next steps.

Level 1 Validation: Collect

The Required Materials:

― A water audit!

― Summary tablesWater supplied – by month and meter

Authorized consumption – by month and account class

― Supply meter test and calibration results (if applicable)

― Whatever else the audit compiler can pass along!

Level 1 Validation: Examine

Financial Indicators Check

NRW volume as % of Water Supplied > 0%

NRW value as % of operating cost < 100%

Operational Efficiency Indicators Check

Apparent Losses per Service Connection per Day > 0

Real Losses per Service Connection per Day > 0

Real Losses per Service Connection per Day per PSI > 0

Infrastructure Leakage Index > 1.0

Anything funky in the performance indicators, audit, or supporting documentation?

Level 1 Validation: Validate

For each input:

― How did the auditor arrive at the input? Is this corroborated by supporting documentation?

― How did the auditor interpret methodology?

― Which data validity grade describes operational practices?

― Does anything need to be changed to make the audit more accurate?

Level 1 Validation: Re-Examine

After validation, did the performance indicators change?

Financial Indicators Check

NRW volume as % of Water Supplied > 0%

NRW value as % of operating cost < 100%

Operational Efficiency Indicators Check

Apparent Losses per Service Connection per Day > 0

Real Losses per Service Connection per Day > 0

Real Losses per Service Connection per Day per PSI > 0

Infrastructure Leakage Index > 1.0

Level 1 Validation: Document

― People – auditor and validator

― Initial performance indicators― Validated performance indicators

― Recommended changes (audit inputs, data validity grades)

― Remaining questions― Recommendations for further validation― Overall impression

Success Story: CA Water Loss TAP

2015 CA UWMP submissions:

46% pass rate

What Happens After Level 1 Validation?

Audits still aren’t perfect! Inaccuracy can persist.

― Level 2 validation – raw data, data transfer

― Level 3 validation – instrument tests, leakage investigation

― Subsequent audits

― Water loss intervention

Thanks!

Comments? Questions? Brilliant ideas?

Lucy Andrews - lucy.andrews@wsoglobal.com

L. Andrews, K. Gasner, R. Sturm, G. Kunkel, W. Jernigan, and S. Cavanaugh. 2016. Level 1 Water Audit Validation: Guidance Manual (Project 4639A). Denver, CO: Water Research Foundation.

L. Andrews, K. Gasner, R. Sturm, G. Kunkel, W. Jernigan, and S. Cavanaugh. 2017. Utility Water Audit Validation: Principles and Programs (Project 4639B). Denver, CO: Water Research Foundation.