Practical, Affordable Practical, Affordable Performance Performance Measurement:Measurement:Why and How?Why and How?

Shelley H. Metzenbaum

smetzenbaum@verizon.net www.complianceconsortium.orgwww.facilityreporting.org

Northeast Environmental Summit September 2005

OverviewOverview• Why Measure?Why Measure?

• How to Measure and Use the How to Measure and Use the Measures?Measures?

• ExamplesExamples

• Issues to ConsiderIssues to Consider

• Workshop TomorrowWorkshop Tomorrow

Why Measure?Why Measure?

• Increase Increase AccountabilityAccountability

• LearnLearn to to Achieve Better Achieve Better OutcomesOutcomes

• Strengthen Strengthen DemocracyDemocracy

Measurement for Measurement for AccountabilityAccountability

(Program Justification?)(Program Justification?)

• EthicalEthical: : Self-Dealing/Fraud (tSelf-Dealing/Fraud (track rack $ flows to whom for what – accounting $ flows to whom for what – accounting & fiscal audits)& fiscal audits)

• FiscalFiscal: Unallowed : Unallowed spending/Waste/Efficiency spending/Waste/Efficiency ($ ($ spent/$ allocated & goods or services spent/$ allocated & goods or services bought per input cost)bought per input cost)

• PerformancePerformance: Improve social : Improve social outcomes/reduce risk/more outcomes/reduce risk/more cost-effective cost-effective (better outcomes; (better outcomes; better outcomes at lowest cost)better outcomes at lowest cost)

Measurement to Measurement to LearnLearn so so Achieve Better Achieve Better

OutcomesOutcomes• DETECTIONDETECTION: Detect : Detect problemsproblems needing needing

attention; attention; progressprogress deserving replication- deserving replication- what works and what doesn’t?what works and what doesn’t? – Look for anomalies, unexpected direction, Look for anomalies, unexpected direction,

speed ups, slow downs, etc.speed ups, slow downs, etc.– Use measurement to trigger focused follow Use measurement to trigger focused follow

up up

• DIAGNOSTICSDIAGNOSTICS: : Why? What is the reason Why? What is the reason for variation? Is it a problem or lesson?for variation? Is it a problem or lesson?

• ASSESSMENTASSESSMENT: : Did it seem to work? Did it seem to work? Replicate, expand, or stop and fix?Replicate, expand, or stop and fix?

Goals/Measures to Goals/Measures to Strengthen DemocracyStrengthen Democracy

• Cannot Do Everything: Goals/ Cannot Do Everything: Goals/ Targets Communicate: Targets Communicate: – What You Will DoWhat You Will Do– What You Won’t DoWhat You Won’t Do

• Are You Doing Right Thing? Are You Doing Right Thing? DemocraciesDemocracies can respond when can respond when unhappy with agency choicesunhappy with agency choices

• How Well Are You Doing It? How Well Are You Doing It? MeasurementMeasurement

Start with Start with Learning Learning Emphasis of Emphasis of MeasurementMeasurement

• Detection, Diagnostics, Detection, Diagnostics, AssessmentAssessment

• Do this well … Accountability & Do this well … Accountability & Democracy will Rise, as wellDemocracy will Rise, as well

• Practical, Affordable ExamplesPractical, Affordable Examples

DETECTION DETECTION : Look for Pattern : Look for Pattern Changes to Sense ProblemsChanges to Sense Problems

Why the Downturn ?

Source: http://www.chesapeakebay.net/status/status_print.cfm?sid=101&print=yes

DETECTION: Volunteer Data -- Look for Unexplained VariationsDescription Town Riv

er Mile

2/ 15/ 00 3/ 21/ 00

5/ 16/ 00

Central Street Bridge

Milford 3.5 1100 380 120

Discharge Pipe @ Central Street

Milford 3.5 1470 2100 3400

2d Discharge Pipe @ Central Street

Milford 3.5 <10 260

Mellen St. Bridge

Bellingham

5.9 490 140

Rt. 126, N. Main

Bellingham

9.0 340 10 120

DETECTION DETECTION : Look for Pattern : Look for Pattern Changes to Find ProgressChanges to Find Progress

Recognize Progress !

Source: http://www.chesapeakebay.net/status/status_print.cfm?sid=101&print=yes

DETECT: Before & After DETECT: Before & After PhotosPhotos

DETECTION/DIAGNOSTICS: DETECTION/DIAGNOSTICS: Inspec. DataInspec. Data“Scout & “Scout & Survey”Survey”% Compliance by Violation Type

0102030405060708090

100

Train

ing

Insp

ectio

ns

Label

s

Aisle

Spac

e

Close

d

Postin

gs

Charac

teri.

..

NH FQG

NH SQG

NH, CO

Detect low comp. rateDiagnose: Is it a problem?

Why is it happening?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

8 10a 13 18 22 35 36i 38 50

(10) Leak Checks

(18-19) Controls

(36i) Emergency Procedures

Diagnostics: MA ERP Self-Audit Data 1997 & 1998

DETECT, DIAGNOSE, ASSESS: Self-Reported Practices Trends (97/98)

MA ERP

Lower than Other Areas:

Problem?

Changes over Time:

Progress?

1996 DRY 1996 DRY WEATHER WEATHER BACTERIABACTERIA

Swim StandardSwim Standard

Some FailuresSome Failures

FailedFailed

1996/97 WET WEATHER 1996/97 WET WEATHER BACTERIABACTERIA

DETECTION and DIAGNOSTICS: Mapping/Segmentation HighlightsProblem Areas and Progress

DIAGNOSTICS/ASSESSMENT: Slice & Dice by Categories to Identify Cause of Problems, Progress

Detection: # Detection: # ComplaintsComplaints(Cleveland Air (Cleveland Air Pollution)Pollution)• General Environmental Mgt. – 64General Environmental Mgt. – 64

• ISG – 52ISG – 52

• Ford Motor Company – 29Ford Motor Company – 29

• Kirkwood Industries – 18Kirkwood Industries – 18

• Landfill – 14Landfill – 14

• Wabash Alloys -12Wabash Alloys -12

DIAGNOSTICS: Complaint DIAGNOSTICS: Complaint Relevance?Relevance?

Source: PA DEP

ASSESSMENT: Pictures ASSESSMENT: Pictures ConsolidatedConsolidated Tailings Pile (CO)Tailings Pile (CO)

Prior to 1989 After Reclamation

1999

Lots of Affordable Ways to Lots of Affordable Ways to DETECT, DIAGNOSE, ASSESSDETECT, DIAGNOSE, ASSESS• Air and Water Quality DataAir and Water Quality Data

– Volunteer & govt collected data usefulVolunteer & govt collected data useful– Water, Energy Supply – Boston Water Water, Energy Supply – Boston Water

and Sewer Commission use data to cut and Sewer Commission use data to cut water use and costswater use and costs

• Cameras – Texas detects problem air Cameras – Texas detects problem air plumes; Rutgers extension helps farmers plumes; Rutgers extension helps farmers cut pesticides and costscut pesticides and costs

• Incident and Inspection Data: Incident and Inspection Data: – Repeat Violators - Accidents Repeat Violators - Accidents – Nature of Non-Compliance – ComplaintsNature of Non-Compliance – Complaints– FiresFires

DETECTION, DIAGNOSTICS, DETECTION, DIAGNOSTICS, ASSESSMENT: Interim ASSESSMENT: Interim Outcome Indicators IdeasOutcome Indicators Ideas

Attention

Comprehension

Beliefs/Attitudes

Motivation

Information

Behavior

Measure Awareness (OR)

Measure Understandin

gMeasure Attitudes

Measure Motivation

Measure Behavioral Changes (ERP)

Newspaper Content Newspaper Content Analysis, Surveys, Analysis, Surveys, Observational Observational Measurement Measurement

0

10

20

30

40

50

60

70

1962-1970 1971-1982 1983-1988 1989-1993

Technology

Regulation

Enforcement

Management

Chemical Industry Trade Press Content Analysis (Hoffman 1999)

STRENGTHEN DEMOCRACY: Specific Goals Clarify What Govt Will Do and What It Won’t

• Where: Lower Charles River

• When: 10 years

• How much: Swimmable

• Who: All Sources

• Invite public attention

• Public can respond

Strengthen Democracy: Make Strengthen Democracy: Make Goals, Progress, Problems Goals, Progress, Problems ClearClearGoal Not Met,

Past and Future

Strategy Described

Goal Met; Commitment to New

Goal Described

Issues to ConsiderIssues to Consider• Data AccuracyData Accuracy: Match to need. Can be lower : Match to need. Can be lower

for Detection; higher for Rules and for Detection; higher for Rules and Enforcement. For assessment, consider Enforcement. For assessment, consider program $program $

• Project and Measurement Project and Measurement ScaleScale: indiv. facility : indiv. facility level work OK only if have context/control. level work OK only if have context/control. Count costs and cost-effectiveness when Count costs and cost-effectiveness when project level. project level. Start with env. problemStart with env. problem..

• Comp asstc and P2 activity counts not useful Comp asstc and P2 activity counts not useful indicator; may have occasional diagnostic indicator; may have occasional diagnostic valuevalue

Tomorrow’s Tomorrow’s Workshop: Workshop:

Brainstorming Your Specific Measurement Challenges Together

Brainstorming Your Specific Measurement Challenges Together

1:30 - 3:30

Shelley H. Metzenbaumsmetzenbaum@verizon.netwww.complianceconsortium.orgwww.facilityreporting.org