Performance Performance MeasuresMeasuresfor Clean Airfor Clean Air
9th Annual Environmental Performance Summit9th Annual Environmental Performance SummitOctober 3-5, 2007October 3-5, 2007Mike Hadrick, US EPAMike Hadrick, US EPA
[email protected]@epa.gov
22
Outline of PresentationOutline of Presentation
A Matter of PerspectiveA Matter of Perspective Measure Types and UsesMeasure Types and Uses Analytical FrameworksAnalytical Frameworks Measures for Ground-level OzoneMeasures for Ground-level Ozone ChallengesChallenges Improvement EffortsImprovement Efforts ConclusionsConclusions Recommended ReadingRecommended Reading
33
A Matter of PerspectiveA Matter of Perspective
77
Measure Types and UsesMeasure Types and Uses
88
Measures for Lower ManagementMeasures for Lower Management
OutputOutput measures track activities and measures track activities and monitor progress and accomplishments in monitor progress and accomplishments in the short run.the short run.
Think of driving two days to a vacation spot Think of driving two days to a vacation spot and hitting milestones on the way...and hitting milestones on the way...
99
Short Run Measures for VacationShort Run Measures for Vacation
Are we on schedule or running late? Are we Are we on schedule or running late? Are we where we planned to be, or did we hit some where we planned to be, or did we hit some detours?detours?
What about our final destination? Will we still What about our final destination? Will we still have the time there to do the things we planned?have the time there to do the things we planned?
Based on the above, do we need to adjust Based on the above, do we need to adjust anything to have a nice vacation once we get anything to have a nice vacation once we get there?there?
1010
Measures for Upper ManagementMeasures for Upper Management
OutcomeOutcome measures track bigger picture or long- measures track bigger picture or long-term trends – such as goal achievement. Back to term trends – such as goal achievement. Back to the vacation...you arrived a couple days ago...the vacation...you arrived a couple days ago...
How’s it going? Are we having fun yet? Was this How’s it going? Are we having fun yet? Was this a good idea? Or is it more like “well, we’re not a good idea? Or is it more like “well, we’re not doing this again!”?doing this again!”?
There’s not much you can do about it now other There’s not much you can do about it now other then lessons learned for next year’s vacation.then lessons learned for next year’s vacation.
1111
Simple Logic Model for Any Type of ProgramSimple Logic Model for Any Type of Program
InputsInputs OutputsOutputs IntermediateIntermediateOutcomesOutcomes
EndEndOutcomesOutcomes
Hierarchy of Indicators – Specific to Environmental ProgramsHierarchy of Indicators – Specific to Environmental Programs
Actions by Actions by RegulatorsRegulators
Responses Responses of Regulated of Regulated Community Community or Societyor Society
Changes in Changes in Pressure or Pressure or
Stressor Stressor QuantitiesQuantities
Changes in Changes in Ambient Ambient
ConditionsConditions
Changes in Changes in Exposure, Exposure,
Body Body Burden, or Burden, or
UptakeUptake
Changes in Changes in Human Human
Health or Health or Ecological Ecological ConditionCondition
Analytical FrameworksAnalytical Frameworks
1212
Measures forMeasures forGround-level OzoneGround-level Ozone
1313
Program Purpose andProgram Purpose andRole of Air OfficeRole of Air Office
Protect human health and the environment Protect human health and the environment from the potential effects of air pollution from the potential effects of air pollution associated with ozone.associated with ozone.
Set health-based standards that protect with Set health-based standards that protect with human health and the environment and human health and the environment and assists states in implementing the standards. assists states in implementing the standards.
1414
Simple Logic Model for OzoneSimple Logic Model for OzoneEPA InputsEPA Inputs EPA OutputsEPA Outputs IntermediateIntermediate
OutcomesOutcomesEndEnd
OutcomesOutcomes
$$$$$$
FTEFTE
Establish air Establish air quality standardsquality standards
Develop technical Develop technical info, methods, & info, methods, & guidance to guidance to implementimplement
Conduct Conduct oversightoversight
Implement Implement federally if States federally if States do notdo not
Reduced Reduced emissions of air emissions of air pollutantspollutants
Lower ambient Lower ambient concentrationsconcentrations
Fewer days with Fewer days with unhealthy airunhealthy air
Reduced Reduced atmospheric atmospheric depositiondeposition
Reduced acidity of Reduced acidity of lakes and streamslakes and streams
Reduction in Reduction in prevalence and prevalence and incidence of air incidence of air pollution-related pollution-related health effectshealth effects
Improved Improved ecosystem ecosystem conditioncondition
1515
Hierarchy of Indicators for OzoneHierarchy of Indicators for Ozone
Actions by Actions by RegulatorsRegulators
ResponseResponseof Regulated of Regulated Community Community or Societyor Society
Changes inChanges inPressure or Pressure or
StressorStressorQuantitiesQuantities
Changes inChanges inAmbient Ambient
ConditionsConditions
Changes inChanges inExposure, Exposure,
BodyBodyBurden, or Burden, or
UptakeUptake
Changes inChanges inHuman Human
Health or Health or Ecological Ecological ConditionCondition
Establish air Establish air quality quality standardsstandards
Develop Develop technical info, technical info, methods, methods, guidance to guidance to implementimplement
Conduct Conduct oversightoversight
Implement Implement federally if federally if States do notStates do not
Implement Implement programsprograms
ComplianceCompliance
Take Take voluntary voluntary actionsactions
Implement Implement alternativesalternatives
Air pollutant Air pollutant or precursor or precursor emissions emissions from various from various source sector source sector categoriescategories
Air pollutantAir pollutantconcentrationsconcentrations
Deposition of Deposition of air pollutantsair pollutants
Lake and Lake and stream aciditystream acidity
Prevalence Prevalence and incidence and incidence of air of air pollution-pollution-related health related health effectseffects
Ecosystem Ecosystem conditioncondition
1616
Short Run MeasuresShort Run Measures
Timeliness of rulemakings, etc. (Deputy)Timeliness of rulemakings, etc. (Deputy)
Status of State Implementation Plan development by Status of State Implementation Plan development by states or review/processing by Regional Office (program states or review/processing by Regional Office (program office)office)
% of permits reviewed or issued within target timeframe % of permits reviewed or issued within target timeframe (PART)(PART)
NONOXX reductions from electric utilities (program office) reductions from electric utilities (program office)
1717
1818
Longer-term IndicatorsLonger-term Indicators
Mobile source emissions of ozone precursors (PART)Mobile source emissions of ozone precursors (PART)
Emissions of ozone precursors from all source Emissions of ozone precursors from all source categories (ROE)categories (ROE)
% reduction in population-weighted ambient % reduction in population-weighted ambient concentration of ozone in monitored counties (PART)concentration of ozone in monitored counties (PART)
Ambient ozone concentrations (ROE)Ambient ozone concentrations (ROE)
1919
2020
2121
2222
2323
2424
2525
End Outcome IndicatorsEnd Outcome Indicators
Asthma prevalence and mortality (ROE)Asthma prevalence and mortality (ROE)
Childhood asthma prevalence and mortality Childhood asthma prevalence and mortality (ROE)(ROE)
Chronic obstructive pulmonary disease (COPD) Chronic obstructive pulmonary disease (COPD) mortality (ROE)mortality (ROE)
Cardiovascular disease (CVD) mortality (ROE)Cardiovascular disease (CVD) mortality (ROE)
2626
ChallengesChallenges
Requirements of statutes, regulations, and court Requirements of statutes, regulations, and court ordersorders
Data not available in time for official reporting Data not available in time for official reporting cyclescycles
Time between activity or expenditure and Time between activity or expenditure and outcomeoutcome
2727
ChallengesChallenges
Who’s performance does measure actually Who’s performance does measure actually capture?capture?
Linking activities to outcomes (causality)Linking activities to outcomes (causality)
Measures overloadMeasures overload
2828
Improvement EffortsImprovement Efforts
2004 NRC Recommendations: 2004 NRC Recommendations: Air Quality Air Quality Management in the United StatesManagement in the United States
EPA Air and R&D offices working together on EPA Air and R&D offices working together on Air Air Accountability Framework Accountability Framework per NRCper NRC
EPA EPA Indicators InitiativeIndicators Initiative
Air office’s Air office’s Measures Improvement EffortMeasures Improvement Effort
2929
ConclusionConclusion
Be clear on the program’s logicBe clear on the program’s logic
Develop measures for each phase of programDevelop measures for each phase of program
Select measures for program evaluation or Select measures for program evaluation or reporting based on intended audience and usereporting based on intended audience and use
3030
Resolution & ZoomResolution & Zoom
- - - - - - - - - - - - - - - - - - - - - -
3131
Recommended ReadingRecommended Reading
Performance-Based Management HandbookPerformance-Based Management HandbookDOE – Oak Ridge Institute for Science and EducationDOE – Oak Ridge Institute for Science and Education
www.orau.gov/pbm/pbmhandbook/pbmhandbook.htmlwww.orau.gov/pbm/pbmhandbook/pbmhandbook.html
Performance Measurement Challenges and Performance Measurement Challenges and StrategiesStrategiesU.S. Office of Management and BudgetU.S. Office of Management and Budget
www.whitehouse.gov/omb/part/challenges_strategies.pdfwww.whitehouse.gov/omb/part/challenges_strategies.pdf
3232
Additional Air InfoAdditional Air Info
www.epa.gov/oar/airtrendswww.epa.gov/oar/airtrends
3333
EPA Report on the EnvironmentEPA Report on the Environment
Describes national environmental conditions and Describes national environmental conditions and trends using existing data and indicatorstrends using existing data and indicators
IDs data gaps and research needsIDs data gaps and research needs
Discusses challenges in filling those gapsDiscusses challenges in filling those gaps
www.epa.gov/roewww.epa.gov/roe
3434