Transitioning to Outcome-Focused Management

Findings from a Cross-Agency Study

prepared for: National Environmental Partnership Summit 2006

Measurement and Targeting Session, Wednesday May 10, 2006

prepared by:

Tracy Dyke RedmondSenior AssociateIndustrial Economics (IEc)tdr@indecon.com617-354-0074 x 193

Topics to Address

• Study objectives and methodology• Background• Definitions• Case study programs• Findings• Reflections on themes from panel

Study Objectives and Methodology

• EPA: Benchmarking study to find best Fed. Agency practices for measuring results of:– stewardship programs– innovation programs

• Data Source for Identifying Cases: OMB Program Assessment Rating Tool (PART) reviews

• Phase 1: paper review of PART assessments

• Phase 2: follow-up on three strong programs and comparison of EPA and other agency PART reviews

Background: What is the PART?

• Framework OMB uses to assess Fed agency programs• PART Questions look at:

– Program purpose and design, – Strategic planning, – Program management, and – Program results.

• PART supports Government Performance and Results Act (GPRA), which requires every federal agency to:– Develop strategic and annual plans, – Measure progress annual, and – Report annually.

Definitions: What are “stewardship” and “innovation”?

For the purpose of this review…• Stewardship programs:

• cultivate a sense of responsibility for the natural environment, or

• lead to actions that protect specific natural resources such air, water, and land.

• Innovative programs search for, test, develop, and/or promote adoption of practices that are more effective or cost effective in achieving agency objectives.

Three Case Studies Model Outcome-Focused Programs

• Coast Guard Marine Environmental Protection prevents and responds to marine environmental risks.

• Department of State Global Educational and Cultural Exchanges strengthens international relations through cross cultural exchange.

• Department of Transportation National Highway Traffic Safety Administration (NHTSA) saves lives, prevents injuries, and reduces traffic-related health care and other costs.

Overarching Findings on Stewardship and Innovation

Programs

• Well-rated programs set stewardship goals and measure progress in terms of:– Outcomes (Coast Guard and NHTSA)– Attitudes (NHTSA and State)

• No well-rated programs found that set or measure innovation goals, but …– Innovation readily occurs in programs that set

outcome-focused goals and use outcome-focused measurements (all three cases)

Finding 1: Performance Measurement Regularly Used for

Program Improvement

Performance measurement not just a reporting requirement, but integrated into program management.•Local managers use routinely collected data to prioritize risks and target problems.

– NHTSA gathers pre-crash, crash, and post-crash data

•HQ managers use data to confirm assumptions and test program impacts.

– NHTSA re-evaluates rule effectiveness after 7 yrs.

Finding 2: Program Goals Should be Clear, Outcome-Focused, Realistic,

and Ambitious • Successful programs clearly define their goals and have at least one

outcome measure tied to each goal.– Coast Guard – oil and chemical spills, nuisance species– NHTSA – traffic fatalities, seat belt use– State – survey of practices, such as following US news

• Programs must balance having realistic and ambitious goals. – NHTSA struggled with overly ambitious goals

• Shift from process to outcomes is hard, but worth it.

Finding 3: Outcome Goals Foster Innovation

•Establishing clear outcome goals with measures shifts staff perspectives about the best ways to do their jobs.

– State – focus on participants’ entire experience– State – creation of alumni group

•When staff were allowed flexibility to make program changes reflecting their new perspectives, it produced positive performance results.

– Coast Guard – shift focus from compliance to oil spills

Finding 4: Federal Programs Build Partnerships by Aiding and Informing

States/Locals• Federal programs often must rely on state/local authorities to

gather data and implement programs.• Feds can add value by:

– Analyzing cross-state data and sharing it with states and local authorities (e.g., NHTSA’s motorcycle helmet analysis)

– Testing programs to find which programs are most effective, and packaging results to promote those that work best (e.g., NHTSA’s Click-it or Ticket campaign)

Finding 5: Program Evaluation Builds Stronger Programs

• In-house evaluation builds stronger programs. – State’s evaluation office is in high demand

• Successful programs set aside funds for evaluation. – State program offices voluntarily contribute to

evaluation office’s budget– Evaluation is new and increasingly valued

• Program evaluation can build on routine/daily performance measures.– NHTSA uses routine measures for look-back

evaluations

Finding 6: Innovative Programs Continuously Scan for Effective/ Cost-

effective Approaches• Innovative programs look for evidence of effective practices from

states, field offices, other countries, the literature, and their own experiments.– NHTSA gathers data about programs believed to reduce crashes and injuries

• In looking for best practices, innovative programs focus on outcomes, and measure progress on this basis.– Coast Guard charts identify biggest contributors to risk, tactics change based

on the data

Finding 7: Stewardship Programs Measure Attitude Changes through

Surveys • Longitudinal studies track broad attitude

changes over time among key target audiences – State conducts regular surveys of program

participants • Comparison groups are important to be

able to demonstrate causality. – State compares attitude changes of participants

to those who did not participate• Broad scale surveys for domestic programs

may face ICR restrictions, but problems not found in cases.

Reflections on Themes from Panel

• How to use existing data to measure program performance– EPA can add value by compiling and analyzing data,

highlighting trends, and identifying effective approaches• Approaches for gathering new data to measure

program performance– Surveys can be helpful, especially when conducted

regularly to establish baseline and trend data• Attribution - can programs "take credit" for

observed changes in performance?– Control groups and before/after comparisons are useful

in teasing out a program’s impacts