41
Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October 2001

Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Embed Size (px)

Citation preview

Page 1: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Developing Public Health Performance Measures: Benefits and Challenges

Mare Schumacher

Maricopa County Department of Public Health

APHA Meetings October 2001

Page 2: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

About Us

Maricopa County Department of Public Health in Phoenix, AZ

440 employees

47 distinct programs (from childhood immunizations to HIV education)

85% grant funded

Page 3: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Developing Measures

Page 4: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

History

Pre-1998: Collected some metrics (mostly grants)

1998-99: Dept. leaders request collection of measures for all programs

2000: Countywide initiative

2000-01: Complete “family” of measures for all 47 programs

Page 5: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Developing Measures

One employee dedicated to process coordination (about ½ time)

Trained managers/supervisors on:Types of pms (e.g. output, outcome)Proper collection methodsProper documentation

Page 6: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Developing Measures

Supervisors sat down with staff to determine goals

When needed, coordinator worked individually with programs

Biggest challenge: the psychology…

Page 7: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Stages of PM Process*

* Apologies to Elisabeth Kubler-Ross

Denial: “This is ‘flavor of the month’. If I ignore it, it will go away shortly.”

Anger: “I haven’t got time for this!”

Panic/Freeze: PMA (Performance Measure Anxiety)

Depression: “I can’t wait to retire.”

Acceptance: “We have to do this.”

Page 8: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Two Sales OptionsCheerleader Model:

Telling employees how super-fantastic these performance measures will be!!!

Empathy (I Feel Your Pain) Model:

Acknowledging hard work that will be required by employees. Helping them through difficulties.

Page 9: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Collection and Maintenance

Maintain PMs in central document/ electronic location w/gatekeeper

Important to keep narrative explaining significant shifts in numbers

Updated semi-annually, now quarterly

Page 10: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Example: Immunizations

Result: Coverage level - % of clients receiving entire series

Output: # of immunizations, # of clients, # of clients 0-2 years old

Efficiency: Cost perimmunization

Page 11: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Example: Nutrition Education

Output: # of students

Efficiency: Cost per student

Result: Increase in knowledge, attitudes, behavioral intent from pre- to post-test

Page 12: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Using and Abusing Measures

Page 13: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Philosophy

Set Goals

Run ProgramFunding Decisions (“Accountability”)

Measure and Analyze Program Results

Page 14: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Benefits

Long term resource planningStaffBudget

A goal to “shoot for”, rally aroundSticking with priorities – are we using our energies for our top priority?

Page 15: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Do we really measure performance?

Page 16: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Performance Measurement Experiment

County XMaricopa County

Goal for Fiscal Year 01:

Actual Performance:

Goal for Fiscal Year 01:

25 25

Actual Performance:

Page 17: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Performance Measurement Experiment

County XMaricopa County

Goal for Fiscal Year 01:

Actual Performance:

Goal for Fiscal Year 01:

25 25

Actual Performance:

At this point in the presentation, two people – representing each of the counties – must fill a cup with marbles. Maricopa County uses a spoon and County X has to use chopsticks. The result is that County X scores lower on its performance measure. Without knowing that County X had chopsticks, we might conclude that it exhibits “poor performance.”

Page 18: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

The “Chopstick Effect”

Funding

Staff shortages

Change in grant requirements

Marlboro has big ad/promotion

And so on…

Page 19: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Issues

May not measure program effort or potential effectiveness

“Widget Factory” model often not applicable in PH

Cost of collecting data some-times not worth it

Goal setting as art

Page 20: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Alternatives

Process evaluation“Up front” research (usually from Academia)Statistical modelingStandard research practices:Test/Control, etc.National Public Health Performance Standards (self-assessment)

Page 21: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

ConclusionsYou can develop measures for your organization…But don’t forget the psychological component.Use performance measures wisely – better for planning than evaluating.Let’s find other ways to accurately determine success (for true “accountability”).

Page 22: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

InformationMare SchumacherMaricopa County Department of PH1845 E. Roosevelt, Phoenix, AZ [email protected]: “Taking Stock of Performance Measurement: Information Resources for Public Managers”, Government Finance Review, April 2001Resource: “Performance Measures Basics” (presentation for program staff). Send e-mail to Mare Schumacher for copy.

Page 23: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

The following is a presentation I used to educate program staff on performance measures at our Department. (It was not presented at the APHA.) Feel free to use it or alter it for your own purposes.I ask nothing in return except that you think of me for a moment every time you use it…

Mare Schumacher

Department of Public Health

Maricopa County, Arizona (Phoenix)

Page 24: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Performance Measures

Presentation to Public Health

Strategic Planning Team

March 1999

Page 25: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

The Past

The night before due, make up #s to put in as performance measuresNo one looks at themNo one checks to see if you measuredNo one thinks about it for 11 months and 27 daysNext year, repeat process

Page 26: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

The Future

County Administration will be looking closely at performance measures

Depts./Programs will be judged on performance measures

Budget may be tied to performance measures

Page 27: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Making the Best of It

The night before due, make up #s

Live with consequences

Come up with performance measures that are meaningful to County and you

We’ve got two choices:

Page 28: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Suggested Process

May be other options...

Page 29: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Meet with Staff

In a meeting with all or many staffHelps set realistic goalsEmployees know what is

expected of them

“What gets measured gets done.”-- Rose Howe, December 1998

Page 30: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Decide Program Mission

Why do we exist?

What contribution to we make to the people of MC?

NOT What do we do?

Page 31: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Example: HIV Planning

What we do:

Have meetings

Contact coalition members

Prepare reports

Get trained in how to build teams

Why we exist

To create a comprehensive HIV prevention plan that will reduce the spread of HIV

Page 32: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Just talk this

We’re not measuring what we do

We are measuring our contribution to the public health

Doesn’t mean what we do is not important, just not the thing we’re measuring

Page 33: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Create Output Measurement

Output measurement:

Activities

Things we do

Examples:Number of applications processedNumber of clientsNumber of immunizationsNumber of coalition meetings

Page 34: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Example: Immunizations*

Conducted 7500 immunizations in 97/98, etc.

Expect to administer 9500 in 99/00

Reasons: Population growing by 5%, will have increase in staff in 99/00

* All numbers made up -- demonstration purposes only

Page 35: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Create Efficiency Measurement

Efficiency measurement:

Cost per output

Resources used to produce the activity

Examples:

Cost per client

Time spent per client

Cost per client compared to certain standard(s)

Page 36: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Example: Immunizations

Clients per hour = 4 in 97/98

Expect 5 by 2000/01 due to new methods

Cost per client = $1.22

Expect cost = $1.45 by 00/01 due to increase cost of vaccine.

Page 37: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Create Impact (Outcome) Measurement

Impact (outcome) measurement:

Indicate the impact of service on the community

Should flow directly from mission

Examples:

Percentage of clients rehabilitated

Reduction in incidence of disease

Page 38: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Example: Immunizations

90% coverage of school-aged children by 00/01

57% coverage of clients at public health immunization sites

Possibly: 400 cases of measles avoided

Page 39: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

All Measures

What data will we use to measure it?

If none available now, how will we get it?

Page 40: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Goals (Future Years)

Make them specific

Make them realistic - how much control do we have over that measure?

What are the reasons for expecting the goal to be higher, lower, etc.

Document why you picked that goal? How did you calculate it?

Page 41: Developing Public Health Performance Measures: Benefits and Challenges Mare Schumacher Maricopa County Department of Public Health APHA Meetings October

Timing

Develop measures with staff

Draft meeting with John/Mare

Final draft meeting

Final measures due

End of March/April

End of April (sign-up sheet)

May 11 SP Meeting

June 1