Embed Size (px)
Citation preview
11
Performance Measurement 101PART 1
Oregon Public Performance Measure Association (OPPMA) Annual Meeting
8:45 – 11:45 Session
with Rita Conrad, Ken Smith and Laura Wipper
Willamette University, Salem, OR
22
Today’s Session • Part 1: The basics Part 1: The basics (~60 min)
– Intro & Brief History– Criteria– Logic models
• Part 2: Small groups Part 2: Small groups (~75 min)– Practice– Report-out
• Part 3: Making it happen Part 3: Making it happen (~60 min)– Three approaches– Dialogue
3
Late 1800’s-1930’s
• New York Municipal Research Board
• Audit of Portland, Oregon
• International City Management Association (ICMA)– Herbert Simon, Nobel Prize winner
4
1970’s Productivity
• Optimistic Views– Harry Hatry (1972; 1978)
• Negative or Cautionary Views– Quinn (1978);– Burkhead and Hennigan (1978)
5
1980’s – Why Accountants?
• Governmental Accounting Standards Board (GASB)
• Government Finance Officers Association (GFOA)
• Association of Government Accountants (AGA)– Oregon is the best state at performance reporting – by a
long shot (Smith, Cheng, Smith, Schiffel 2006 – available online)
6
Original Oregon Model @ ODOT• Roots in work of Glenn Felix
– Productivity matrix• Baseline – or average – levels of performance
• Historical best or optimum goals
• Relative weights, performance levels and performance index
• Efficiency and Effectiveness– Cost vs. Quality
– Gainshare pilot at ODOT proves value is in the measures and the conversation
7
Striving for Balance
•Cost per product, service or result•Labor per product, service or result•Direct hours•Indirect hours•Administrative hours
•Outcome/Goal•What’s Important Along the Way•Customer Satisfaction•Work Life Quality
8
Work Life Quality
• At ODOT• Regularly gauged the capacity of the organization
– Sick leave
– Safety
– Turnover
– Employee Survey
– Equal Opportunity Employment
– Training
9
Development of Measures
• Series of workshops– Involved almost all staff– Aligned activities to purpose, outcomes, goals– Measures were customized by staff
• Alignment through Key Result Areas
• Some were “roll-ups”
10
Statewide Rollout• Statewide measurement initiative began in
1991 with 16 agencies – Phase 1
• More phases followed – 100+ agencies
• Agency-level measurement facilitated by mentors from previous phases
• Expanded upon ODOT model– Efficiency and effectiveness– Tied goal-orientation to Oregon Benchmarks
11
Some 90’s Success Stories• DMV
– Positive experience with a negative index• Ways and Means liked the candid conversation
• Adult and Family Services– Moved focus from application processing to family self-
sufficiency• Lead the nation in welfare reform
• Vocational Rehabilitation– Decided federal goal of two months back on the job not
high enough to justify cost• Set goal at 18 months
• Department of Corrections– Sharper focus on direct recipient of program efforts
• 180 degree swing on recidivism
12
Original Oregon ModelAdopted across the US
• Implemented nationwide by US Army Corps of Engineers– One of first two agencies to act following
federal Government Performance Results Act of 1993
• At one point, 70 Federal agencies were implementing the “Oregon” model following success of Corps
13
Elements of Oregon’s Key Performance Measure System
Tiered, linked system for external reporting Common language Logic models Criteria Budget
14
SB555 gives Progress Board first time
evaluation function
HB 3358 gives performance
measurement role to Progress Board
Linking state performance to benchmarks
93 94 95 96 97 98 99 00 01 02 03 04 05 06 07
Measuring Up
by Jonathan Walters
98
Oregon laws required agency
performance measures 93 09
08
Progress Board prepares KPM-
Benchmark Crosswalks for W&M
subcommittees
15 Oregon Benchmarks and Performance Measures
Activity #1
Getting to Know You
1717
Why measure performance? To get results.
It’s at the core of results-based management• Fosters internal learning and improvement
Is the ship running well?
• Provides greater external accountabilityIs the ship on course?
• Informs policy-making Should the ships change course?
1818
CRITERIA What makes a good performance measure?
1. Common language2. Alignment (line-of-sight)3. Accurate and reliable data4. A few key measures5. Time-framed targets6. Someone in charge7. Comparisons8. Customer satisfaction
19
• OUTCOME, high-level = Societal result
• OUTCOME, intermediate = Entity result
• OUTPUT = Product or service
• INPUT = Time or money
• EFFICIENCY =Input per output or Input per outcome
Criterion #1. Common language
19
20
3. High-Level Outcomes (HLO) – Is the world you are attempting to affect changing?
Oregon Benchmark #65, Juvenile Recidivism – % of juveniles referred w/in 12 months …
Practical application
2. Intermediate Outcomes – Is the strategy producing the desired result? % of treated youth with reduced risk factors
1. Outputs – Is the strategy getting done? # of youth completing treatment
20
21
Criterion #2. Alignment - line-of-sight to goals and high-level outcomes
Goal
Key Performance Measures
IntermediateOutcomes
Outputs
HLO - Mission or Benchmark
Impact
Strategies to achieve the goal
21
2222
Criterion #3. Accurate and reliable data
• Trustworthy data is essential.Example: verifiable records trump estimates
• Per measure - at least one data point, preferably several
• Data should match the measure.
23
Criterion #4. Few key measures – different measures for different purposes
More agency influ
ence
More public in
terest
Goal: Prevent Juvenile Crime
% of grantees trained INCREASES.
Output
“So That”
% of high risk youth completing program INCREASES.
Intermediate Outcome
“So That”
% of served youth with mitigated risk factorsINCREASES.
Intermediate Outcome
“So That”
Juvenile recidivism DECREASES.
High-Level Outcome
How high up do you want
to go?23
24
Criterion #5. Time-framed targets
• TARGET = Desired level at a given point in time
– Ambitious but realistic
– Set targets using:
• trend data
• comparisons
• expert opinion
2007 2007 TargetTarget: :
1515
2003 2003 Actual: Actual:
2020Example: hours of travel Example: hours of travel delay per capita per yeardelay per capita per year
24
25
Criterion #6. Someone in charge
• Management, staff and stakeholders need to know where the buck stops
• Critical for improving performance
25
26
Criterion #7. Measures of customer satisfaction
• This is a required Key Performance Measure (KPM) for Oregon state agencies.
• Rationale: Those who pay for their government should be satisfied with their government.
26
27
Criterion #8. Comparisons
• How does actual performance compare to industry standards?
• Rationale: to better understand whether action is needed.
• Difficult but not impossible!
• Agency suggestion: “peer” agencies develop common measures and comparator
“We resolved
complaints within three
weeks on average.”
Is this good or bad?
27
2828
LOGIC MODEL - a tool for thinking it through
Getting started1. IMPORTANT: Engage those whose
performance will be measured.
2. Have a clear statement of goals.
3. “Look up” to mission & high-level outcomes (benchmarks)
4. “Look down” to what you are doing and how you are measuring performance
AgencyGoals
Performance Measures
High-level outcomes
2929
Logic model example #1 – Citizen involvement in land use planning
Impact
STRATEGYJointly sponsor, with cities,
regional educational events for private citizens every quarter.
INTERMEDIATE OUTCOMES % participating citizens with
improved understandingCustomer satisfaction ratings
GOAL: Citizen involvement (C.I.) in
land use planning
HLO: % of cities with neighborhood
organizations.
Agency Performance
Measures
OUTPUTS# citizens trained.
# C.I. guidelines distributed.
3030
Logic model example #2 – Juvenile crime
Impact
STRATEGY Award grants to local contractors to conduct “best practice” juvenile crime prevention programs (JCP).
INTERMEDIATE OUTCOMES % of juveniles in JCP programs with significantly mitigated risk
factors.
GOALReduce juvenile
crime.
HLOJuvenile Arrests
(OBM#61)
Agency Performance
Measures
OUTPUTS# grants awarded by county
# days of TA delivered by county
3131
Logic model example #3 – Healthy, thriving children
Impact
STRATEGY Award grants to local contractors to design/deliver “best practice”
parent education classes
INTERMEDIATE OUTCOMES % of children from participating (trained) families entering school
ready to learn
GOAL Healthy, thriving
children.
HLO: % of kindergarteners ready to learn
(Benchmark #18)
Agency Performance
Measures
OUTPUTS# grants awarded by county
32
PART 2:Small group exercise – 20 min.
1. Pick an organization from within your group
2. Pick a goal from that organization
3. Work through the logic model as a team
4. Select two key measures – the most important – for box #3
5. Select a spokesperson to report out
32
3333
Performance Measurement 101:Part 3
Oregon Public Performance Measure Association (OPPMA) Annual Meeting
8:45 – 11:45 Session
with Rita Conrad, Ken Smith and Laura Wipper
Willamette University, Salem, OR
34
Part 3 – Making it Happen
• Performance Management– Integrated across many systems/disciplines– Multiple Steps that usually include:
• Strategy Reports with data Discuss/Argue Learn/Change Revisit Strategy
– No single dominant view – but common tools:• Balanced Scorecard, Strategy Maps, LEAN, Six
Sigma, TQM, Variance Analysis, Organization Alignment/Incentives, etc.
– Numerous barriers, challenges and gaps
35
Some Models for Measurement
36
Background
• Laura has worked in ODOT and other agencies for almost 20 years
• Ken has studied performance reporting by cities and states for over 17 years
37
Why create a “model”
• Our hope is to help participants create their own mental models of:– Where they and their organization are at– What are the next steps/models they can use– What choices and priorities they should consider as they
move forward
• The models are not intended to be judgmental – where one level is always better or more appropriate….the costs/benefits for each approach need to be carefully considered….not every model works for every person
38
Measuring to Managing
39
Three Models Overview
Single Measures Operating Unit “system” Integrated MACRO System
Description Lists one or many measures – does NOT show logic model how the measures connect in a logical fashion
Illustrates or describes how various measures or explanatory information impacts the desired outcome for a single entity or system
Illustrates or describes how various measures or explanatory information impacts the desired outcome for a several interacting entities/systems
Examples Some of the Oregon APPR’s Balanced ScorecardOregon Benchmarks “actual”Carl DeMaio’s “Perf Logic
Model”
GASB “ideal” with explanatory info
Oregon Benchmarks “ideal”City of Coral Springs – case study
Strengths Easy – can be great place to startCollecting and sharing raw data
often starts good conversations
Logic model connections internal to the system/entity
Exhaustive model
Weaknesses Lack of logic model limits usage – and allows different interpretations
No connection to explanatory factors outside the entity such as other orgs & socio-economic environment
Very complicated – especially if using statistical tools like regression
Equation If we measure – performance will improve
Outcome = org measure 1 org measure 2 org measure 3
Outcome = Org measures + Outside Orgs + Environment
40
A Model for Integrated Measures
41
An Example using Education
42
An Example using ODOT
43
44
45
46
47
48
“5 Principles” by Bob Paladino
• Similar to many other models – I like his since he uses both government and business examples – and book available on Amazon $25
• I also like his model since it focuses explicitly on the need for:– a Chief Performance Officer (CPO)– Using multiples tools (BSC, LEAN, Maps, etc)
p.s. I also like him because he is a CPA
49
Five Key Principles
1. Establish CPM (Corporate Performance Management) Office and Officer
2. Refresh and Communicate Strategy
3. Cascade and Manage Strategy
4. Improve Performance
5. Manage and Leverage Knowledge
50
Illustration of “5 Key Principles”
51
Challenges/Barriers
• “9 out of 10 companies fail to implement their business strategies” Paladino p. 11
– Hey there government folks – the so called “business sector” struggles with putting these good ideas into action just like we do
52
Paladino’s Four Barriers to Implementation
• Vision– Only 5% of the workforce understands the strategy
• Management– 85% of executive teams spend less than one hour per
month discussing strategy• Resource
– 60% of organizations don’t link budgets to strategy• People
– Only 25% of managers have incentives linked to strategy
53
Bob Behn - Harvard lecturer (Has a nice monthly newsletter)
• Why Performance Management is NOT sweeping the world? (2002)– Practical reasons: It just does not work– Political reasons: Does not win elections– Managerial reasons: It is da?n hard– Psychological reasons:
• Creates legit fear and• Requires organization changes
54
Building Your Plan
• Three different tools – same basic idea to self-evaluate using a Gap Analysis
– ActiveStrategy online “Strategy Activator”• 19 questions
– Paladino Chapter 9 – “3 Step Self-Diagnostic”• 35 best practices (Score yourself – Gap – Action)
– Weidner “Accountability and Transparency”• 17 questions
55
Additional Case Studies:(time permitting)
• City of Coral Springs, Florida– www.activestrategy.com/about_us/coralsprings_case_study_p1.aspx
• North Carolina Benchmarking Project– www.sog.unc.edu/programs/perfmeas/
• ExPeRT website @ Willamette– www.willamette.edu/centers/publicpolicy/projects/expert/
56
A few links to useful sourcesBOOKS• Balanced Measures for Strategic Planning, by Kathleen Monahan (2000)• Balanced Scorecard, by Robert Kaplan and David Norton (1st book in 1992 –
several additional books and articles)• Five Key Principles of Corporate Performance Management, by Bob Paladino
(2007)
BLOGS• Governing.com is very good – 10 or so writers like Miller, Metzenbaum, etc.
COMPUTER CONSULTANTS• http://www.publicperformancesystems.com/ • Active Strategy – 9 phase model
ORGANIZATIONS• National Performance Management Advisory Commission
