1 Performance Measurement 101 PART 1 Oregon Public Performance Measure Association (OPPMA) Annual...

11

Performance Measurement 101PART 1

Oregon Public Performance Measure Association (OPPMA) Annual Meeting

8:45 – 11:45 Session

with Rita Conrad, Ken Smith and Laura Wipper

Willamette University, Salem, OR

22

Today’s Session • Part 1: The basics Part 1: The basics (~60 min)

– Intro & Brief History– Criteria– Logic models

• Part 2: Small groups Part 2: Small groups (~75 min)– Practice– Report-out

• Part 3: Making it happen Part 3: Making it happen (~60 min)– Three approaches– Dialogue

3

Late 1800’s-1930’s

• New York Municipal Research Board

• Audit of Portland, Oregon

• International City Management Association (ICMA)– Herbert Simon, Nobel Prize winner

4

1970’s Productivity

• Optimistic Views– Harry Hatry (1972; 1978)

• Negative or Cautionary Views– Quinn (1978);– Burkhead and Hennigan (1978)

5

1980’s – Why Accountants?

• Governmental Accounting Standards Board (GASB)

• Government Finance Officers Association (GFOA)

• Association of Government Accountants (AGA)– Oregon is the best state at performance reporting – by a

long shot (Smith, Cheng, Smith, Schiffel 2006 – available online)

6

Original Oregon Model @ ODOT• Roots in work of Glenn Felix

– Productivity matrix• Baseline – or average – levels of performance

• Historical best or optimum goals

• Relative weights, performance levels and performance index

• Efficiency and Effectiveness– Cost vs. Quality

– Gainshare pilot at ODOT proves value is in the measures and the conversation

7

Striving for Balance

•Cost per product, service or result•Labor per product, service or result•Direct hours•Indirect hours•Administrative hours

•Outcome/Goal•What’s Important Along the Way•Customer Satisfaction•Work Life Quality

8

Work Life Quality

• At ODOT• Regularly gauged the capacity of the organization

– Sick leave

– Safety

– Turnover

– Employee Survey

– Equal Opportunity Employment

– Training

9

Development of Measures

• Series of workshops– Involved almost all staff– Aligned activities to purpose, outcomes, goals– Measures were customized by staff

• Alignment through Key Result Areas

• Some were “roll-ups”

10

Statewide Rollout• Statewide measurement initiative began in

1991 with 16 agencies – Phase 1

• More phases followed – 100+ agencies

• Agency-level measurement facilitated by mentors from previous phases

• Expanded upon ODOT model– Efficiency and effectiveness– Tied goal-orientation to Oregon Benchmarks

11

Some 90’s Success Stories• DMV

– Positive experience with a negative index• Ways and Means liked the candid conversation

• Adult and Family Services– Moved focus from application processing to family self-

sufficiency• Lead the nation in welfare reform

• Vocational Rehabilitation– Decided federal goal of two months back on the job not

high enough to justify cost• Set goal at 18 months

• Department of Corrections– Sharper focus on direct recipient of program efforts

• 180 degree swing on recidivism

12

Original Oregon ModelAdopted across the US

• Implemented nationwide by US Army Corps of Engineers– One of first two agencies to act following

federal Government Performance Results Act of 1993

• At one point, 70 Federal agencies were implementing the “Oregon” model following success of Corps

13

Elements of Oregon’s Key Performance Measure System

Tiered, linked system for external reporting Common language Logic models Criteria Budget

14

SB555 gives Progress Board first time

evaluation function

HB 3358 gives performance

measurement role to Progress Board

Linking state performance to benchmarks

93 94 95 96 97 98 99 00 01 02 03 04 05 06 07

Measuring Up

by Jonathan Walters

98

Oregon laws required agency

performance measures 93 09

08

Progress Board prepares KPM-

Benchmark Crosswalks for W&M

subcommittees

15 Oregon Benchmarks and Performance Measures

Activity #1

Getting to Know You

1717

Why measure performance? To get results.

It’s at the core of results-based management• Fosters internal learning and improvement

Is the ship running well?

• Provides greater external accountabilityIs the ship on course?

• Informs policy-making Should the ships change course?

1818

CRITERIA What makes a good performance measure?

1. Common language2. Alignment (line-of-sight)3. Accurate and reliable data4. A few key measures5. Time-framed targets6. Someone in charge7. Comparisons8. Customer satisfaction

19

• OUTCOME, high-level = Societal result

• OUTCOME, intermediate = Entity result

• OUTPUT = Product or service

• INPUT = Time or money

• EFFICIENCY =Input per output or Input per outcome

Criterion #1. Common language

19

20

3. High-Level Outcomes (HLO) – Is the world you are attempting to affect changing?

Oregon Benchmark #65, Juvenile Recidivism – % of juveniles referred w/in 12 months …

Practical application

2. Intermediate Outcomes – Is the strategy producing the desired result? % of treated youth with reduced risk factors

1. Outputs – Is the strategy getting done? # of youth completing treatment

20

21

Criterion #2. Alignment - line-of-sight to goals and high-level outcomes

Goal

Key Performance Measures

IntermediateOutcomes

Outputs

HLO - Mission or Benchmark

Impact

Strategies to achieve the goal

21

2222

Criterion #3. Accurate and reliable data

• Trustworthy data is essential.Example: verifiable records trump estimates

• Per measure - at least one data point, preferably several

• Data should match the measure.

23

Criterion #4. Few key measures – different measures for different purposes

More agency influ

ence

More public in

terest

Goal: Prevent Juvenile Crime

% of grantees trained INCREASES.

Output

“So That”

% of high risk youth completing program INCREASES.

Intermediate Outcome

“So That”

% of served youth with mitigated risk factorsINCREASES.

Intermediate Outcome

“So That”

Juvenile recidivism DECREASES.

High-Level Outcome

How high up do you want

to go?23

24

Criterion #5. Time-framed targets

• TARGET = Desired level at a given point in time

– Ambitious but realistic

– Set targets using:

• trend data

• comparisons

• expert opinion

2007 2007 TargetTarget: :

1515

2003 2003 Actual: Actual:

2020Example: hours of travel Example: hours of travel delay per capita per yeardelay per capita per year

24

25

Criterion #6. Someone in charge

• Management, staff and stakeholders need to know where the buck stops

• Critical for improving performance

25

26

Criterion #7. Measures of customer satisfaction

• This is a required Key Performance Measure (KPM) for Oregon state agencies.

• Rationale: Those who pay for their government should be satisfied with their government.

26

27

Criterion #8. Comparisons

• How does actual performance compare to industry standards?

• Rationale: to better understand whether action is needed.

• Difficult but not impossible!

• Agency suggestion: “peer” agencies develop common measures and comparator

“We resolved

complaints within three

weeks on average.”

Is this good or bad?

27

2828

LOGIC MODEL - a tool for thinking it through

Getting started1. IMPORTANT: Engage those whose

performance will be measured.

2. Have a clear statement of goals.

3. “Look up” to mission & high-level outcomes (benchmarks)

4. “Look down” to what you are doing and how you are measuring performance

AgencyGoals

Performance Measures

High-level outcomes

2929

Logic model example #1 – Citizen involvement in land use planning

Impact

STRATEGYJointly sponsor, with cities,

regional educational events for private citizens every quarter.

INTERMEDIATE OUTCOMES % participating citizens with

improved understandingCustomer satisfaction ratings

GOAL: Citizen involvement (C.I.) in

land use planning

HLO: % of cities with neighborhood

organizations.

Agency Performance

Measures

OUTPUTS# citizens trained.

# C.I. guidelines distributed.

3030

Logic model example #2 – Juvenile crime

Impact

STRATEGY Award grants to local contractors to conduct “best practice” juvenile crime prevention programs (JCP).

INTERMEDIATE OUTCOMES % of juveniles in JCP programs with significantly mitigated risk

factors.

GOALReduce juvenile

crime.

HLOJuvenile Arrests

(OBM#61)

Agency Performance

Measures

OUTPUTS# grants awarded by county

# days of TA delivered by county

3131

Logic model example #3 – Healthy, thriving children

Impact

STRATEGY Award grants to local contractors to design/deliver “best practice”

parent education classes

INTERMEDIATE OUTCOMES % of children from participating (trained) families entering school

ready to learn

GOAL Healthy, thriving

children.

HLO: % of kindergarteners ready to learn

(Benchmark #18)

Agency Performance

Measures

OUTPUTS# grants awarded by county

32

PART 2:Small group exercise – 20 min.

1. Pick an organization from within your group

2. Pick a goal from that organization

3. Work through the logic model as a team

4. Select two key measures – the most important – for box #3

5. Select a spokesperson to report out

32

3333

Performance Measurement 101:Part 3

Oregon Public Performance Measure Association (OPPMA) Annual Meeting

8:45 – 11:45 Session

with Rita Conrad, Ken Smith and Laura Wipper

Willamette University, Salem, OR

34

Part 3 – Making it Happen

• Performance Management– Integrated across many systems/disciplines– Multiple Steps that usually include:

• Strategy Reports with data Discuss/Argue Learn/Change Revisit Strategy

– No single dominant view – but common tools:• Balanced Scorecard, Strategy Maps, LEAN, Six

Sigma, TQM, Variance Analysis, Organization Alignment/Incentives, etc.

– Numerous barriers, challenges and gaps

35

Some Models for Measurement

36

Background

• Laura has worked in ODOT and other agencies for almost 20 years

• Ken has studied performance reporting by cities and states for over 17 years

37

Why create a “model”

• Our hope is to help participants create their own mental models of:– Where they and their organization are at– What are the next steps/models they can use– What choices and priorities they should consider as they

move forward

• The models are not intended to be judgmental – where one level is always better or more appropriate….the costs/benefits for each approach need to be carefully considered….not every model works for every person

38

Measuring to Managing

39

Three Models Overview

Single Measures Operating Unit “system” Integrated MACRO System

Description Lists one or many measures – does NOT show logic model how the measures connect in a logical fashion

Illustrates or describes how various measures or explanatory information impacts the desired outcome for a single entity or system

Illustrates or describes how various measures or explanatory information impacts the desired outcome for a several interacting entities/systems

Examples Some of the Oregon APPR’s Balanced ScorecardOregon Benchmarks “actual”Carl DeMaio’s “Perf Logic

Model”

GASB “ideal” with explanatory info

Oregon Benchmarks “ideal”City of Coral Springs – case study

Strengths Easy – can be great place to startCollecting and sharing raw data

often starts good conversations

Logic model connections internal to the system/entity

Exhaustive model

Weaknesses Lack of logic model limits usage – and allows different interpretations

No connection to explanatory factors outside the entity such as other orgs & socio-economic environment

Very complicated – especially if using statistical tools like regression

Equation If we measure – performance will improve

Outcome = org measure 1 org measure 2 org measure 3

Outcome = Org measures + Outside Orgs + Environment

40

A Model for Integrated Measures

41

An Example using Education

42

An Example using ODOT

43

44

45

46

47

48

“5 Principles” by Bob Paladino

• Similar to many other models – I like his since he uses both government and business examples – and book available on Amazon $25

• I also like his model since it focuses explicitly on the need for:– a Chief Performance Officer (CPO)– Using multiples tools (BSC, LEAN, Maps, etc)

p.s. I also like him because he is a CPA

49

Five Key Principles

1. Establish CPM (Corporate Performance Management) Office and Officer

2. Refresh and Communicate Strategy

3. Cascade and Manage Strategy

4. Improve Performance

5. Manage and Leverage Knowledge

50

Illustration of “5 Key Principles”

51

Challenges/Barriers

• “9 out of 10 companies fail to implement their business strategies” Paladino p. 11

– Hey there government folks – the so called “business sector” struggles with putting these good ideas into action just like we do

52

Paladino’s Four Barriers to Implementation

• Vision– Only 5% of the workforce understands the strategy

• Management– 85% of executive teams spend less than one hour per

month discussing strategy• Resource

– 60% of organizations don’t link budgets to strategy• People

– Only 25% of managers have incentives linked to strategy

53

Bob Behn - Harvard lecturer (Has a nice monthly newsletter)

• Why Performance Management is NOT sweeping the world? (2002)– Practical reasons: It just does not work– Political reasons: Does not win elections– Managerial reasons: It is da?n hard– Psychological reasons:

• Creates legit fear and• Requires organization changes

54

Building Your Plan

• Three different tools – same basic idea to self-evaluate using a Gap Analysis

– ActiveStrategy online “Strategy Activator”• 19 questions

– Paladino Chapter 9 – “3 Step Self-Diagnostic”• 35 best practices (Score yourself – Gap – Action)

– Weidner “Accountability and Transparency”• 17 questions

55

Additional Case Studies:(time permitting)

• City of Coral Springs, Florida– www.activestrategy.com/about_us/coralsprings_case_study_p1.aspx

• North Carolina Benchmarking Project– www.sog.unc.edu/programs/perfmeas/

• ExPeRT website @ Willamette– www.willamette.edu/centers/publicpolicy/projects/expert/

56

A few links to useful sourcesBOOKS• Balanced Measures for Strategic Planning, by Kathleen Monahan (2000)• Balanced Scorecard, by Robert Kaplan and David Norton (1st book in 1992 –

several additional books and articles)• Five Key Principles of Corporate Performance Management, by Bob Paladino

(2007)

BLOGS• Governing.com is very good – 10 or so writers like Miller, Metzenbaum, etc.

COMPUTER CONSULTANTS• http://www.publicperformancesystems.com/ • Active Strategy – 9 phase model

ORGANIZATIONS• National Performance Management Advisory Commission

57

Contact Information

Rita ConradRitasSpace@comcast.net

Ken SmithSmithK@Willamette.edu

Laura WipperLaura.R.WIPPER@odot.state.or.us