Self Review Training 24 June 2010 Performance Improvement Framework

Self Review Training

24 June 2010

Performance Improvement Framework

Central

Agencies DPMC SSC Treasury

2PIF challenge: Showing value to a sceptical public (or reviewer)

-10%

0%

10%

20%

30%

40%

50%

99/00 00/01 01/02 02/03 03/04 04/05 05/06 06/07 07/08 HYEFU 08/09

Ch

an

ge

in R

ea

l Pri

ce

or

Sta

ff

Financial Year

Mythological Ministry of MagicMission: Making Money Materialise

Staff Growth (FTE)

Personnel Expense per Employee (FTE)

Personnel Expenses (ex. ACC)

Total Operating Expenses

Central


#1.TechnicalEfficiency

#2.Allocative Efficiency

#3.Cost

Effectiveness

VALUE FOR MONEY

(Best Result for NZers)

Value for Money is hard to assess directly … but poor results at ANY corner of the ‘triangle’ shows under-performance

Central


#1. PRICE, QUANTITY & STANDARDS (TECHNICAL EFFICIENCY)

Effect of Flying Hours on Maritime Patrol Prices

-20%

-10%

0%

10%

20%

30%

40%

50%

60%

70%

80%

FY 98/99 FY 99/00 FY 00/01 FY 01/02 FY 02/03 FY 03/04 FY 04/05 FY 05/06 FY 06/07

Financial Year

Rea

l cha

nge

in p

rice

or

outp

ut

Total flying hours

Real cost per flying hour

Real cost (Personnel)

Real cost (Total Expenses)

Gulf of Oman (Output D16 - 659 extra hours)

1. How did prices of big outputs change?

2. Did output volume or quality change proportionately?

3. Did quality change?4. Did o’heads grow?5. Can we show

citizens actually benefit from this?

‘Did new spending actually increase or improve output ?’

Central


#2. DID OUTPUT GET TO THE PROBLEM AREA, & EFFECT CHANGE

Large Proportion of New Prison Space Used for Less Serious or Less Risky Offenders

-200

-150

-100

-50

0

50

100

150

200

250

300

Ext

ra P

riso

ners

('03

vs

'07)

Seriousness Score

Reoffending Risk

Seriousness Score 198 251 218 137 63 -1 15 104 17

Reoffending Risk -157 155 127 85 99 81 129 164 118 247

" 0 " " 1 " " 2 " " 3 " " 4 " " 5 " " 6 " " 7 " " 8 " " 9 " " 10 "

Can we link extra $$$ & output to priority groups, areas, needs & opportunities?

Who has / is the problem? Who got interventions or $$$? Who missed out?

How well do allocative rules really work?

Can we deliver more in areas that will benefit more, & limit waste?

Allocative Efficiency ‘Can we get better results without spending more ?’

Central


#3. INCREASED PRICE (OR OUTPUT) IMPROVES CORE OUTCOMES

1. Did end outcomes improve much? 2. Did gains mirror / equal spending?3. Did gains precede or lag changes?4. Intermediate outcomes improved?

‘Are results good, bad or simply indifferent ?’

Look for diminishing returns.Scale & timing ($ vs. outcome)?

Investing to reduce crime …

-20%

-10%

0%

10%

20%

30%

Ch

an

ge

Re

lati

ve

to

FY

01

/02

Real Funding Violence

Drugs & AntiSocial Dishonesty

ERO Approach - Context

ERO evaluates the performance of schools and early childhood services

ERO’s self-review was undertaken as a pilot review for the PIF (July 2009)

The context for ERO’s self-review was “the alignment of services and supporting infrastructure to better support Government priorities for education”

This organisation-wide context meant that ERO tested all aspects of the framework

Brent McPherson- June 2010

ERO Approach - Team

The team:

was appointed by the Chief Executive

had knowledge about the business and governance

had access to information

had evaluation skills

Level Position

(tier 2) National Manager Corporate Services

(tier 2) National Manager Review Services (Central)

(tier 3) Strategy & Performance Manager

(tier 3) Manager Information Services

(tier 3) Chief Finance Officer

(tier 3) Manager Standards & Contracts


ERO Approach - Method

Critical areas identified in the framework were allocated between review members - in pairs (allocated areas were independent to current portfolios)

Each pair recorded their assessment, action points & evidence in a template (to justify their judgement)

The team synthesised results & discussed judgements (moderation)

The report (and recommendations) was discussed with the Senior Management team


Benefits to ERO

The self-review: reinforced a commitment to all staff to

promote a culture of continuous improvement reinforced ERO’s own credibility

(ERO promotes evaluation capacity externally) identified a set of recommendations for action

(the recommendations have been accepted by SMT and integrated into the work programme)

encouraged the Chief Executive to think about ERO’s future approach to internal evaluation


Why Do A Self-Review?

respond to Govt priorities (ie how to “maximise the value received from its education services”)

promote continuous improvement through a formal and structured approach

promote accountability – and a balance of internal and external review (ERO ‘s approach to schools)

prepare for a formal review review the organisation (do it in chunks?) basis for engaging with the Minister (PIAs, SOI)


Issues and ChallengesTo improve the value of the self-review, next time ERO would….. plan aheadschedule at a time appropriate to ERO(and set time aside to conduct the self-review) engage managers and staff in the planning (and engage the senior management team throughout) identify priorities – to reduce time spent on areas where there are high levels of comfortincorporate a greater field component - to test the cascade of initiatives to the field (ERO is a field agency)develop an associated communications plan


Issues and Challenges

using a moderation process to synthesise the results

ERO removed bias by:

ensuring that the views expressed were backed up by supporting evidence

ensuring that review team members did not review their own portfolio


Issues and Challenges

ensure the team has access to evaluation skills

Other recommendations/challenges:

plan for conflicting opinion

treat it as a formal review (ready the documents)

what is the current culture of the agency?


Documents

Self Review Training 24 June 2010 Performance Improvement Framework