WHAT CAN WE LEARN FROM SUCCESSFUL COUNTRY SYSTEMS? PHILIPP KRAUSE PRMPR 1

1

WHAT CAN WE LEARN FROM SUCCESSFUL COUNTRY SYSTEMS?

PHILIPP KRAUSEPRMPR

2

Overview

Why bother? A selection of country cases How do differences matter? What do “successful” systems have in

common?

3

Why bother?

Two schools of thought The “best practice” school The “It’s all about the context” school

The truth lies somewhere in between – there are important characteristics shared by successful cases, but differences between countries matter tremendously

Most important reason: Countries do M&E for different reasons and different users – what you need depends on what you want to do with it

Our definition of success: High degree of utilization M&E information meets quality standards and is reliable System is sustainable over time

4

Overview


common?

5

Australia: Main Features

Mandate to evaluate each program every 3-5 years

Portfolio Evaluation Plans to be prepared annually for the following three years

Department of Finance: Steering & quality control Sector Departments: planning, implementation Evaluation results primarily used for budgetary

decisions: allocations of funds for new policies and reallocation of savings (i.e. the discretionary part of annual budget process)

System lasted from 1987-1996 – sustainability?

6

Australia: How does it work?

Sector Departm

ents(and outrider

agencies)

Treasury

Department of Prime

Minister and Cabinet

Department of

Finance

Parliament

Citizens

Inputs to PEPs

Inputs to PEPs

Inputs to Portfolio Evaluation Plans (PEPs)

Formal notification of

PEPsInvolvement in evaluations

Publication of department’s

evaluation reports

Reporting of keyEvaluation findings in department’s budget paper (prospective) and

in their annual reports (retrospective)

Source: Mackay 2011

7

United Kingdom: Main Features Comprehensive system of performance monitoring

and targets: Spending Reviews (multiannual budgets) and Public Service Agreements (“resources for delivery”) in return for (relative) managerial flexibility

Oversight and leadership from central executive (Prime Minister, PM’s Delivery Unit, Treasury)

No systematic evaluations in the executive government

Problem: “Gaming in Targetworld” Value-for-Money Audits: 60 per year from National

Audit Office

8

Delivery Unit*

UK: How does it work?

HM Treasury

Parliament

audits

Prime Minister’s Office is accountable to

reports to

National Audit Office

Ministers

Spending Unit

monitors priority areas

reporting/monitoring

is accountable to

Negotiations over PSAs

* Moved to HM Treasury in 2007, abolished 2010

9

Mexico: Main Features

M&E of social policies delegated to a specialized technical agency – CONEVAL

CONEVAL in charge of evaluation portfolio, development of methods, dissemination

Key oversight decisions by inter-ministerial committees – system involves many stakeholders (Congress, Finance, Presidency, Public Admin Ministry, Sector Ministries, CONEVAL)

Implementation of evaluation and day-to-day operation of monitoring done by ministries

10

Mexico: How does it work?

Inter-ministerial Committee

SectorMinistr

y

CONEVAL

SectorMinistr

y

SectorMinistr

y

CONEVAL

•Define the evaluation plan•Decide what and how to evaluate

•Coordinate and steer evaluation system•Maintain quality control•Define monitoring parameters

•Select and hire evaluators•Supervise evaluation implementation•Monitor indicators and targetsPrograms are evaluated

by ministries •Collection of evaluation results•Dissemination of evaluation results

•Utilization of evaluation results

1.

2.

3.

4.

5.Congress, Presidency, Sector Ministries, Finance

11

Chile: Main Features

System is highly centralized – closely linked to and based on a highly centralized, top-down budget process

Budget office manages most details, and is main user of information – utilization fostered by close link to budget process

Management control by hierarchical oversight Little buy-in from other (potential) stakeholders Some impact on allocations and program management

Chile: Utilization of Government Evaluations—2000–05Effect onprogram

Minor adjustmentof program, forexample, improvedprocesses orinformation systems

Major adjustmentof managementprocesses, for example,changed targetingcriteria, or newinformation systems

Substantial redesignof program ororganizational structure

Institutional relocationof program

Programtermination

Programsaffected

24% 38% 25% 5% 8%Percentage of all evaluated programs. Source: Guzman 2007

12

Chile: How does it work?12

Budget Office

Budget Office

Budget Office

Programs

Budget Office



Programs are evaluated directly by Budget Office

•Select and hire evaluators•Supervise evaluation implementation•Monitor indicators and targets

•Collection of evaluation results•Dissemination of evaluation results


1.

2.

3.

4.

5.

Budget Office

Congress

13

Centralization vs. Delegation

Inter-ministerial Committee

Mexico

SectorMinistr

y

CONEVAL

SectorMinistr

y

SectorMinistr

y

Chile

Budget Office

Budget Office

Budget Office

Programs

Budget Office

CONEVAL

Congress, Presidency, Sector Ministries, Finance



Programs are evaluated directly by Budget Office

•Select and hire evaluators•Supervise evaluation implementation•Monitor indicators and targets

Programs are evaluated by ministries •Collection of evaluation results

•Dissemination of evaluation results


Budget Office

Congress

1.

2.

3.

4.

5.

14

Overview


common?

Institutional Differences Matter: Index of

Legislative Budgetary Powers

Chile

Mexico

Source: Wehner 2007

16

Difference in Purpose

Budgetary: To inform budgetary decisions, best allocation of resources between sectors and programs, also to enforce operational savings in annual budget

Accountability: External accountability towards legislature, stakeholders and public. To make systematic information on performance available and strengthen the public evidence base of policy decisions

Control: To develop better central government information on implementation and service delivery of public programs as tools to hold managers to account

17

Matrix of Country Comparison

Country Purpose Leadership Operation Users

Chile Budget/Control Direct Centralized Single

Mexico Budget/Accountability Delegated Decentralized Multiple

Australia Budget Direct Decentralized Single

Canada Budget Direct Decentralized Single

UK/NAO* Accountability Direct Centralized Multiple

* Supreme Auditor

18

Features that Matter

Centralization requires the right institutional structure – a centralized M&E system design in a fragmented public sector will fail

In a system with multiple stakeholders delegation to an impartial agency might be a viable option – but beware of over-engineering and objective overload

Who is to gain and who has to worry about buying into a M&E system: The senior civil service, the legislature, ministries, service delivery units, the finance ministry, the head of government (PM or President)?

Staying in control of overall steering and quality of outputs does not equal having to internalize all aspects of M&E implementation – strategic delegation might be smart for buy-in and workload

Is it possible to imagine a long-term sustainable, well utilized M&E system that does not have a stable link to budgetary decisions?

19

Overview

Why bother? A selection of country cases How do differences matter? What do “successful” systems have

in common?

20

Lessons Start M&E Systems Successfully* (1)

Somewhere in government is substantive demand for M&E information. This is necessary to start and sustain an M&E system

Actors need to have incentives to engage with M&E. They are key for M&E to be conducted and for the information produced to be utilized

Simple is better – successful M&E systems tend to deliver just what users want, not more. They also serve only those objectives that result in utilization

Success is more likely with a powerful champion(s) to lead the push for institutionalization of M&E – rather than a legislative or technical exercise

*very liberally adapted from Mackay 2010

21

Lessons Start M&E Systems Successfully (2)

It is important to have the stewardship of a central, capable ministry that can design, develop, and manage the system

Some reforms may start with a bang, but it requires patience, determination, and a long-term effort to build an effective M&E system

For donors: It helps the process to start with a diagnosis of what M&E functions already exist in the country (and why other M&E functions do not exist – they usually don’t for a reason)

22

THANKS!

For further information, and to access the sources cited here, please visit:

http://go.worldbank.org/7MZRWD6K50

http://go.worldbank.org/7MZRWD6K50

Documents

WHAT CAN WE LEARN FROM SUCCESSFUL COUNTRY SYSTEMS? PHILIPP KRAUSE PRMPR 1