The Role of Monitoring & Evaluation for Public Policies¤ge/Bangkok... · The Role of Monitoring & Evaluation for Good Governance October 2015 King Mongkut’s University of Technology

The Role of Monitoring & Evaluation

for Good Governance

October 2015

King Mongkut’s University of Technology

North Bangkok

Prof. Dr. Reinhard Stockmann

Director of the Center for Evaluation,

Saarland University, Germany

Content

1. Traditional vs. Modern Administration

2. New Public Management, Outcome-oriented governance, evidence based policy

3. Monitoring & Evaluation as instruments for rational policy decision making

4. Functions of evaluation in society

5. Utility of M&E in the Program Cycle

6. Recommendations for development

October 2015 Prof. Dr. Reinhard Stockmann 2

Traditional vs. Modern Administration

New Public Management, Outcome-oriented governance, evidence based policy

Monitoring & Evaluation as instruments for rational policy decision making

Functions of evaluation in society

Utility of M&E in the Program Cycle

Recommendations for development

Traditional Administration

Control via input factors

• Financial resources

• personnel

• equipment

Control via volume of resource allocation

Puts „wrong“ incentives ineffective

activities

Negative image of public administration

3 October 2015 Prof. Dr. Reinhard Stockmann







New Public Management

Aim:

To improve performance and the process of service delivery in public

administration.








Principles of NPM

5

Customer orientation

Competitive orientation

Output and outcome orientation

Quality orientation

October 2015 Prof. Dr. Reinhard Stockmann







Methodological Challenges

Traditional control- and financing instruments are not suitable for outcome-oriented management

M&E-instruments are essential for this new task

6

Outcomes need to be measured and causes be identified

= pre-condition for outcome-oriented

management








Tasks of Monitoring

Control of implementation according to the planning

To supply management with data about program development (input, output) and goal achievement (impact) on a continuous basis as well as with other impact data

continuous, routine activity








Monitoring examples


Level Example Indicators

entire system (society)

social indicator system

happiness satisfaction in life medical support educational qualification

policy-system (subsystem of

society)

employment and unemployment

employment rates employment by

economic sector profession social position

unemployment rate

program-level development cooperation

program

improvement of educational opportunities for girls

improved knowledge better job opportunities







Planning- and Steering Instrument

9

Monitoring

result- oriented process-

oriented

Comprises

output, goal

achievement and

impacts of a

program

Comprises control

of implementation

according to the

planning

Navigation








Evaluation

10

Evaluation as a “misused” concept:

monitoring reports, experts’ reports, reviews, judgments, assessments








Everyday Meaning

11

Something is assessed by someone

according to some criteria in some way. (Kromrey 2001: 106)








Definition of Evaluation

12

by being related to a clearly defined object

by being conducted by people with specialized competences

by intersubjectively verifiable criteria

by objectifying empirical data collection methods

by applying systematically comparative procedures in the assessment of that information

Evaluations conducted scientifically or professionally distinguish themselves








Criteria for ‚Good‘ Evaluations

professional, fulfilling scientific standards

useful for political or administrative decisions

participative, including different stakeholders

feasable, balance between demand / requirements and feasibility








Requirements of Evaluation

14

An evaluation should be guided by both the clarified purposes of the evaluation and the information needs of its intended users (Utility).

An evaluation should be planned and conducted in a realistic, thoughtful, diplomatic, and cost-effective manner (Feasibility).

In the course of the evaluation, all stakeholders should be treated with respect and fairness (Propriety).

An evaluation should produce and disclose valid and useful information and findings pertaining to the evaluation question (Accuracy).








Constraints between the Requirements

15

Scientific standards/ accuracy

Utility

Feasibility








Comparison M&E

16

Monitoring Evaluation

continuous, routine task, often inhouse

controls implementation

process

descriptive data collection according to

program indicators

process control

sporadically, special occasion, often external

puts goals, objectives, program theory into question

advanced methods, causal-impact

analysis

strategic control

use

objective

method

result








Combining M&E

M&E form a unit

Evaluation uses information of

monitoring, thus synergies occur

Both instruments do cover different

information needs and are necessary for

process and strategic control








Triple function of Evaluation

18

Evaluation

Implementation

Programs

Policy

Policy strategies

Society

Norms and values

opens up learning potential

provides recommendations

creates transparency

helps supervision

improves level of information

opens up opportunities for democratic participation

makes policy more effective, efficient and sustainable

makes policy more credible

allows policy to be assessd using sustainablity criteria

Management Legitimacy Enlightenment








M&E in the Program Cycle

19

planning

implemen-tation

end of sponsor-

ship

implemen-tation

survey of baseline data

continuous monitoring

replanning (annual

planning)

formative evaluation

midterm review

external evaluation

final evaluation

Program run

M+E cycle








Benefits of M&E

20

planning implementation impact

preformative formative/ summative

summativ

ex-ante on-going ex-post








Benefits at Planning Stage

M&E answer questions like…

Are program goals and objectives clearly formulated?

What measures are needed by the target group?

How can the measures be implemented?

What general conditions have to be taken in mind?

What resources for the program implementation are needed?








Benefits at Implementing Stage


How has the program developed? Goals achieved?

What kind of implementation-problems exist?

Are the existing goals/objectives still the ‘right’ objectives?

Are the intended target groups reached?

Who benefits from and who is harmed by the program?

Were program-risks considered correctly?








Benefits at Impact Stage



Which intended and non-intended effects occurred?

Did intended effects arise among the target group?

Are impacts caused by interventions?

Are impacts relevant for the country’s policy?

Is there any diffusion of impacts?

Are the impacts of the program sustainable?







Steering with the Help of M&E

24

Objectives at level of

operation

management

M&E-system

Goal adjustment steering

data collection + assessment with

the help of

deliver information

for

assesses,

decides

comparisons








Preconditions for M&E

Lessons learned from implementing and using M&E systems:

Five factors are important:

(1) Acceptance

(2) Clear Structure

(3) Qualification

(4) Utilization

(5) Quality Standards








(1) Acceptance

Define M&E as a priority task

Make all stakeholders involved understand the whole purpose of an M&E-system

Involvement of stakeholders in the development of an M&E-system from beginning on

Regular informing of all stakeholders involved

No misuse of the information for control or punishment

Make open and constructive criticism possible

Ensure that recommendations are implemented

26

Establish a culture of learning








(2) Clear Structure

Clear and transparent formulation of M&E-goals

Tasks have to be identified

Clarify communication channels

Provision of personnel and financial resources

Establish a manageable M&E-system

Implementation into the organization structure

27

Apply clear responsibilities








(3) Qualification

Convince personnel of task and qualify it

Convince decision-makers of an organisation that the M&E-system functions so that they use it

Support by external experts upvalues work

28

Monitoring and Evaluation are part of

Capacity Development








(4) Utilization

Reliable, valid, feasible, comprehensible and prompt results

Results must be relevant for steering

Clarify responsibilities for the use of the results

29

Functioning relationship between monitoring-system and management








(5) Quality Standards

30

Utility – M&E serves the practical information needs

Feasability – Methods, organizational processes, timetable and implementation are realistic, carefully considered and cost-effective

Propriety – M&E activities are implemented legally, ethically and in regard to the context conditions

Accuracy – M&E outputs include and impart reliable information

M&E is used by decision-makers








Recommendations…

31

… to strengthen the role of M&E for policy

making:

1. Advocacy in state and society

2. Development of M&E cultures in governmental institutions and nonprofit organizations in terms of “learning cultures” – not “control cultures”

3. Stronger integration of interests of stakeholders and participants

4. Buildup of internal M&E capacities

5. Strengthening of independent external M&E capacity

6. Establishing of legal basic principles

7. Funding of research and development








Recommendations for Activities…

32

1. Quantitative expansion and qualitative improvement of vocational training and further training of M&E experts

2. Development of training services for operative and management staff

3. Improving of professionalization of M&E

4. Development of networks and media for communication (e.g. by evaluation societies, journals, publications, conferences, etc.)

5. Increase of international cooperation and exchange

6. Development of Evaluation standards for evaluations, codes of conduct etc.

7. Further development of theories and methods of evaluation research








… to develop evaluation capacities:


Prof. Dr. Reinhard Stockmann

Thank you for your attention!

Documents

The Role of Monitoring & Evaluation for Public Policies¤ge/Bangkok... · The Role of Monitoring & Evaluation for Good Governance October 2015 King Mongkut’s University of Technology