20.04.23 Page 1

The impact oriented M&E-system of the German-costarican program on waste management

Dr. Stefanie KrappEvaluation Unit, German Technical Cooperation (GTZ)

Annual Conference of the Canadian Evaluation Society Victoria, CanadaMay 4th, 2010

20.04.23 Page 2

1. Evaluation system of GTZ

2. The waste management program in Costa Rica

3. The program M&E-System

4. Evaluation design

5. Some selected results

6. Conclusions

Contents

20.04.23 Page 3

Planning a development measure

Implementation Completion

e.g. project progress review

e.g. final evaluation

e.g. ex-post evaluation

Sustainability

Results

e.g. ex-ante evaluation

Results-based monitoring

Evaluations

Managing for development results

20.04.23 Page 4

Project progress review (PPR)

e-VAL

End of project evaluation

e-VAL

Ex-post evaluation2-5 years after completion

of project

Continuous results-based monitoring in every development measure is the prerequisite for all evaluations.

Self-evaluationSteering by

operative units

Independent evaluationSteering by

Evaluation Unit

External evaluation

GTZ instruments

Evaluation by the Ministry of

Economic Cooperation

and Development

(BMZ)

Evaluation by auditors

on behalf of BMZ

Evaluation system

GTZ Evaluation system

Decentralised evaluationSteering by

operative units

Mostly interim evaluation

20.04.23 Page 5

Component 1: Comunication and

dialogue

Public and private sector and the civil

society work constructively together in the

waste management.

Component 2: Strategies, plans and legal frame

Relevant stakeholders

accomplish the new legal frame conditions and

political development strategies in

waste management.

Component 4: Competition and

environment friendly behavior of the industrie

Industry, services and trade

implement production

methods and measures for

resource recuperation.

Component 3: Waste

management in municipalities

Improved abilities of selected

municipalities to carry out an

integrated waste management.

German-costarican program “Competition and Environment”

Goal: Improvement of waste management in municipalities and the industry in selected regions of Costa Rica (04/2005 – 03/2013)

20.04.23 Page 6

Intended Impacts

Diffusion: The implemented structures and processes in selected municipalities are taken over by other municipalities and companies which lead to the improvement of waste management in other regions and sectors.

Positive image of Costa Rica as an ecotourism country

Improvement of health

Costa Rica as knowledge carrier in waste management and innovative environmental technologies

Growing environmental awareness

Growing resource efficiency

….

German-costarican program “Competition and Environment”

20.04.23 Page 7

Performance Framework:

Result chains for each component with indicators on each level (output, use of

output, outcomes, impacts)

Internal Monitoring:Baselines & continuous

performance self-assessment with data collection,

time series analysis,

household survey, foto

documentation (output, use of

output, outcome)

Internal and external Eval.:

a) Project Progress Reviews

b) Specific component related

topicsc) Anual panel

according to outcome and

impact indicators

¡Participatively developed and implemented!¡Linked with the program management system!

The impact oriented M&E-System of the program CYMA

M&E System

20.04.23 Page 8

Time series analysis Ex. 1: Development of the quota of correctly deposited, composted or recycled waste in the municipalities

Before Aftert1

70%

t2

75%

t3

78%

t4

85%

t5

90%

Intervention:Implementation of waste management plans

in the municipalities

Ex. 2: Development of the quota of households connected to the waste management system in the municipalities

Internal Monitoring

20.04.23 Page 9

Program group (P) = 8 pilot municipalities

Counterfactual: Comparison group by matching procedure (C) = 3 municipalities not related to the program

Baseline and continuous collection of data according to relevant indicators of P and C

Standardised telephone survey of municipal households about the waste situation in the municipalities (P and C)

Randomwalk: Photographical documentation of the „visual“ waste situation in the municipalities (P and C)

Internal Monitoring C3

20.04.23 Page 10

Panel evaluation design

Impact model:

Capacity of the

counterparts, Outcome and

Impact

Longitudinal study, incl.

expert assessment,

guided interviews

Evaluation frame:

indicators of the impact

model

External Evaluation

20.04.23 Page 11

Outcome

CYMA

Fin. ResourcesCommunication

Staff

Organisational structure

Goal system

Impacts

Component 4Component 1

Component 2

Counterparts

Component 3

Improved waste management in municipalities and industries in selected regions of CR.

Partner structure

Impact Model

20.04.23 Page 12

Baseline

2006

2nd survey

2007

3rd survey

2008

4th survey

2010

The same variables

with the same operationalisation (guided interviews)

are measured with the same people

at different points of time.

Before After

Panel groups

External Evaluation: Panel

20.04.23 Page 13

(2006, 2007, 2008, 2010)

Experts

Stakeholders

Counter-parts

3 Ministries (Planning, Health, Environment), Chamber of Industries, IFAM, 4 pilot municipalities, program respresentatives

Municipalities, Tourism, university professors, Journalists

Gtz-Program PRODELO, 4 municipalities, chambers, ministeries (Economy, Agriculture), NGOs, Institutes (CNP+L, CEGESTI, CICAP), Associations (ACEPESA, ACIPLAST, FEMETROM, FEDEMUR, CONADECO), local respresentatives (PRU-GAM, UNGL), political respresentatives, companies (Holcim, Dos Pinos, Prolusa, Sur Química, Gente Reciclando, Fundación Escazú Recicla)

Panel groups

20.04.23 Page 14

Sumary

Monitoring data:

Baseline Time series Standardised telephone household survey Foto documentation of the waste situation in municipalities

Double-Difference Analysis:Comparison of data collected at t3 with data collected at t1 (baseline data) and t2 of the program and comparison group

Data of the Panel survey

Comparison of data collected at t4 with data collected at t1 (baseline), t2 and t3

comparison with monitoring results at different points of time

Determine the attribution of program impacts

20.04.23 Page 15

Law on waste management almost ratified, National Plan of Waste Management, Municipal Waste Management Plans

Diffusion of resource gentle and environment friendly production in the private industry

Increased separated disposal of waste

Intensified interaction in the waste sector and therefore continuing presence of the topic in the public

Appreciation of the political role of the chamber of industries

Costa Rica as knowledge carrier

BUT: still critical pollution (mainly oncerning water, waste disposal and recycling)

Selected results

20.04.23 Page 16

Main functions and goals of the M&E-System Joint learning

Information on how the program is doing

Steering information for the program management

Evidence of program results (outcomes and impacts)

Basis for the development of a national waste management monitoring system in Costa Rica

Main conclusion:

The M&E system is not perfect but tries to show how the program is doing and gives valid information on outcomes and hints on impacts under the given circumstances by applying different methods as „rigorous“ as posible!

Conclusions

20.04.23 Page 17

logical result chains and useful indicators flexible system that allows adaptations baselines continuous data collection good data basis in the municipalities participation of counterparts in the development and

implementation of the M&E-system (ownership) willing program manager M&E system connected with the management system staff responsible for the monitoring system combination of internal and external evaluations anchored in

the monitoring system pretension of rigorous evaluation designs to attribute impacts

(determination of a comparison group right at the beginning)

Conclusions: Prerequisites

20.04.23 Page 18

Thank you! Merci! Gracias! Danke!

Dr. Stefanie KrappEvaluation Unit, German Technical Cooperation (GTZ)

stefanie.krapp@gtz.de

www.gtz.de