1

Impact Evaluation in the Global Environment Facility:

Looking back for Our Common Future

Cairo, April 2009

David M. Todddtodd@thegef.org

Lee Alexander Risbylrisby@thegef.org

2

Global Environment Facility (GEF): The 10 second Overview

• Established in1991• Funds projects undertaken by developing and

transition countries to achieve environmental benefits

• Funding areas: Climate change; biodiversity; international waters; ozone; persistent organic pollutants and land degradation

• Over 10US$ billion in project financing since 1991• GEF Structure: Council (Governance & Strategy);

Secretariat (Project approvals & Monitoring); Evaluation Office (Independent Evaluation)

• Projects implemented by MDBs and UN agencies

3

Impact Evaluation (IE) in GEF: Overview• IE launched in 2005: Demand from GEF Council

(donors and recipients) • Set against background of complexity of the

environment and complexity of projects• Designs employed so far:

– Straightforward before/after analysis with comparison between projects (Ozone)

– With/without + before/after analysis through quasi-experimental design (Biodiversity)

– Theory-of-Change (TOC) (Biodiversity and Ozone)– Combination of before / after and TOC (Ozone)

• “mixed design / methods” approach where possible should be pursued:

5

Ozone Depleting Substances Impact Evaluation in Countries with

Economies in Transition

Building and Implementing Mixed-Design and Method Evaluation

6

Ozone Depleting Substances IE in Countries with Economies in Transition (CEITs):

• GEF began ODS activities in CEITs in 1992 to phase-out CFCs and other chemicals that contribute to Ozone damage (The “Ozone Hole”)

– Supports Montreal Protocol compliance– 26 projects (18 countries) and approximately US$380m of financing.

• Projects aimed at stopping consumption and production of Ozone Depleting Chemicals and technologies

• Key stakeholders: Governments, Implementing Agencies and private sector • Main issues for the evaluation:

– Impact of the GEF on production and consumption of ODS – Impact on legal/illegal trade ODS– Impact on Stockpiling of ODS – Sustainability and catalytic effects– Attribution / contribution issue

• Scope: Portfolio impact – why and how? Project and data synergy

7

ODS IE: Tailoring Design to Context • Data availability assessment:

– Quantitative data on production and consumption exists for CEITs early 1990s – to present.

– Significant quant-qual literature on ODS phase-out, many evaluations but no impact evaluations

– Academic and technical literature on ozone science and environmental / human health impacts

• Availability of scientific modeling of ozone hole recovery• And also on human and animal health impacts (skin cancers)

• Initial Evaluation Design assessment– ODS portfolio would provide an opportunities for mix design and

method approaches:• Quasi-experimental with a TOC – responding to what worked

and why …

8

ODS IE: Tailoring Design to Context

• But, desk review found no viable control / counter-factual countries exist– All other countries developed and developing have taken actions

to stop ODS consumption / production• Only exceptions in 2008: Vatican, San Marino and Timor

• Adjusted design to reflect observed data and sample limitations:– Before & after measures of changes:

• By chemical • Time • Internal comparison between GEF Bank and UN countries• External matched comparison between GEF and Multi-lateral Fund

countries• + Business as usual forecast: what would have happened without

the MP and the GEF project?– TOC: development of TOC based on review of project documents and

available terminal evaluations

9

Output: National commitment to eliminate use (production & consumption)

Output: Coordinated Regional Policies

Output: Technology producers and distributers supported to convert

Output: Users supported to adopted sustainable technology

Outcome: Phased-out supply and demand for ODS technology

Assumption Availability of new technologies

Intermediate: ODS substances phased-out

Intermediate: ODS technology phased-out

Impact: ODS tons eliminated

Impact Evaluation ODS Theory of Change

Assumption: Enterprises are reached (large, medium and small)

Impact: Ozone layer improved

Assumption: New Technologies are affordable

Strategy: To protect human health and the environment by assisting countries to phase-out ODS consumption and production, while enabling alternative technologies and practices

Assumption: Enterprises investing are sustainable

Assumption: Effective monitoring and reporting on ODS

Assumption: Recovery and recycling schemes or stockpiling meet temporary demand for ODS

Threat Reduction: Reversion to ODS technologies

Threat Reduction: Illegal trade in ODS

10

Methodology• TOC provides input into ODS IE evaluation matrix: sub-questions /

data sources / methods and indictors• Pre-fieldwork meta-evaluation of existing terminal evaluations

– Including means-ends linkage assessment (e.g., economic and trade instruments – and ends – market transformation)

• Statistical methods:– Descriptive and Regression-based analyses

• Taking into account differences in time, population, GDP etc• Fieldwork (qualitative method) used to confirm and / or dis-confirm

TOC, with particular focus on ‘assumptions / risks and threats’– 4 detailed country case studies– 8 ‘lite’ country case studies: Drawing on parallel UNDP – UNEP terminal

evaluation of GEF-ODS projects – Semi-structured interviews with Government Ozone Units (20%) Private

Sector (60%), Customs and Excise (20%)– Questionnaire survey to validate key issues arising from qualitative data

collection– Triangulation between and within countries

11

Forthcoming findings

• Draft report released mid-May– International ODS impact evaluation workshop in

Tashkent: June 2009

• Now over to David to finish with a delicious ROTI

Review of Outcomes to Impacts (ROtI)

Impact Evaluation Framework

Impact

ReducedThreats to

GEB

Enhancedstatus of

GEBOutcomeOutputs State/

conditionState/

condition

Assumption Assumption Assumption

Threats Based AnalysisOutcomes-Impacts TOC AnalysisProject Logframe Analysis

Assess direct effects of the project

Assess how these effects are leading to impacts

Assess whether impacts have actually occurred

+ ++ +++

Project stakeholders create their own theory of change

How the ROtI addresses a key methodological dilemma in impact evaluation:

representation versus reliability

Reliability:Data Quality

Impact Evaluations($100K)

1

Field-based ROtIs

($10-20K) 5-10

Desk–based Rotis based on Terminal

Evaluations($400)

250

Represent-ation:

Number of projects

per $100,000

Steps in field ROtI

Results from Field Review

• Pilot in Seychelles showed that ROtI based on field review can produce detailed qualitative assessment (with some quantitative support) of achievement of outcomes and of progress from this stage towards achievement of impacts.

• Key questions could all be assessed : did assumptions hold, were intended impact drivers active and were intermediate stages achieved.

The role of the ROtI

• The desk ROtI enables a rapid scaling up of proxy information on impacts

• The field ROtI provides moderate scaling up possibilities, with higher data quality

• The Impact Evaluation provides detailed and reliable data on a few projects

• The three levels together can provide a much richer knowledge base on impacts than is currently available

19

Thank you