Fidida… 2/23.12.08. kansi 14.1.2009 13:34 Sivu 2REPORT 2008:6 The Cross-cutting hemes in the Finnish Development Cooperation ISBN: 978-951-224-714-6 (printed), ISBN: 978-951-224-715-3

REPORT 2008:6 The Cross-cutting hemes in the Finnish Development Cooperation ISBN: 978-951-224-714-6 (printed), ISBN: 978-951-224-715-3 (pdf), ISSN: 1235-7618

REPORT 2008:5 Finnish NGO FoundationsISBN: 978-951-724-709-2 (printed), ISBN: 978-951-724-710-8 (pdf), ISSN: 1235-7618

REPORT 2008:4 FIDIDA: An Examle of Outsourced Service 2004–2008ISBN: 978-951-724-690-3 (printed), ISBN: 978-951-724-691-0 (pdf), ISSN: 1235-7618

REPORT 2008:3 Evolving New Partnerships between Finland and Namibia ISBN: 978-951-724-701-6 (printed), ISBN: 978-951-724-702-3 (pdf), ISSN: 1235-7618

REPORT 2008:2 Local Cooperation Funds – Role in Institution Building of Civil Society Organizations ISBN: 978-951-724-701-6 (printed), ISBN: 978-951-724-702-3 (pdf), ISSN: 1235-7618

REPORT 2008:1 Finnish Partnership Agreement SchemeISBN: 978-951-724-672-9 (printed), ISBN: 978-951-724-673-6 (pdf), ISSN: 1235-7618

SPECIAL EDITION FAO: Utmaning till förnyelse. Sammanfattning2008:1 (SWE) ISBN: 978-951-724-670-5 (print), ISBN: 978-951-724-671-2 (pdf), ISSN: 1235-7618

SPECIAL EDITION FAO: Haasteena uudistuminen. Lyhennelmä2008:1 (FI) ISBN: 978-951-724-655-2 (painettu), ISBN: 978-951-724-659-0 (pdf), ISSN: 1235-7618

SPECIAL EDITION FAO: The Challenge of Renewal. Summary2008:1 (ENG) ISBN: 978-951-724-657-6 (printed), ISBN: 978-951-724-661-3 (pdf), ISSN: 1235-7618

REPORT 2007:3 Implementation of the Paris Declaration – FinlandISBN: 978-951-724-663-7 (printed), ISBN: 978-951-724-664-4 (pdf), ISSN: 1235-7618

REPORT 2007:2 Meta-Analysis of Development Evaluations in 2006ISBN: 978-951-724-632-3 (printed), ISBN: 978-951-724-633-1 (pdf), ISSN: 1235-7618

REPORT 2007:1 Finnish Aid to AfghanistanISBN: 978-951-724-634-7 (printed), ISBN: 978-951-724-635-4 (pdf), ISSN: 1235-7618

REPORT 2006:3 Review of Finnish Microfinance CooperationISBN: 951-724-569-6 (printed), ISBN: 951-724-570-X (pdf), ISSN: 1235-7618

REPORT 2006:2 Evaluation of CIMO North-South Higher Education Network ProgrammeISBN: 951-724-549-1, ISSN: 1235-7618

REPORT 2006:1 Evaluation of Environmental Management in Finland´s Development CooperationISBN: 951-724-546-7, ISSN: 1235-7618

REPORT 2005:6 Evaluation of Support Allocated to International Non-Govermental Organisations (INGO)ISBN: 951-724-531-9, ISSN: 1235-7618

REPORT 2005:5 Evaluation of the Service Centre for Development Cooperation in Finland (KEPA)ISBN: 951-724-523-8, ISSN: 1235-7618

REPORT 2005:4 Gender Baseline Study for Finnish Development Cooperation ISBN: 951-724-521-1, ISSN: 1235-7618

REPORT 2005:3 Evaluation of Finnish Health Sector Development Cooperation 1994–2003ISBN: 951-724-493-2, ISSN: 1235-7618

REPORT 2005:2 Evaluation of Finnish Humanitarian Assistance 1996–2004ISBN: 951-724-491-6, ISSN: 1235-7618

REPORT 2005:1 Ex-Ante Evaluation of Finnish Development Cooperation in the Mekong RegionISBN: 955-742-478-9, ISSN: 1235-7618

Fidida… 2/23.12.08. kansi 14.1.2009 13:34 Sivu 2

T REPORT 2004:4 Refocusing Finland’s Cooperation with NamibiaISBN: 955-724-477-0, ISSN: 1235-7618

REPORT 2004:3 Evaluation of the Development Cooperation Activities of Finnish NGOs and Local Cooperation Funds in TanzaniaISBN: 951-724-449-5, ISSN: 1235-7618

REPORT 2004:2 Evaluation of Finland’s Development Cooperation with Bosnia and HerzegovinaISBN: 951-724-446-0, ISSN: 1235-7618

REPORT 2004:1 Evaluation of Finnish Education Sector Development CooperationISBN: 951-724-440-1, ISSN: 1235-7618

REPORT 2003:3 Label Us Able – A Pro-active Evaluation of Finnish Development co-operation from the disability perspective ISBN 951-724-425-8, ISSN 1235-7618

REPORT 2003:2 Evaluation of Finnish Forest Sector Development Co-operationPART 2 ISBN 951-724-416-9 ISSN 1235-7618

REPORT 2003:2 Evaluation of Finnish Forest Sector Development Co-operation PART 1 ISBN 951-724-407-X, ISSN 1235-7618

REPORT 2003:1 Evaluation of the Finnish Concessional Credit SchemeISBN 951-724-400-2, ISSN 1235-7618

REPORT 2002:9 Evaluation of the Development Cooperation Activities of Finnish NGOs in KenyaISBN 951-724-392-8, ISSN 1235-7618

REPORT 2002:8 Synthesis Study of Eight Country Programme EvaluationsISBN 951-724-386-3, ISSN 1235-7618

REPORT 2002:7 Review of Finnish Training in Chemical Weapons VerificationISBN 951-724-378-2, ISSN 1235-7618

REPORT 2002:6 Kansalaisjärjestöjen Kehyssopimusjärjestelmän arviointiISBN 951-724-376-6, ISSN 1235-7618

REPORT 2002:5 Evaluation of the Bilateral Development Co-operation Programme between Kenya and FinlandISBN 951-724-373-1, ISSN 1235-7618

REPORT 2002:4 Evaluation of Bilateral Development Co-operation between Nicaragua and FinlandISBN 951-724-372-3, ISSN 1235-7618

REPORT 2002:3 Evaluation of the Bilateral Development Co-operation between Ethiopia and FinlandISBN 951-724-370-7, ISSN 1235-7618

REPORT 2002:2 Evaluation of the Bilateral Development Co-operation between Mozambique and Finland ISBN 951-724-367-7, ISSN 1235-7618

REPORT 2002:1 Evaluation of the Development Co-operation Programme between Nepal and FinlandISBN 951-724-368-5, ISSN 1235-7618

REPORT 2001:9 Evaluation of the Bilateral Development Co-operation between Finland and ZambiaISBN 951-724-365-0, ISSN 1235-7618

REPORT 2001:8 Evaluation of the Bilateral Development Co-operation between Vietnam and FinlandISBN 951-724-361-8, ISSN 1235-7618

REPORT 2001:7 Evaluation of Diesel Power Plants in Four Countries: TanzaniaISBN 951-724-356-1, ISSN 1235-7618

REPORT 2001:6 Evaluation of Diesel Power Plants in Four Countries: PeruISBN 951-724-355-3, ISSN 1235-7618
























T

REPORT 2008:7 ISBN: 978-951-724-716-0 (printed), ISBN: 978-951-724-717-7 (pdf), ISSN: 1235-7618Kosovo ountry rogramme C P REPORT 2004:4 Refocusing Finland’s Cooperation with Namibia

ISBN: 955-724-477-0, ISSN: 1235-7618











































T


















TREPORT 2004:4 Refocusing Finland’s Cooperation with Namibia

ISBN: 955-724-477-0, ISSN: 1235-7618

















Evaluation of the Bilateral Development Co-operation between Finland and Zambia

REPORT 2005:1 Ex-Ante Evaluation of Finnish Development Cooperation in the Mekong RegionISBN: 955-742-478-9, ISSN: 1235-7618REPORT 2008:7

ISBN: 978-951-724-716-0 (printed), ISBN: 978-951-724-717-7 (pdf), ISSN: 1235-7618Kosovo ountry rogramme C P


ISBN: 978-951-724-7 - (printed), ISBN: 978-951-724-7 - (pdf), ISSN: 1235-7618Finland´s Development Cooperation in Central Asia and South Caucasus

28 3 29 0REPORT 2009 :1

ISBN: 978-951-724-7 - (printed), ISBN: 978-951-724- - (pdf), ISSN: 1235-7618Agriculture and Rural Development. Preliminary Study

46 7 747 4REPORT 2009 :2



ISBN: 978-951-724-7 - (printed), ISBN: 978-951-724- - (pdf), ISSN: 1235-76187

Development Research55 9 56 6

Support to REPORT 2009 :3

ISBN: 978-951-724-7 - (printed), ISBN: 978-951-724- - (pdf), ISSN: 1235-7618Agriculture and Rural Development. A Preliminary Study

46 7 747 4REPORT 2004:4 Refocusing Finland’s Cooperation with Namibia

ISBN: 955-724-477-0, ISSN: 1235-7618





















REPORT 2009 :2























TREPORT 2008:6 The Cross-cutting hemes in the Finnish Development Cooperation ISBN: 978-951-224-714-6 (printed), ISBN: 978-951-224-715-3 (pdf), ISSN: 1235-7618






















T



Support to REPORT 2004:4 Refocusing Finland’s Cooperation with NamibiaISBN: 955-724-477-0, ISSN: 1235-7618





















REPORT 2009 :3























T

ISBN: (printed), ISBN: 978-951-724- - (pdf), ISSN: 1235-7618Meta-analysis of Development Cooperation on HIV

978-951-724-769-6 770 2REPORT 2004:4 Refocusing Finland’s Cooperation with Namibia

ISBN: 955-724-477-0, ISSN: 1235-7618





















REPORT 2009 :4 / AIDS

ISBN: (printed), ISBN: 978-951-724- - (pdf), ISSN: 1235-7618978-951-724-7 - 7DEMO Finland Development Programme V


ISBN: 955-724-477-0, ISSN: 1235-7618





















REPORT 2009 : 6

REPORT 2009:5 Finnish Aid in Western Kenya; Impact and Lessons Learned ISBN: 978-951-724-783-2 (printed), ISBN: 978-951-724-786-3 (pdf), ISSN: 1235-7618

REPORT 2009:7 The North-South-South Higher Education Network Programme ISBN: 978-951-724-790-0 (printed), ISBN: 978-951-724-791-7 (pdf), ISSN: 1235-7618

REPORT 2009:8 Natural Disasters, Climate Change and Poverty ISBN: 978-951-724-807-5 (printed), ISBN: 978-951-724-808-2 (pdf), ISSN: 1235-7618













































T

REPORT 2009:9 Meta-Analysis of Development Evaluations in 2007 and 2008 ISBN: 978-951-724-809-9 (printed), ISBN: 978-951-724-810-5 (pdf), ISSN: 1235-7618

1Meta-analysis of Evaluations 2007-2008

Evaluation

Meta-analysis of Development Evaluations in 2007 and 2008

Evaluation report 2009:9

2 Meta-analysis of Evaluations 2007-2008


Evaluation


Paula J. WilliamsMaaria Seppänen

Evaluation report 2009:9

MINISTRY FOR FOREIGN AFFAIRS OF FINLAND______________________

This evaluation was commissioned by Ministry for Foreign Affairs of Finland to Impact Consulting Oy Ltd. The Consultants bear the sole responsibility for the contents of the report. The report does not necessarily reflect the views of the

Ministry for Foreign Affairs in Finland.


This report can be accessed at http://formin.fi and hard copies can be requested from [email protected] Evaluation (EVA-11)The Ministry for Foreign Affairs of FinlandP.O. Box 519FI-00023 GOVERNMENTFinland

ISBN 978-951-724-809-9 (printed) ISBN 978-951-724-810-5 (pdf)ISSN 1235-7618Cover design: Anni PalotiePrinting house: Hakapaino Oy, Helsinki, 2009

Anyone reproducing the content or part of the content of the report should acknowledge the source. Proposed reference: Williams P & Seppänen M 2009 Meta-analysis of Development Evaluations in 2007 and 2008. Evaluation report 2009:9. Ministry for Foreign Affairs of Finland, Hakapaino Oy, Helsinki, 85 p. ISBN 978-951-724-809-9.


PREFACE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiACRONYMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .vABSTRACTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

Finnish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1Swedish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2English . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

SUMMARIES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4Finnish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4Swedish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10English . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 1.1 Finnish Development Policies and Development Cooperation . . . . . . . .251.2 Evaluation of Finnish Development Cooperation . . . . . . . . . . . . . . . . . .291.3 Comparative Analyses of Development Evaluations . . . . . . . . . . . . . . . .30

2 APPROACH, METHODOLOGY AND LIMITATIONS . . . . . . . . . . . . . . . . .322.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .322.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 2.3 Limitations of this Meta-analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35

3 CURRENT PRACTICE IN FINNISH DEVELOPMENT EVALUATION . .36 4 OVERVIEW OF THE EVALUATION REPORTS . . . . . . . . . . . . . . . . . . . . . .385 QUALITY OF THE EVALUATION REPORTS . . . . . . . . . . . . . . . . . . . . . . . .42

5.1 Quality of the Terms of References of the Evaluation Reports . . . . . . .42 5.2 Quality of the Evaluation Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44

6 QUALITY OF THE AID INTERVENTIONS . . . . . . . . . . . . . . . . . . . . . . . . .516.1 Overall Rating of the Aid Interventions. . . . . . . . . . . . . . . . . . . . . . . . . . .51 6.2 Relevance, Policy Coherence and Complementarity. . . . . . . . . . . . . . . . . .52

6.2.1 Relevance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .536.2.2 Policy Coherence and Complementarity . . . . . . . . . . . . . . . . . .54

6.3 Effectiveness and Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .556.3.1 Effectiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .556.3.2 Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57

6.4 Sustainability and Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .576.4.1 Sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .586.4.2 Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62

6.5 Finnish Added Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62 6.6 Baseline Data, Indicators and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . .656.7 Overall Quality of the Aid Interventions . . . . . . . . . . . . . . . . . . . . . . . . .676.8 Comparison by Region and Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68

7 IMPLEMENTATION OF THE DEVELOPMENT POLICY PROGRAMME . .68 7.1 Status of Implementation of Interventions in this Sample . . . . . . . . . . . 68

CONTENTS

i


7.2 Key Elements of Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .708 CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71

8.1 Trends over Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 718.2 Broadening of the Use of Development Evaluations . . . . . . . . . . . . . . . . 74

9 RECOMMENDATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78

ANNEX 1 TERMS OF REFERENCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81ANNEX 2 PEOPLE CONSULTED 1)

ANNEX 3 THE EVALUATED REPORTS BY AUTHOR 1)

ANNEX 4 INTERVIEW GUIDE 1)

ANNEX 5 NARRATIVE QUALITATIVE EVALUATION OF REPORT 1)

ANNEX 6 EVALUATION REPORT ASSESSMENT FORM 1)

ANNEX 7 EVALUATION CRITERIA FOR AID QUALITY 1) ANNEX 8 EVALUATION TEAM 1)

1) Annexes 2-8 are non-edited and contained in the attached CD

TABLESTable 1 Meta-analysis Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23Table 2 Aid Instruments and Types of Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . 39Table 3 Criteria Specified in the TOR for Assessing the Projects and Programmes . . . . 43Table 4 Overall Evaluation Report Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48Table 5 Ratings of Aid Interventions by Evaluation Criteria . . . . . . . . . . . . . . . . . . 52Table 6 Ratings of Aid Interventions by Relevance and Policy Coherence. . . . . . . 53Table 7 Ratings of Aid Interventions by Effectiveness and Efficiency . . . . . . . . . . 55Table 8 Ratings of Aid Interventions by Sustainability and Impact. . . . . . . . . . . . . 58Table 9 Ratings of Aid Interventions by Finnish Added Value . . . . . . . . . . . . . . . . 63Table 10 Ratings of Aid Interventions by Baseline Data, Indicators and Monitoring . . . 65Table 11 Ratings of Aid Interventions by Overall Aid Quality . . . . . . . . . . . . . . . . 67Table 12 Comparison of Report Quality and Aid Quality by Geographical Region . and Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68

FIGURESFigure 1 Key Elements of the Meta-analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Figure 2 Analysis of Realization of Policy Principles . . . . . . . . . . . . . . . . . . . . . . . 33Figure 3 Summary Example of Evaluation Report Quality Rating . . . . . . . . . . . . . 34Figure 4 Summary Example of Project or Programme Quality Rating . . . . . . . . . 35Figure 5 Reports by Country, Region and Partner Status . . . . . . . . . . . . . . . . . . . . . 40Figure 6 Sectors of Development Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

ii


PREFACE

In the Ministry for Foreign Affairs of Finland the evaluation function is divided into the centralised and decentralised evaluations. The former are larger thematic or sectoral evaluations carried out by the Development Evaluation (EVA-11) which is attached to the office of the Under-Secretary of State for development. The latter regard individual projects and they are done by the regional departments and units and sometimes also by the embassies. Thus, the Ministry for Foreign Affairs of Finland commissions a large number of evaluations, mid-term reviews (MTR) and other assessments of aid.

The overall purpose of meta-analyses is to offer an opportunity to learn from past experience and to extract good practices for quality development of development cooperation through assessing the quality of development evaluations and the development cooperation itself as well as assessing how the Finnish development policy has been operationalised.

There has been two meta-analyses done prior to the current Meta-analysis and they covered both centralised and decentralised evaluations. The first one was carried out in 1996 covering years 1988-1995 (150 reports) but focusing only on assessing the quality of aid. The second Meta-analysis assessed the quality of evaluations as well as quality of aid and it was carried out in 2007 covering the year 2006 (29 reports).

The Meta-analysis at hand assessed the quality of evaluations and the quality of aid focusing particularly on sustainability and effectiveness. It covered the decentralised evaluations (33 reports) only.

Overall, the reports and the interventions assessed in this Meta-analysis were found to be of good, although often mixed, quality – having some notable achievements, but also noted of areas needing further improvement. Compared with the two previous meta-analyses, this study found improvements in the quality of evaluation reports.

However, the current Meta-analysis identified the need for greater attention to be given to issues of sustainability and impacts, data and monitoring, and the standardising and sharing information derived from evaluations to promote greater learning within the Ministry and the larger development community.

Helsinki, 30 December 2009

Aira PäivökeDirectorDevelopment Evaluation

iii


iv


ACRONYMS

DAC Development Assistance Committee [of OECD]€ Currency EuroEC European CommissionEFA Education for AllEU European UnionEVA-11 MFA Development EvaluationFAO Food and Agriculture Organization of the United NationsFED Multi-donor Fund for Equality and Sexual and Reproductive

Rights, NicaraguaGIS Geographical Information SystemGNI Gross National IncomeHIV/AIDS Human Immunodeficiency Virus /Acquired Immune

Deficiency Syndrome ICI Institution Cooperation Instrument ICT Information and Communication TechnologyIFI International Financing InstitutionsINGO International Non-governmental OrganisationLCF Local Cooperation FundsMDGs Millennium Development GoalsMFA Ministry for Foreign Affairs of FinlandMTR Mid-Term Review (Mid-Term Evaluation Report)NGO Non-governmental OrganisationODA Official Development AssistanceOECD Organisation for Economic Co-operation and Development TA Technical AssistanceTOR Terms of ReferenceUM Ulkoasiainministeriö / Utrikesministeriet [Ministry for Foreign

Affairs of Finland]UN United NationsUNDP United Nations Development ProgrammeUNIDO United Nations Industrial Development OrganizationUS$ Currency United States Dollar WASH Water Supply, Sanitation and HygieneWB World Bank

v

10 Meta-analysis of Evaluations 2007-2008vi


Meta-analyysi Kehitysevaluaatioista vuosilta 2007 ja 2008

Paula J. Williams ja Maaria Seppänen

Ulkoasiainministeriön evaluointiraportti 2009:9

ISBN 978-951-724-809-9 (painettu); ISBN 978-951-724-810-5 (pdf); ISSN 1235-7618

Raportti on luettavissa kokonaisuudessaan http://formin.finland.fi

_________________________________

TIIVISTELMÄ

Meta-analyysissä tutkittiin 33 etukäteisarvion ja evaluaatioraportin otos vuosina 2007 ja 2008 toteutetuista evaluaatioista. Ne koskivat 36 Suomen ulkoasiainministeriön (UM) tukemaa hanketta ja kehitysohjelmaa. Tavoitteena oli vetää yhteen evaluaati-oiden opetuksia, löytää suositeltavia hyviä käytäntöjä sekä saattaa nämä laajemmin suuren yleisön saataville.

Arvioinnissa analysoitiin 1) evaluaatioraporttien laatu ja niiden toimeksiannot (TOR), 2) kehityshankkeiden laatu ja 3) vuoden 2007 kehityspoliittisen ohjelman toimeenpano erityisesti tuloksellisuuden ja kestävyyden suhteen. Meta-analyysissä evaluaatioraportit pisteytettiin kymmenen taloudellisen yhteistyön ja kehityksen järjestön (Organisation for Economic Co-operation and Development, OECD) kehitysapukomitean (Deve-lopment Assistance Committee, DAC) evaluaation laatukriteerin mukaan. Kehitys-hankkeet pisteytettiin OECD/DAC:n viiden evaluaatiokäsitteen mukaan (tarkoituk-senmukaisuus, tuloksellisuus, tehokkuus, vaikutuksellisuus ja tulosten kestävyys) sekä kolmen lisäkriteerin mukaan (politiikkajohdonmukaisuus ja täydentävyys; suomalai-nen lisäarvo; ja seuranta tiedonhallintojärjestelmineen).

Yleisesti ottaen raporttien ja hankkeiden laatu todettiin hyväksi, tosin epätasaiseksi niin, että mukaan mahtui merkittäviä saavutuksia mutta myös selviä parannusta vaativia koh-tia. Useimmat otokseen kuuluvat hankkeet oli aloitettu ennen vuonna 2007 hyväksyttyä uutta kehityspoliittista ohjelmaa, mutta otokseen kuuluneiden etukäteisarvioiden koh-teista voidaan päätellä, että kehitysyhteistyö on siirtymässä kohti ympäristöalan hankkei-ta. Verrattuna kahteen aikaisempaan meta-analyysiin nyt raporteista löydettiin suurempi huomio evaluaation avainkäsitteille ja yleinen raporttien laadun paraneminen.

Meta-analyysi totesi, että kehityshankkeissa täytyy ottaa paremmin huomioon kestä-vyys ja vaikutuksellisuus sekä seurantajärjestelmän kehittäminen. Myös evaluaatioiden käyttöä tule tehostaa ja tiedon levittämistä edistää, jotta UM ja laajempi kehitysyhteis-työstä kiinnostunut yleisö voi paremmin oppia niistä.

Avainsanat: kehitysyhteistyön evaluaatio, etukäteisarvio, meta-analyysi, kehityspolitiikka


Metaanalys av Utvecklingsutvärderingar under 2007 och 2008

Paula Williams och Maaria Seppänen

Utrikesministeriets utvärderingsrapport 2009:9

ISBN 978-951-724-809-9 (print); ISBN 978-951-724-810-5 (pdf); ISSN 1235-7618

Rapporten finns i sin helhet på addressen http://formin.finland.fi

_________________________________

ABSTRAKT

I denna metaanalys undersöktes ett sampel på 33 bedömnings- och utvärderingsrap-porter som utförts under 2007 och 2008 för 36 projekt och program som understötts av utrikesministeriet. Syftet är att ta lärdom av färsk utvärderingserfarenhet, fastställa god praxis och göra denna lärdom mer allmänt tillgänglig.

Bedömningen analyserade: 1) kvaliteten på utvärderingsrapporterna samt deras respek-tive uppdragsbeskrivningar, 2) kvaliteten på hjälpinsatserna och 3) genomförandet av utvecklingspolitiken från 2007 i synnerhet vad gäller främjandet av en effektiv och hållbar utveckling. I metaanalysen jämfördes kvaliteten på utvärderingsrapporterna med tio av OECD/DAC:s kvalitetsstandarder för utvärdering. Projekten och programmen bedöm-des med hjälp av de fem OECD/DAC-kriterierna för utvärdering av relevans, effektivitet, verkningsgrad, inverkan och hållbarhet samt tre ytterligare kriterier för politisk koherens och komplementaritet; finländskt mervärde; och data, indikatorer och uppföljande.

På det hela taget fann man rapporterna och insatserna vara av god kvalitet om även ofta blan-dad. Det förekom anmärkningsvärda prestationer och vissa områden som fordrar vidare för-bättring. De flesta insatser som utvärderats i denna metaanalys är daterade innan Statsrådets principbeslut på utvecklingspolitiskt program från oktober 2007 fattades, men senare bedöm-ningar återspeglar övergången till alltmer miljömässigare insatser. I jämförelse med två tidigare metaanalyser konstaterade denna undersökning att en ökad uppmärksamhet hade fästs vid centrala utvärderingskriterier och förbättringar av utvärderingsrapporterna.

Denna metaanalys fastställer behovet av att fästa större uppmärksamhet vid frågor gällande hållbarhet och inverkan, data och uppföljning, samt standardiseringen och förmedlandet av informationen från utvärderingar för att främja kunskapen inom ministeriet och den större utvecklingsgemenskapen.

Nyckelord: utvärdering, bedömning, metaanalys, utvecklingspolitik



Paula J. Williams and Maaria Seppänen

Evaluation Report of the Ministry for Foreign Affairs of Finland 2009:9

ISBN 978-951-724-809-9 (printed); ISBN 978-951-724-810-5 (pdf); ISSN 1235-7618

The full report can be accessed at http://formin.finland.fi

_________________________________

ABSTRACT

This Meta-analysis examined a sample of 33 appraisal and evaluation reports con-ducted in 2007 and 2008 of 36 projects and programmes supported by the Ministry for Foreign Affairs of Finland (MFA). The purpose is to draw lessons learned from recent evaluation experience, identify good practices, and make those lessons more widely available.

The assessment analysed: 1) the quality of the evaluation reports and their respective Terms of Reference (TOR), 2) the quality of the aid interventions, and 3) implemen-tation of the 2007 development policy, particularly in terms of promoting effective and sustainable development. The Meta-analysis rated the evaluation report quality against ten quality standards of the Organisation for Economic Co-operation and Development’s Development Assistance Committee (OECD/DAC). The projects and programmes were assessed using the five OECD/DAC evaluation criteria of relevance, effectiveness, efficiency, impact, and sustainability, and three additional cri-teria of policy coherence and complementarity; Finnish added value; and data, indi-cators, and monitoring.

Overall, the reports and the interventions are of good, although often mixed, quality – having some notable achievements, and other areas needing further improvement. Most interventions assessed pre-date the October 2007 development policy, but more recent appraisals reflect the shift towards more environmental interventions. Compared with two previous meta-analyses, this study found increasing attention to key evaluation criteria and improvements in evaluation reports.

This Meta-analysis identifies the need for greater attention to be given to sustaina-bility and impacts, data and monitoring, and standardising and sharing information from evaluations to promote greater learning within the MFA and larger development community.

Keywords: evaluation, appraisal, Meta-analysis, development policy


YHTEENVETO

Vuosina 2007 ja 2008 toteutettujen kehitysyhteistyön evaluaatioiden meta-analyysiUlkoasiainministeriön (UM) kehitysyhteistyön evaluaatio (EVA-11) on teettänyt toi-meksiantona yhteenvetoanalyysin vuosina 2007 ja 2008 ministeriön eri alue- ja muiden yksikköjen toteuttamasta kolmestatoista etukäteisarviosta ja kahdestakymmenestä evaluaatioraportista. EVA-11:n teettämät evaluaatiot eivät kuuluneet tarkastelun otok-seen. Meta-analyysi keskittyi arvioimaan evaluaatioraporttien laatua, niissä kuvattujen kehitysyhteistyöhankkeiden laatua sekä Suomen vuonna 2007 hyväksymän kehitys-poliittisen ohjelman toimeenpanoa. Erityisesti kiinnitettiin huomiota hankkeiden ja ohjelmien tuloksellisuuteen ja kestävyyteen, sekä niiden onnistumiseen vaikuttaneisiin tai onnistumista haitanneisiin tekijöihin.

Vuoden 2007 kehityspoliittinen ohjelmaSuomen hallitus hyväksyi lokakuussa 2007 uuden kehityspoliittisen ohjelman Deve-lopment Policy Programme 2007: Towarsds a Sustainable and Just Word Community (Ministry for Foreign Affairs of Finland 2007a), joka painottaa kestävää kehitystä ja köyhyyden vähentämistä. Kestävän kehityksen katsotaan siinä koostuvan kolmesta toisiaan tuke-vasta ulottuvuudesta – taloudellinen kehitys, yhteiskunnallinen kehitys ja ympäristön-suojelu. Ohjelma tähtää myös kehityspolitiikan johdonmukaisuuteen, täydentävyyteen ja tuloksellisuuteen sekä läpileikkaavien teemojen huomioon ottamiseen kehitysyh-teistyöhankkeissa. Ohjelma painottaa Suomen pyrkimystä harjoittaa kehitysyhteistyö-tä siellä, missä Suomi voi tarjota lisäarvoa asiantuntemuksensa tai muiden suhteellista etua tarjoavien tekijöiden ansiosta.

Yleiskatsaus ulkoministeriön evaluaatiokäytäntöön Suomen ulkoasiainministeriö kuuluu taloudellisen yhteistyön ja kehityksen järjestön (Organisation for Economic Co-operation and Development, OECD) kehitysapu-komitean (Development Assistance Committee, DAC) evaluaatioverkostoon ja on sitoutunut noudattamaan kansainvälistä käytäntöä kehitysyhteistyössään, mukaan lu-kien ennakkoarvioiden ja evaluaatioiden toteuttaminen. Suunniteltujen hankkeiden ja kehitysohjelmien ennakkoarvioinnista ovat vastuussa alueyksiköissä maavastaavat ja/tai edustustojen henkilökunta yhdessä toimialakohtaisten neuvonantajien sekä muiden oleellisten ministeriön tahojen kanssa. Niiden tehtävänä on myös väliarviointien ja loppuevaluaatioiden toteuttaminen.

Meta-analyysi

Tavoitteet Meta-analyysin tavoitteena on vetää yhteen kuluneina vuosina toteutettujen evaluaa-tioiden johtopäätöksistä poimittavissa olevat opetukset, etsiä suositeltavia käytäntöjä sekä saattaa nämä yleisölle helposti saavutettavissa olevaan muotoon. Tarkastelun kes-kiössä on evaluaatioraporttien ja kehitysyhteistyöhankkeiden laatu suhteessa OECD/DAC:n arviointiperiaatteisiin ja evaluaatiokäsitteisiin. Meta-analyysissä on myös tar-


kasteltu uuden kehityspoliittisen ohjelman toimeenpanoa, erityisesti toteutettujen hank-keiden kestävyyttä ja tuloksellisuutta sen perusteella, mitä raportit niistä kertovat. Tarkastelun kohdeUM:n EVA-11 kokosi meta-analyysiä varten joukon evaluaatioraportteja ministeriön eri osastoista ja yksiköistä. Kaikki ne olivat ulkopuolisten konsulttien toimeksiantoina toteuttamia riippumattomia evaluaatioita lukuun ottamatta yhtä, jonka laatimisessa oli ulkopuolisen konsultin lisäksi ollut mukana UM:n toimialakohtainen neuvonanta-ja.

Meta-analyysin otos kattoi 33 raporttia. Kolmetoista niistä oli etukäteisarviointeja (appraisals). Kaksikymmentä raporttia koski puoliväliarviointeja (mid-term review), päät-tyneiden hankkeiden loppu- tai jälkievaluaatioita tai muita toteutukseen liittyneitä arviointeja. Näistä kahdestakymmenestä yksi evaluaatioraportti käsitteli kahta sek-toriohjelmaa, ja yksi tarkasteli kolmea eräällä maantieteellisellä alueella kolmessa eri maassa toteutettua hanketta. Näin ollen meta-analyysin otos käsitti 33 raporttia 36 hankkeesta tai ohjelmasta.

Tarkastelun kohteena olleet raportit käsittelivät 36 hanketta yhteensä 24 maassa. Mel-kein puolet niistä oli Suomen pitkäaikaisia yhteistyömaita ja yksi kolmasosa väkivaltai-sista kriiseistä toipuvia, tai niiden kourissa edelleen olevia maita tai alueita. Raporteista viisi oli kirjoitettu espanjaksi, 27 englanniksi ja yksi suomeksi. Raportit tarkastelivat laajaa kirjoa rahoitusinstrumentteja. Mukana oli joitakin kansainvälisten järjestöjen kautta toteutettavia hankkeita ja useiden avunantajien yhteisaloitteita tai sektoriohjel-mia. Yksi evaluaatio käsitteli paikallisen yhteistyön määrärahaa, yksi raportti koski kor-kotukiluottoa ja viisi kansainvälisten kansalaisjärjestöjen hankealoitteita. 37 prosenttia hankkeista sijoittui ympäristö- ja luonnonvarahallinnon sektorille ja 31 prosenttia kä-sitteli koulutusta ja tutkimusta.

Menetelmät Meta-analyysi toteutettiin kirjallisena dokumentaatioanalyysinä. Raportit tutkittiin ja arvioitiin suhteessa toimeksiannossa lueteltuihin kysymyksiin sekä kriteereihin, jotka puolestaan perustuivat OECD/DAC:n evaluaatiokäsitteisiin ja saman järjestön stan-dardeihin, joilla evaluaatioiden laatua arvioidaan.

Evaluaatioraporttien laadun arvioimisessa käytettiin kymmentä OECD/DAC:n eva-luaation laatustandardia: 1) evaluaation tarkoitusperä, tavoite ja perustelut, 2) evalu-aation laajuus, 3) konteksti, 4) menetelmät, 5) tietolähteet, 6) riippumattomuus, 7) evaluaation eettisyys, 8) laadunvarmistus, 9) evaluaatiotulosten tarkoituksenmukaisuus ja 10) kattavuus. Näiden laatustandardien perusteella laskettiin kullekin raportille ar-vosana. Tämän lisäksi meta-analyysi suoritti raporteista laadullisen arvion, jossa huo-mioitiin erityisiä tai muuten huomion arvoisia elementtejä kehitysaloitteista sekä niitä kuvaavista raporteista.

Kehityshankkeiden (projektien ja ohjelmien) laadun arvioimiseksi meta-analyysi pis-


teytti apualoitteet viiden OECD/DAC:n evaluaatiokäsitteen mukaan (tarkoituksen-mukaisuus, tuloksellisuus, tehokkuus, vaikutuksellisuus ja kestävyys), minkä lisäksi ne arvioitiin kolmen muun kriteerin suhteen: 1) politiikkajohdonmukaisuus ja täydentä-vyys, 2) suomalainen lisäarvo, 3) seurantamenetelmät ja seurantaindikaattorit mukaan lukien lähtötilanteen kartoitus (baseline). Näiden kahdeksan kriteerin perusteella lasket-tiin hankkeille laatuarvosana. Dokumenttianalyysin lisäksi haastateltiin UM:n henkilö-kuntaa Helsingissä, joitakin evaluaatioita suorittaneita henkilöitä ja yhden edustuston neuvonantajia, sekä käytettiin lisäaineistona internetistä löytyviä julkisia asiakirjoja.

Havainnot

Raporttien laatu Meta-analyysi pyrki arvioimaan toimeksiantoon kuuluvat raportit OECD/DAC:n kymmenen evaluaation laatustandardin mukaan. Tämä osoittautui hyvin vaikeaksi, koska raporteissa ei aina tuotu esiin seikkoja, joita olisi tarvittu analyysiä varten. Osa kyseisestä informaatiosta on mahdollisesti evaluaatioita teettäneiden virkamiesten tie-dossa, mutta ei meta-analyysin suorittaneen tiimin ulottuvilla.

Meta-analyysi totesi evaluaatioraporttien laadun yleisesti ottaen hyväksi OECD:n evaluaatioiden laatustandardien mukaan. Neljä raporttia arvioitiin erittäin hyväksi, seitsemän hyvästä erittäin hyväksi, ja kaksi huonoksi. Erittäin hyvät raportit täyttivät laatukriteerit suunnilleen kokonaisuudessaan, kun taas hyviksi katsottiin raportit, jotka täyttivät laatukriteerit enimmäkseen. Huonoiksi arvioitiin raportit, jotka täyttivät laatu-kriteerit vain osittain. Enimmäkseen raportit vastasivat kohtuullisen hyvin toimeksian-toihinsa. Joissakin tapauksissa raportit ylittivät toimeksiannoissa esitetyt vaatimukset kuten silloin, kun toimeksianto ei ollut tuonut esiin OECD/DAC:n evaluaatiokäsittei-tä, kun taas raportissa niitä oli käytetty kehityshankkeen analysoinnissa.

Yhdestätoista parhaat arvosanat saaneesta raportista suurin osa liittyi evaluaatioihin, jotka oli teetetty yhteistyössä muiden rahoittajien kanssa. Useimmiten kyse oli laa-joista, monta avunantajaa käsittävistä sektoriohjelmista tai kansainvälisten järjestöjen kanssa toteutettujen hankkeiden evaluaatioista, joita varten oli käytössä huomatta-va määrä resursseja mitä tulee konsulttien ihmistyöpäiviin ja avunantajien hallinnon panokseen kokonaisuudessaan. Toisaalta otoksessa oli joitakin erittäin hyviä pienin resurssein kahdenvälisistä hankkeista toteutettuja evaluaatioita, joiden tekijät olivat erinomaisesti perillä evaluaation kohteena olevasta aiheesta ja/tai maasta ja jotka toi-vat erinomaisesti esiin hankkeen erityispiirteet toimintaympäristössään sekä osoittivat hyvää asiantuntemusta Suomen kuten myös paikallisesta kehityspolitiikasta.

Kehityshankkeiden laatu Evaluaatioraporttien kohteena olevan 36 hankkeen osalta yleisarvio oli hyvä kahdek-san evaluaatiokriteerin suhteen. Kolme hanketta pisteytettiin erittäin hyväksi, kaksi hyvästä erittäin hyväksi, 23 hyväksi, kaksi hyvän ja huonon välillä olevaksi ja viisi huo-noksi. Erittäin hyviksi arvioitiin hankkeet, jotka olivat saaneet arvioinneissa korkeat arvosanat viiden OECD:n evaluaatiokäsitteen (tarkoituksenmukaisuus, tuloksellisuus,


tehokkuus, kestävyys ja vaikutuksellisuus) suhteen sekä näiden lisäksi suhteessa poli-tiikkajohdonmukaisuuteen, tiedonhallintaan ja seurantakäytäntöön. Vähemmän hyvin sijoittuivat sen sijaan hankkeet, joissa oli arviointien mukaan vaikeuksia toimeenpanos-sa, mikä johti alentuneeseen tehokkuuteen ja tuloksellisuuteen sekä haittasi tulosten kestävyyden ja vaikutuksellisuuden saavuttamista. Huonoiksi arvioiduista hankkeista yksi oli ollut etukäteisarvioinnin kohteena, minkä jälkeen hankesuunnitelma oli uusit-tu ja parannettu ja rahoitus hyväksytty, ja hanketta toimeenpannaan parhaillaan. Eva-luaatiokäsitteistä korkeimmat arviot olivat raporteissa saaneet tarkoituksenmukaisuus ja politiikkajohdonmukaisuus, kun taas heikoimmiksi arvioitiin tehokkuus ja tiedon-hallinta ja seuranta. Kestävyys ja tuloksellisuus arvioitiin raporteissa yleensä hyviksi. Vain neljäsosa raporteista oli arvioinut ja pohtinut suomalaista lisäarvoa hankkeissa.

Raportit käsittelivät kestävyyttä syvällisemmin. Yleisin tarkastelu liittyi institutio-naaliseen ja taloudelliseen kestävyyteen, vaikkakin jotkut raportit ottivat huomioon laajemman kirjon kestävyyteen liittyviä tekijöitä. Raporteissa vallitsi jonkinlainen yh-teisymmärrys siitä, että hyvää kestävyyttä edistävät vahva paikallinen omistajuus, po-liittinen tahto, hallituksen sitoutuminen tavoitteisiin, hyödynsaajien osallistuminen ja voimallistaminen, kapasiteettien kehittäminen, sopivien politiikkakehyksien luominen ja rahoituksen jatkuvuus. Toisin päin ottaen näiden tekijöiden puuttumista pidettiin yleensä kestävyyttä haittaavana vaikutuksena tai suorastaan riskinä. Vain kaksi raport-tia viittasi riskianalyysiin, ja kaksi muuta raporttia mainitsi ajatuksen kehittää kestä-vyyssuunnitelmia (Sustainability Action Plans) hankkeita varten.

Vuoden 2007 kehityspoliittisen ohjelman toimeenpano Vuosina 2007 ja 2008 toteutettujen evaluaatioiden perusteella voidaan vetää johto-päätöksiä kehityspoliittisen ohjelman (Ministry for Foreign Affairs of Finland 2007a) toimeenpanosta vain rajoitetusti. Kaksitoista evaluaatiota oli tehty ennen lokakuuta 2007, jolloin uusi kehityspoliittinen ohjelma virallisesti hyväksyttiin hallituksen peri-aatepäätökseksi. Kaikkien muiden paitsi kuuden hankkeen suunnittelu oli alkanut ai-kaisemman kehityspoliittisen ohjelman aikana. Jotkin vuosina 2007 ja 2008 evaluoidut hankkeet ovat sittemmin jatkuneet uudella vaiheella tai seurantahankkeella. Meta-ana-lyysin tietoon ei kuitenkaan tullut, oliko uusi kehityspoliittinen ohjelma prioriteettei-neen vaikuttanut niiden toteutuksen muotoon tai suunnitteluun. Tarkastelun kohtee-na ollut otos evaluaatioista ilmaisee, että sektorikohtainen siirtymä on tapahtumassa kohti ympäristöä ja luonnonvaroja käsitteleviä hankkeita ja ohjelmia.

Vertailu aikaisempiin meta-analyyseihin Vuosina 1988–1995 toteutettujen evaluaatioiden yhteenvetotutkimus (Koponen J & Mattila-Wiro P 1996) ja vuoden 2006 meta-analyysi (White P & Stenbäck T 2007) oli-vat tulleet moniin samoihin havaintoihin kuin käsillä oleva meta-analyysi liittyen eva-luaatioraporttien ja kehityshankkeiden laatuun. On kuitenkin havaittavissa myös tiet-tyä laadun parantumista niin evaluaatioraporteissa kuin apualoitteissakin. Erityisesti huomattiin, että kestävyys, politiikkajohdonmukaisuus ja täydentävyys sekä lisääntynyt tuloksellisuus, mukaan lukien suomalainen lisäarvo, saavat yhä enemmän huomiota raporteissa. Tarvitaan kuitenkin lisää parannusta tiedonhallinnan ja seurannan sekä


vaikutuksellisuuden ja tehokkuuden suhteen. Evaluaatioissa painotetaan edelleen nii-den suppeaa roolia hankehallinnon apuvälineinä, ja vaaditaan lisätoimenpiteitä, jotta niiden potentiaali laajempaan hyötyyn oppivan organisaation välineinä ja julkishallin-non avoimuuden ja vastuuvelvollisuuden keinoina toteutuisi.

Johtopäätökset ja suosituksetYleisenä johtopäätöksenä voidaan todeta, että Suomen kehitysyhteistyö saa aikaa muutosta, mutta vielä on tilaa parannuksille niin yksittäisten hankkeiden kuin myös hankehallintojärjestelmän kannalta. Evaluaatioilla on merkittävä rooli laadun paranta-misessa, ja niinpä onkin tärkeää parantaa evaluaatioraporttien tasoa ja käyttää niiden antamaa informaatiota järjestelmällisemmin ja strategisemmin Suomen kehitysyhteis-työn laadun kohentamiseksi. Meta-analyysin perusteella voidaan suositella seuraavaa:

1. Tulevien meta-analyysien, tai muiden evaluaatiotulosten tiedonlevittämisen keinojen helpottamiseksi olisi tärkeää sopia evaluaatioissa käytettävistä stan-dardeista ja sitten kehittää välineitä niiden noudattamiseksi, ja kouluttaa eva-luaatioita hallinnoiva henkilökunta niiden käyttöön. Vaikka yleisesti ottaen evaluaatioita on käytetty hyvin kehitysyhteistyön hankehallinnon välineitä, niiden käyttöä potentiaalisena oppimisen ja julkisen hallinnon avoimuuden ja vastuuvelvollisuuden välineinä voisi vielä paljonkin parantaa. UM:n tuli-si määritellä minimistandardit niille asioille, jotka kussakin evaluaatiossa on tuotava esiin, mutta samalla tulisi sallia tietty joustavuuden taso kunkin yk-sittäisen toimeksiannon erityiskysymyksien huomioon ottamiseksi. Tämän lisäksi UM:n tulee vahvistaa arkistointiansa ja tiedonhallintajärjestelmäänsä, jotta evaluaatioiden esiin tuomat asiat tulevat helposti saataville.

2. Vaikka evaluaatiot olivat yleisesti ottaen hyvälaatuisia, niitä voisi edelleen kohentaa. Avaintekijä tässä pyrkimyksessä on toimeksiantojen (TOR) laatu. Niissä tulisi ottaa esiin kaikki evaluaatiokäsitteet ja arviointiperiaatteet sekä käyttää enemmän resursseja evaluaatioihin niin hallinnon ajankäytön kuin myös konsulttityöpäivien suhteen.

3. Kehityshankkeiden laadun parantamiseksi tarvitaan suurempaa huomiota seurantajärjestelmän kehittämiselle mukaan lukien baseline-alkutilanneana-lyysin tekeminen, indikaattorit ja niiden seuranta, jotta tuloksellisuutta ja kes-tävyyttä voidaan mitata. Tarvitaan enemmän huomiota ja koulutusta tulos-johtajuuden suhteen, jotta tällainen muutos saadaan aikaan.

4. Evaluaatioiden tulosten seurantaa ja suositusten toimeenpanoa pitäisi doku-mentoida paremmin kenties kehittämällä välineitä (esim. hallinnon seuranta-raportteja).

5. Tarvitaan suurempaa huomiota evaluaatioiden ja ennakkoarvioiden tuottami-en opetusten jakeluun eri tiedonsiirtokanavien kautta niin ministeriön sisällä kuin ulkopuolellakin. Eräs mahdollinen väline tätä varten voisivat olla 4-6 si-


vun tiedotteet evaluaatioista, jotka sitten laitettaisiin tulostettavaan muotoon ministeriön internetsivuille.

6. Uuden kehityspoliittisen ohjelman toimeenpanossa olisi tärkeä tunnustaa, että kehitysyhteistyön hankkeiden ja ohjelmien painopisteen siirtymä sekto-rilta toiselle vie aikaa ja edellyttää hallinnollista opastusta uusien avainkäsittei-den ja prioriteettien toiminnallistamisessa.


SAMMANFATTNING

Metaanalys av utvecklingsutvärderingar under 2007 och 2008Utrikesministeriets (UM) utvecklingsutvärdering (EVA-11) beställde en metaanalys av 13 bedömningar och 20 utvärderingsrapporter som utförts av olika geografiska enhe-ter inom ministeriet under 2007 och 2008. Utvärderingar som EVA-11 direkt åtagit sig ingick inte i samplet. Metaanalysens fokus låg på att bedöma kvaliteten på utvär-deringsrapporterna, kvaliteten på hjälpinsatserna som bedömdes i rapporterna samt genomförandet av UM:s utvecklingspolitiska program 2007. Speciellt fästes uppmärk-samhet vid att undersöka huruvida insatserna var effektiva och främjade en hållbar utveckling. Faktorer för utvecklingsinsatsernas relativa framgång och misslyckande undersöktes också.

Utvecklingspolitiskt program 2007I oktober 2007 antog Finlands regering en politik för utvecklingssamarbetet, Develop-ment Policy Programme 2007: Towarsds a Sustainable and Just Word Community, som framhä-ver en hållbar utveckling och fattigdomsbekämpning. En hållbar utveckling anses ha tre besläktade dimensioner: en ekologisk, en ekonomisk och en social. Politiken tar också fasta på politisk koherens, komplementaritet och effektivitet samt på att säker-ställa att flera genomgående teman behandlas under utvecklingsinsatserna. Politiken framhäver Finlands strävan att stöda utvecklingssamfund där Finland kan bidra med värde på grund av sina relativa fördelar vad gäller expertis eller andra faktorer.

Översikt över utrikesministeriets bedömningar och utvärderingarUtrikesministeriet är med i OECD/DAC Evaluation Network och har förbundit sig att följa internationell praxis i sitt program för utvecklingssamarbete, inklusive be-dömningar och utvärderingar. Ländernas programansvariga, som omfattar personal på ambassader samt handläggare, konsulter för relevant sektor samt andra på minis-teriet ansvarar för att bedömning av nya föreslagna projekt eller program utförs. De är även ansvariga för halvtidsutvärderingar och slutliga utvärderingar av utvecklings-projekt och -program.

Metaanalys

Syfte och målsättning Syftet med metaanalysen är att ta lärdom av färsk utvärderingserfarenhet, fastställa god praxis och göra denna lärdom mer allmänt tillgänglig. Målsättningen med be-dömningen är att utvärdera den övergripande kvaliteten på utvärderingsrapporterna och kvaliteten på hjälpinsatserna jämfört med OECD/DAC:s kvalitetsstandarder och utvärderingskriterier, samt att bedöma operationaliseringen av utvecklingspolitiken, speciellt med avseende på hållbarhet och effektivitet.

Analyssampel För denna metaanalys sammanställde utrikesministeriets utvecklingsutvärdering en


brokig mängd utvärderingsrapporter från olika enheter inom ministeriet. Alla rap-porter var oberoende utvärderingar som lagts ut till konsulter utanför ministeriet, förutom en rapport som skrivits gemensamt av en sektorkonsult vid ministeriet och en annan konsult.

Bedömningen granskade 33 rapporter. 13 rapporter var bedömningar eller för-handsbedömningar. 20 rapporter var halvtidsutvärderingar, utvärderingar av övrigt genomförande, slutliga eller utvärderingar ex-post av pågående eller avslutade utveck-lingsprojekt eller -program. Av dessa 20 rapporter var en rapport en sektorgranskning av två program och en annan rapport var en granskning av tre projekt som genom-förts i tre olika länder i en region. Således omfattar hela samplet 33 rapporter om 36 separata projekt eller program.

De 33 rapporterna hänför sig till 36 pågående insatser i sammanlagt 24 länder. Näs-tan hälften fanns i Finlands centrala samarbetsländer och en tredjedel i länder (eller områden) som återhämtar sig från (eller som fortfarande genomgår) våldsamma kri-ser. Av de 33 rapporter som bedömdes var fem på spanska, en på finska och 27 på engelska. Rapporterna omfattade en uppsjö av olika instrument för finansiering, inklusive några insatser som genomförts med en multilateral organisation eller som en del av ett samarbete mellan flera bidragsgivare eller stödprogram för en sektor. En utvärdering handlade om aktiviteter som understöds av anslaget för lokalt samarbete, en bedömning gällde räntestöd och fem bedömningar gällde INGO-förslag. 37 % av projekten gällde sektorerna för miljö och naturresurshantering och 31 % utbildning och forskning.

Metod Denna bedömning var i första hand en teoretisk granskning av det tillhandahållna materialet. Forskningsteamet granskade materialet och bedömde varje utvärderings-rapport enligt ett antal centrala frågor och kriterier baserade på metaanalysens uppd-ragsbeskrivning, OECD/DAC:s utvärderingskriterier och OECD/DAC:s kvalitets-standarder för utvärdering.

För att utvärdera kvaliteten på utvärderingsrapporterna använde forskningsteamet OECD/DAC:s tio kvalitetsstandarder för utvärdering: (1) utvärderingens logiska grund, syfte och målsättning; (2) utvärderingens omfattning; (3) kontext; (4) utvär-deringsmetod; (5) informationskällor; (6) oberoende; (7) utvärderingsetik; (8) kvali-tetsgaranti; (9) utvärderingsresultatens relevans; och (10) fullständighet. Med dessa kriterier som bas betygsatte forskningsteamet den övergripande kvaliteten på varje rapport. Utöver detta genomförde forskningsteamet även en kvalitativ bedömning av rapporterna och noterade alla ovanliga eller anmärkningsvärda aspekter av rapporter-na, projekten och programmen. För att utvärdera bistånds kvalitet, dvs. programmen och projekten, betygsatte forsk-ningsteamet hjälpen med hjälp av fem OECD/DAC-kriterier – relevans, effektivitet, verkningskraft, inverkan och hållbarhet – samt tre övriga kriterier – (1) politisk kohe-


rens och komplementaritet, (2) finländskt mervärde, och (3) baslinjedata, indikatorer och uppföljande. Baserat på dessa åtta kriterier betygsatte forskningsteamet sedan insatsens övergripande kvalitet. Forskningsteamet genomförde även en kvalitativ be-dömning av rapporterna. För ytterligare information utförde forskningsteamet några intervjuer med personal på ministeriet och andra personer förknippade med centrala resurser, kontaktade några personer som arbetar på finländska ambassader och hitta-de en del kompletterande information på Internet.

Forskningsresultat

Utvärderingsrapporternas kvalitet Forskningsteamet som utförde metaanalysen strävade efter att bedöma utvärderings-rapporterna enligt OECD/DAC:s tio kvalitetsstandarder för utvärdering. Detta visade sig vara mycket svårt, eftersom rapporterna inte omfattade en del av den information som behövdes för att kunna bedöma dessa frågor. De som översåg utvärderingarna kanske t.ex. kände till viss information, men denna information fanns inte till forsk-ningsteamets förfogande.

Forskningsteamet betygsatte den övergripande kvaliteten på utvärderingsrapporterna som allmänt god vad gäller OECD/DAC:s tio kvalitetsstandarder för utvärdering. Kvaliteten på fyra av rapporterna ansågs vara mycket god, på sju rapporter god till mycket god och på två dålig. Kvaliteten på de rapporter som uppfyllde kvalitetsstan-darderna ansågs vara mycket god, medan de rapporter där kvaliteten ansågs vara god hade uppfyllt de flesta av standarderna. Rapporterna där kvaliteten ansågs vara dålig uppfyllde endast i viss mån dessa standarder. I de flesta fall överensstämde rapporter-na tämligen väl med respektive uppdragsbeskrivning som fastställts för dem. I vissa fall gick rapporterna längre än vad som fastställts i uppdragsbeskrivningen, som t.ex. genom att bedöma insatsen enligt vissa eller samtliga fem OECD/DAC-kriterier även om dessa inte hade nämnts i uppdragsbeskrivningen.

Det bör anmärkas att av de elva förhandsbedömnings- och utvärderingsrapporter som erhöll de högsta betygen hade flera av dem utarbetats i samarbete med andra utvecklingspartner. I vissa fall gällde dessa utvärderingar större sektorprogram för flera bidragsgivare eller multilaterala organisationer, som avsatt betydande resurser (stora team, mer tid, större insats från bidragsgivarledningen) för programmen. Andra rapporter som var mycket bra var mindre rapporter som gällde bilaterala program där utvärderingsteamet var mycket insatt i frågan och/eller de berörda länderna och tillhandahöll en god överblick av projektet inom dess övergripande kontext och hade ingående kunskap om finländsk utvecklingspolitik samt lokalpolitik.

Biståndinsatsernas kvalitet Den övergripande kvaliteten ansågs vara god på de 36 insatserna vad gällde de åtta utvärderingskriterierna. Kvaliteten på tre insatser ansågs vara mycket god, på två god till mycket god, på 23 god, på två mellan dålig och god och på fem ansågs kvaliteten vara dålig. De projekt eller program vars kvalitet ansågs vara mycket god var de som


erhöll bra betyg vad gäller OECD/DAC:s utvärderingskriterier för relevans, effekti-vitet, verkningsgrad, hållbarhet och inverkan samt för politisk koherens och infor-mationens tillgänglighet och användande. Å andra sidan var de projekt eller program vars kvalitet inte erhöll lika bra betyg sådana som hade problem med genomförandet, vilket ledde till försämrad effektivitet och verkningsgrad och problem vad gäller att uppnå hållbarhet och en positiv inverkan. Av de insatser vars kvalitet ansågs vara dålig var en förhandsbedömning av ett projektförslag som senare reviderades och förbättrades och sedan godkändes för finansiering och som nu genomförs. De utvär-deringskriterier som de olika utvärderarna gav högst betyg var relevans och politisk koherens. Verkningsgrad och data, indikatorer och uppföljning fick de sämsta bety-gen. Kvaliteten på hållbarhet och effektivitet ansågs i allmänhet vara god. Endast en fjärdedel av insatserna hade utvärderats i avseende på finländskt mervärde.

Hållbarhetsaspekten studerades mer ingående. I de flesta fall fokuserade utvärderin-garna och bedömningarna på institutionell och finansiell hållbarhet, även om vissa beaktade ett bredare fält av aspekter. Alla var överens om att större faktorer som bidrar till en god hållbarhet är lokal äganderätt, politisk vilja, regeringens engagemang, deltagande och självbestämmande, kapacitetsutveckling, skapandet av lämpliga poli-tiska ramar och fortsatt finansiering. Brist på dessa element ansågs ofta utgöra hinder eller risker för hållbarheten. I vissa länder såsom Palestina, Bolivia och Kenya ansågs den instabila politiska situationen utgöra ett betydande hinder eller en risk. Endast två rapporter hänvisade till riskbedömning och två nämnde idén att skapa handlingspro-gram för hållbar utveckling.

Genomförande av det utvecklingspolitiska programmet 2007 Baserat på samplet av bedömningar och utvärderingar som utförts under 2007 och 2008 kan endast begränsade observationer göras vad gäller det utvecklingspolitiska programmet för 2007. Tolv av utvärderingarna gjordes i oktober 2007 eller tidigare, dvs. innan den nya politiken formellt hade antagits. Planeringen av alla förutom sex insatser hade redan påbörjats under en tidigare politik. En del av de projekt eller program som utvärderades under 2007 eller 2008 har fortsatt eller så har nya faser eller uppföljningsprojekt inletts. Dock hade forskningsteamet för metaanalysen in-gen information om huruvida sådana fortsatta insatser har ändrats i något avseende för att återspegla prioriteringarna i politiken 2007. Samplet tyder på att en övergång vad gäller sektortyngdpunkt är på gång där större vikt placeras vid projekt och pro-gram i sektorerna för miljö och naturresurser. Mer och mer uppmärksamhet fästs vid frågor om politisk koherens, komplementaritet och förbättrad effektivitet, inklusive finländskt mervärde.

Jämförelse med tidigare metaanalyserEn sammanfattande undersökning av utvärderingar åren 1988-1995 (Koponen J & Mattila-Wiro P 1996) och en metaanalys av utvärderingar 2006 (White P & Stenbäck T 2007) gav många liknande resultat när det gäller utvärderingarnas och utvecklings-projektens kvalitet. Vissa förbättringar har dock noterats i utvärderingarnas och bis-tåndets kvalitet. Allt större vikt fästs vid frågor som rör hållbarhet, politisk koherens


och komplementaritet samt ökad effektivitet, inklusive finländskt mervärde. Ytter-ligare förbättringar behövs inom kunskaps- och informationshantering, inverkan och effektivitet. Utvärderingarnas fokus ligger fortfarande på deras roll inom projektför-valtning, medan det finns skäl att vidareutveckla deras potential att utnyttjas i ett bre-dare syfte som medel för inlärning och ansvarsskyldighet.

Slutsatser och rekommendationerSist och slutligen bör det anmärkas att det finländska utvecklingssamarbetet spelar en viktig roll, men det finns rum för förbättring både vad gäller enskilda insatser och systemet som helhet. Utvärderingar spelar en viktig roll i en sådan kvalitetsförbättring, och således behövs ansträngningar för att uppgradera den övergripande kvaliteten på utvärderingarna och för att använda informationen från dem på ett mer systematisk och strategiskt sätt och således öka den övergripande kvaliteten på Finlands program för utvecklingssamarbete. Baserat på metaanalysen görs följande rekommendationer:

1. För att främja framtida metaanalyser av utvärderingar, eller andra sätt att för-medla utvärderingsresultaten, skulle det vara viktigt att komma överens om vilka standarder som ska användas och sedan utveckla verktygen och tillhan-dahålla utbildning för att bistå utvärderarna och deras överordnade i att uppnå dessa standarder. Även om utvärderingar generellt sett har använts flitigt som ett ledarskapsverktyg kan deras potential att sprida kunskap och förklarlighet ytterligare förbättras. Utrikesministeriet bör etablera minimistandarder som ska beaktas i alla bedömningar och utvärderingar men också tillåta flexibilitet vad gäller att lägga till ytterligare information (speciella frågor) eller andra ändringar enligt den enskilda uppgiftens behov. Dessutom krävs åtgärder för att stärka ministeriets arkiverings- och databassystem för att säkerställa till-gången till ett mer omfattande register av sådana utvärderingar.

2. Även om utvärderingarnas kvalitet i allmänhet var god, kunde den förbättras ytterligare. Centrala faktorer för att förbättra utvärderingarna är att förbättra uppdragsbeskrivningarna så att de täcker alla grundläggande utvärderingskri-terier, och att tillhandahålla fler resurser för utvärdering (inklusive mer tid för både ledarskap och konsultering).

3. För att förbättra kvaliteten på utvecklingsinsatserna bör mer uppmärksam-het fästas vid frågor om baslinjedata, indikatorer och uppföljning, avsedd in-verkan samt effektivitet och hållbarhet. Denna förändring kommer att kräva att en större tonvikt placeras på utbildning inom ledarskap för resultat.

4. Åtgärder bör vidtas för att dokumentera uppföljningen och åtgärder basera-de på utvärderingen, kanske genom en bredare användning av verktyg såsom ledningens åsikter.

5. Mer måste göras för att kunna förmedla den kunskap som erhålls från utvär-deringar och bedömningar, genom olika kommunikationskanaler både inom


och utanför ministeriet. Ett möjligt verktyg kan vara en serie av korta utvär-deringsanteckningar, 4-6 sidor, som skulle finnas tillgängliga på ministeriets webbplats samt i tryckt format.

6. Vad gäller genomförandet av regeringens utvecklingspolitiska program är det viktigt att förstå hur mycket tid som behövs för att genomföra betydande ändringar i portfolion och den verksamhetsledning som behövs för att göra centrala nya prioriteringar och koncept driftsklara.


SUMMARY

Meta-analysis of Development Evaluations in 2007 and 2008The Ministry for Foreign Affairs of Finland (MFA) Development Evaluation (EVA-11) commissioned a Meta-analysis of 13 appraisals and 20 evaluation reports under-taken by different geographical units of the MFA in 2007 and 2008. Evaluations undertaken directly by the EVA-11 were not included in the sample. The Meta-analysis focused on assessing the quality of the evaluation reports, the quality of the aid interventions assessed in the reports, and the implementation of the Finnish 2007 Development Policy Programme. Particular emphasis was placed on looking at whet-her the interventions were effective and promoting sustainable development. Factors for relative success and failure of the development interventions were also studied.

Development Policy Programme 2007In October 2007, the Government of Finland adopted a new development coope-ration policy, Development Policy Programme 2007: Towards a Sustainable and Just World Community, which places major emphasis on sustainable development and poverty reduction (Ministry for Foreign Affairs of Finland 2007a). Sustainable development is seen as having three interrelated dimensions - ecological, economic, and social. The policy also focuses on policy coherence, complementarity, and effectiveness, as well as ensuring that several cross-cutting themes are addressed through the develop-ment interventions. The policy emphasizes Finland’s desire to support development cooperation where Finland can add value, because of its comparative advantages in terms of expertise or other factors.

Overview of MFA Appraisals and Evaluations MFAis a member of the Organisation for Economic Co-operation and Develop-ment’s Development Assistance Committee (OECD/DAC) Evaluation Network, and has committed itself to follow international practices in its development coope-ration programme, including its appraisals and evaluations. Concerned country (or re-gion) programme staff, comprised of Embassy staff as well as desk officers, relevant sectoral advisors, and others at MFA headquarters, are responsible for carrying out appraisals of new proposed projects or programmes. They also have responsibility for mid-term and final evaluations of development projects and programmes.

Meta-analysis

Purpose and Objectives The Meta-analysis aims to draw lessons learned from recent evaluation experience, identify good practices, and make those lessons more widely available. The objectives of the assessment are to evaluate the overall quality of the evaluation reports and the quality of the aid interventions against the OECD/DAC quality standards and evaluation criteria, as well as to assess the operationalisation of the new development policy, particularly in terms of sustainability and effectiveness.


Sample Analysed For this Meta-analysis, MFA EVA-11 compiled a diverse set of evaluation reports from various units within the MFA. The reports were all independent evaluations, outsourced to consultants outside of the MFA, except for one report that was co-authored by a MFA sectoral advisor and a consultant.

The assessment looked at 33 reports. Thirteen reports were appraisals or pre-app-raisals. Twenty reports were mid-term, other implementation, final or ex post evalua-tions of ongoing or completed development projects or programmes. Of the 20, one evaluation report was a sectoral review of two programmes, and a second report was a review of three projects done in three different countries in one region, so the total sample includes 33 reports on 36 separate projects or programmes.

The 33 reports pertain to 36 interventions ongoing in a total of 24 countries. Almost half were in Finland’s long-term key partner countries, and another third in countries (or regions) recovering from (or still undergoing) violent crises. Of the 33 reports assessed, five were in Spanish, one in Finnish, and 27 in English. The reports covered a variety of different funding instruments, including some interventions undertaken with a multilateral organization or as part of a multi-donor partnership or sector support programme. One evaluation was of activities supported by Local Coope-ration Funds (LCF), one appraisal concerned a concessional credit project, and five appraisals were of international non-governmental organisations (INGO) proposals. Thirty-seven percent of the projects were in the environment and natural resource management sectors, and 31 percent in education and research.

Methodology This assessment was primarily a desk review of the documentation provided. The Team reviewed the documentation, assessing each evaluation report against a number of key questions and criteria, which were based upon the Meta-analysis Terms of Reference (TOR), the OECD/DAC evaluation criteria, and OECD/DAC evaluation quality standards.

To evaluate the quality of the evaluation reports, the Team used the ten OECD/DAC evaluation quality standards -- (1) rationale, purpose and objectives of the evaluation; (2) evaluation scope; (3) context; (4) evaluation methodology; (5) information sour-ces; (6) independence; (7) evaluation ethics; (8) quality assurance; (9) relevance of the evaluation results; and (10) completeness. On the basis of the ratings, the Team then assigned an overall quality rating to the report. The Team also undertook a qualitative assessment of the reports, and took note of any unusual or noteworthy aspects of the reports, projects or programmes. To evaluate the quality of the aid, i.e., the programmes and projects, the Team rated the aid using five OECD/DAC criteria – relevance, effectiveness, efficiency, impact, and sustainability – as well as three other criteria – (1) policy coherence and comple-mentarity, (2) Finnish added value, and (3) baseline data, indicators, and monitoring.


Based on these eight criteria, the Team then rated the overall quality of the interven-tion. For additional information, the Team conducted some interviews with MFA personnel and other key resource persons, contacted some individuals working in Finnish Embassies, and found some supplementary information on the internet.

Findings

Quality of the Evaluation Reports The Meta-analysis Team endeavoured to assess the evaluation reports against the ten OECD/DAC evaluation quality standards. This effort proved to be very difficult, as some information needed to assess these issues was not included in the evaluation re-ports. Some information, for example, might be known by the evaluation managers, but was not available to the Meta-analysis Team.

The Meta-analysis Team rated the overall quality of the evaluation reports as generally good in terms of the ten OECD/DAC evaluation quality standards. Four reports were rated as very good, seven as good to very good, and two rated as poor. Very good reports were those that fulfilled the quality standards, whereas the good reports had fulfilled most of the standards. The poor reports were those that only partially fulfilled these standards. In most cases, the reports responded reasonably well to their respective terms of reference (TOR). In some cases, the evaluation reports went beyond their TOR, such as assessing the intervention according to some or all of the five OECD/DAC criteria even if they were not all mentioned in the TOR.

It should be noted that of the eleven evaluation and appraisal reports most highly rated, several were prepared in collaboration with other development partners. In some cases, these were evaluations of major multi-donor sector programmes or mul-tilateral organisations, which devoted considerable resources (large teams, more time, more donor management input) into the exercises. Some other very good reports were smaller ones, done of bilateral programme, where the evaluation team members were highly knowledgeable about the issues and/or countries concerned, provided a good overview of the project within its overall context, and had a thorough knowled-ge of Finnish development policies as well as local policies.

Quality of the Aid Interventions For the 36 aid interventions, in terms of the eight evaluation criteria, their overall quality was rated as good. Three interventions were rated very good, two were rated good to very good, 23 were rated good, two between poor and good, and five were rated poor. The very good projects or programmes were those that rated well for the OECD/DAC evaluation criteria of relevance, effectiveness, efficiency, sustainability and impacts, as well as policy coherence and availability and use of information. In contrast, other projects or programmes less well rated were those that had imple-mentation difficulties, leading to lowered effectiveness and efficiency, and difficulties in moving towards sustainability and positive impacts. Of the interventions rated poor, one was an appraisal of a project proposal, which was subsequently revised and


improved, then approved for funding and is now being implemented. The evaluation criteria most highly rated by the various evaluators were relevance and policy cohe-rence. The weakest ratings were for efficiency and data, indicators and monitoring. Sustainability and effectiveness were generally rated as good. Only one-quarter of the interventions had been evaluated in terms of Finnish added value.

The issue of sustainability was considered in more depth. Most often, evaluations and appraisals focused on institutional and financial sustainability, although some did con-sider a wider range of issues. It was commonly agreed that major factors contributing to good sustainability are local ownership, political will, government commitment, participation and empowerment, capacity development, building appropriate policy frameworks, and continuing financing. Conversely, lack of these elements was often regarded as obstacles or risks to sustainability. In some countries, such as Palestine, Bolivia, and Kenya, the volatile political situation was considered to be a significant obstacle or risk. Only two reports referred to risk assessments, and two mentioned the idea of creating Sustainability Action Plans.

Implementation of the Development Policy Programme 2007 Based on the sample of appraisals and evaluations conducted in 2007 and 2008, only limited observations can be made regarding the Development Policy Programme 2007 (Ministry for Foreign Affairs of Finland 2007a). Twelve of the evaluations were done in October 2007 or earlier, i.e., before the new policy was formally adopted. The design of all but six interventions had begun under a previous policy. Some of the projects or programmes evaluated in 2007 or 2008 have continued, or new phases or follow-up projects have been started. The Meta-analysis Team had no information, however, on whether such continued interventions had been modified in any way to reflect the priorities of the 2007 policy. The sample suggests that a shift in sectoral emphasis is ongoing, with more focus being given to projects and programmes in the environment and natural resources sectors.

Comparison with Previous Meta-Analyses A synthesis study of 1988-95 evaluations (Koponen J & Mattila-Wiro P 1996) and a Meta-analysis of 2006 evaluations (White P & Stenbäck T 2007) had come up with many similar findings, regarding the quality of the evaluations and the interventions. Nonetheless, some improvements in evaluation and aid quality have been noted. In-creasingly, attention is being given to issues of sustainability, policy coherence and complementarity and improved effectiveness, including Finnish added value. Further improvements in knowledge and information management, impact and efficiency are still needed. The use of evaluations continues to stress their management functions, and their potential for broader use, in terms of learning and accountability, needs to be further developed.

Conclusions and RecommendationsAs an overall conclusion, it can be noted that Finnish development cooperation is making an important difference, but there remains room for improvement, both on


the level of individual interventions and for the system as a whole. Evaluations play an important role in such quality improvement, and thus efforts are needed to up-grade the overall quality of evaluations, and to use information from them more systematically and strategically, to increase the overall quality of Finland’s develop-ment cooperation programme. On the basis of this Meta-analysis, the following are recommended:

1. To facilitate future meta-analyses of evaluations, or other means of sharing evaluation results, it would be important to agree upon the standards to be used, and then develop the tools and provide the training to assist the eva-luators and evaluation managers to meet these standards. Although overall the evaluations have been well used as management tools, their potential use for broader learning and accountability could be further improved. MFA should establish minimal standards to be addressed in all appraisals and eva-luations but allow flexibility for adding additional information (special issues) or other modifications as needed for the individual assignment. Moreover, efforts need to be made to strengthen the MFA’s archiving and database systems, to ensure that more comprehensive records of such evaluations are accessible and available.

2. Although the evaluations were generally of a good quality, they could be furt-her improved. Key elements for improving the evaluations are improving TORs, to ensure that they cover all basic evaluation criteria, and providing more resources for the evaluation (including both more management time and more consultant time).

3. To improve the quality of development interventions, more attention needs to be focused on issues of baseline data, indicators, and monitoring, inten-ded impacts, and effectiveness and sustainability. This change will require much more emphasis on and training in managing-for-results.

4. Efforts should be made to document follow-up and action based upon the evaluations, perhaps through the wider use of tools such as management response formats.

5. Greater efforts are needed to share lessons learned from evaluations and ap-praisals, through various communication channels, both within and outside of the MFA. One possible tool might be a series of short evaluation briefing notes, 4-6 pages, which could be made available on the MFA web pages, as well as in printed format.

6. In terms of implementation of the Finnish ’s Development Policy Program-me, it is important to recognize the time needed to make significant changes in the portfolio, and the operational guidance needed to operationalise key new priorities and concepts.


Findings

Conclusions

Recommendations

The appraisals and evaluations vary considerably in their scope, evaluation criteria and issues. Some files were incomplete, i.e., for several evaluations, the TOR were not available for the Meta-analysis. This situation was also noted two years ago, in the 2006 Meta-analysis of Development Evaluations.

Although MFA has an electronic project database (archive), some evaluation reports, their TORs, or other relevant information does not make it to the database. The MFA does not have complete information on the total number of decentralised evaluations done each year.

To facilitate future meta-analyses of evaluations, or other means of sharing evaluation results, the MFA should consider development of some basic evaluation tools, provide training in their use, and ensure that standard minimal formats and criteria are included in all evaluations. Such tools, however, need to allow flexibility for adding additional information (special issues) as needed for the individual assignment. Moreover, efforts need to be made to strengthen the MFA’s archiving and database systems, to ensure that more comprehensive records of such evaluations are available.

TOR for evaluations and appraisals vary significantly in content and comprehensiveness. Evaluators generally respond to their TOR, and address the issues identified in the TOR. Evaluation quality also varies – understandably – with the resources (personnel, time) available.

The TOR for evaluations and appraisals do not follow standard formats and contents. MFA reports should all address the five OECD/DAC evaluation criteria, and any other criteria the MFA decides to include as core elements for evaluations, such as coherence and Finnish added value. Adequate resources must be available to undertake the assessments.

Although the evaluations were generally of good quality, they could be further improved. Key elements for improving the evaluations are improving the TOR, and improving the resources for the evaluation (including both management and consultant inputs).

The aid interventions were generally regarded by evaluation teams as being very relevant and coherent with policies. Few of the reports, however, had much information concerning knowledge management, i.e., baseline data, indicators, and monitoring. In many projects and programmes, effectiveness, sustainability and impact needed greater attention.

The design and implementation of development interventions need to focus more on strategic issues, and to do so, need better information to assess their effectiveness, performance, achievements, and progress towards sustainability and impacts.

To improve the quality of development interventions, more attention needs to be focused on issues of baseline data, indicators, and monitoring, intended impacts, and issues of effectiveness and sustainability. This change will require much more emphasis on and training in managing-for-results.


While a meta-analysis of evaluation and appraisal reports can consider the interventions at a particular point in time, they do not provide information on follow-up.

The evaluation and appraisal reports alone are inadequate to assess their impact. Information needs to be documented on the management response and any other follow-up.

Efforts should be made to document follow-up and action based upon the evaluations, perhaps through wider use of tools such as management response formats.

Many of the appraisals and evaluation reports focus on assessing specific issues on which the MFA needs to make a decision, regarding implementation or funding. They are less likely to focus on broader lessons learned. Final or ex post evaluations are most likely to be conducted when there is a plan for follow-up support, or replication elsewhere. If an intervention has not performed well and thus is unlikely to be continued, then it is also unlikely to have a final evaluation.

Decentralised appraisals and evaluations tend to be circulated primarily among country team members. It is difficult for others in the MFA, or the broader development community to obtain information on appraisals and evaluations conducted. Potential lessons learned, thus, are not shared more widely. It is important to learn not only from successes and best practices, but from less successful initiatives.

Greater efforts are needed to share lessons learned from evaluations and appraisals, through various communication channels, both within and outside of the MFA. One possible tool might be a series of short 4-6 page evaluation briefing notes, which could be made available on the MFA web pages(as well as in printed format).

Only some of the 2007-2008 reports assessed examined the degree to which the intervention responds to key elements of the October 2007 development policy, such as focusing on the three dimensions of sustainability or Finnish added value.

More guidance is needed on how to operationalise the new policy in project design, appraisal, implementation, and evaluation. For example, guidance is needed on how to operationalise the concepts of sustainability and Finnish added value.

In terms of implementation of the Finnish Development Policy Programme, it is important to recognize the time needed to make significant changes in the portfolio, and the operational guidance needed to operationalise key new priorities and concepts.


Table 1 Meta-analysis sample.

ReportNumber

2007 and 2008 Evaluations and Reviews Assessed Date ofReport

EvaluationType

I Rural sustainable livelihoods and poverty reduction supportprogramme in Boaco and Chontales (FOMEVIDAS), Nicaragua

June2007

OtherEvaluation

II Multi-donor Fund for Equality and Sexual and ReproductiveRights (FED), Nicaragua

June2008

Mid-TermEvaluation

III Sector Wide Support Programme for Productive RuralDevelopment (PRORURAL), Nicaragua

Nov.2008

Mid-TermEvaluation

IV Reproductive Health, Equality and Rights Project (SARED),Nicaragua

July 2008 FinalEvaluation

V Andean Regional Project of Bilingual Intercultural Education(EIBAMAZ), Peru, Ecuador, and Bolivia

June2008

Mid-TermEvaluation

VI Finland-South Africa Cooperation in Education Sector (Finnishsupport to Inclusive Education and Higher EducationProgrammes)

Nov.2008

VI-A. Finnish support to Inclusive Education Programme inSouth Africa

FinalEvaluation

VI-B. Finnish support to Higher Education Programme in SouthAfrica

FinalEvaluation

VII Cooperation Framework on Innovation Systems Programmebetween Finland and South Africa (COFISA)

Dec.2008

FinalEvaluation

VIII Biogas Projects, Ethiopia Mar.2007

Ex postEvaluation

IX Gender and Governance Programme (GGP), Phase II, Kenya June2008

FinalEvaluation

X Finnish Support Programme to the National Institute forEducational Development (INDE), Mozambique

Dec.2008

Mid-TermEvaluation

XI Support to Rural Development in Zambézia Province(PRODEZA), Zambézia, Mozambique

Jan. 2009 Mid-TermEvaluation

XII Palestinian-Finnish Education Programme, Phase III Sept.2008

Mid-TermEvaluation

XIII Land Administration Project in the Territories of the PalestinianAuthority

Oct.2008

OtherEvaluation

XIV Entrepreneurship Development Programme for Women inFood Processing in Central Viet Nam (Phase II)

May2007

FinalEvaluation

XV Nepal Education for All (EFA) Programme 2004-2009 May2007

Mid-TermEvaluation

XVI Forest Policy and Economics Education and Research, FOPER,Western Balkans (Albania, Bosnia and Herzegovina, Croatia,and Macedonia)

Mid-2007 Mid-TermEvaluation

XVII Three (3) Education Projects in the Western Balkans (Serbia,Bosnia and Herzegovina, and Montenegro)

2007

XVII-A. Serbian Teacher Education Development Programme(STEP)

FinalEvaluation

XVII-B. Programme for Finnish Cooperation in the EducationSector of Bosnia and Herzegovina 2003-2006 (CES)

FinalEvaluation

XVII-C. Development of the Education Sector in Montenegro Mid-TermEvaluation

XVIII Environmental Geographic Information System (GIS) forMontenegro, Phase I

June2007

FinalEvaluation

XIX Support to Disability Policy Development in Bosnia andHerzegovina (2006-2009)

Oct.2007

Mid-TermEvaluation

XX Post-Emergency Reconstruction in the Field of Meteorology,Phase II, Mozambique

Oct.2008

Ex postEvaluation

NOTE: Report VI covered two programmes (VI-A and VI-B) and report XVII covered three projects(XVII-A,

XVII-B, and XVII-C).

Phase II, Mozambique 2008 EvaluationReport VI covered two programmes (VI-A and VI-B) and report XVII covered three projects

XVII-B, and XVII-C).XVII-B, and XVII-C).



ReportNumber


EvaluationType


June2007

OtherEvaluation


June2008

Mid-TermEvaluation


Nov.2008

Mid-TermEvaluation




June2008

Mid-TermEvaluation


Nov.2008


FinalEvaluation


FinalEvaluation


Dec.2008

FinalEvaluation


Ex postEvaluation


FinalEvaluation


Dec.2008

Mid-TermEvaluation




Mid-TermEvaluation


Oct.2008

OtherEvaluation


May2007

FinalEvaluation


Mid-TermEvaluation




2007


FinalEvaluation


FinalEvaluation



June2007

FinalEvaluation


Oct.2007

Mid-TermEvaluation


Oct.2008

Ex postEvaluation







ReportNumber


EvaluationType


June2007

OtherEvaluation


June2008

Mid-TermEvaluation


Nov.2008

Mid-TermEvaluation




June2008

Mid-TermEvaluation


Nov.2008


FinalEvaluation


FinalEvaluation


Dec.2008

FinalEvaluation


Ex postEvaluation


FinalEvaluation


Dec.2008

Mid-TermEvaluation




Mid-TermEvaluation


Oct.2008

OtherEvaluation


May2007

FinalEvaluation


Mid-TermEvaluation




2007


FinalEvaluation


FinalEvaluation



June2007

FinalEvaluation


Oct.2007

Mid-TermEvaluation


Oct.2008

Ex postEvaluation






Table 1 Meta-analysis sample (continued).

Report Number

2007 and 2008 Appraisals and Pre-appraisals Assessed Date of Report

Evaluation Type

XXI Programme for Promotion of Gender Equality and Women's Rights (Pro-Género), Nicaragua

June 2007 Appraisal

XXII Hydro-Meteorological Project, Peru May 2008 Appraisal XXIII Water Supply, Sanitation and Hygiene Capacity Building Pool

Fund (WASH CBPF), Ethiopia Oct. 2007 Appraisal

XXIV Integrated Development of Eastern Nile - Watershed Management (IDEN-WM) Project, Ethiopia [Renamed: Watershed Management Sub-component of Tana-Beles Sub-Basin Integrated Water Resources Development Project (TBIWRDP)]

Mar. 2008 Appraisal

XXV Integrated Development of Eastern Nile - Watershed Management (IDEN-WM) Project, Sudan [Renamed, Regional: Egypt, Ethiopia and Sudan Eastern Nile Watershed Management Project, Community Watershed Management Sub-Project, Sudan]

May 2008 Pre-appraisal

XXVI Rural Water Supply, Sanitation and Hygiene Programme in Benishangul-Gumuz Region, Ethiopia (WASH Beni)

Dec. 2008 Appraisal

XXVII Finnish Support to National Forest Program (SNFP), Mozambique

Jan. 2009 Appraisal

XXVIII Lake Victoria Basin Commission (LVBC) and Partnership, Kenya, Tanzania and Uganda

Dec. 2008 Appraisal

XXIX International Union for the Conservation of Nature (IUCN) Programme, Sustaining Rural Communities and their Traditional Landscapes through Strengthened Environmental Governance in Transboundary Protected Areas of the Dinaric Arc, Western Balkans (Albania, Bosnia and Herzegovina, Croatia, Montenegro, Serbia and Slovenia)

Nov. 2008

Appraisal

XXX European Centre for Nature Conservation (ECNC) Project, Biodiversity and Ecosystem Services for Local Sustainable Development in the Western Balkans (Albania, Bosnia and Herzegovina, Kosovo, and Macedonia, Montenegro, and Serbia)

Nov. 2008

Appraisal

XXXI Mercy Corps Project, Community Eco Economic Support Initiatives (CESI) in Kosovo

Oct. 2008 Appraisal

XXXII Catholic Relief Services (CRS) Project on Sustainable Alliances for Education in Kosovo

Oct. 2008 Appraisal

XXXIII European Centre for Minority Issues (ECMI) Project, Support to Minority Communities during and after the Decentralisation Process in Kosovo

Oct. 2008 Appraisal


1 INTRODUCTION

This Meta-analysis of Development Evaluations in 2007 and 2008 examines a samp-le of evaluation reports conducted for the Ministry for Foreign Affairs of Finland (MFA). It is based upon an assessment of 33 reports concerning 36 projects and programmes. The 33 reports consist of 12 appraisals, one pre-appraisal and 20 mid-term, other implementation, final or ex post evaluations. The list of reports, projects and programmes is provided in Table 1. The assessment had three major objectives: (1) evaluate the overall quality of the evaluation reports; (2) evaluate the quality of the aid interventions; and (3) assess the operationalisation of the 2007 development policy, particularly in terms of sustaina-bility and effectiveness (Ministry for Foreign Affairs of Finland 2007a).

Section 1 provides an overview of Finnish development policy and development cooperation, Finnish development evaluation and evaluation meta-analyses, and other comparative analyses. In Section 2, this current Meta-analysis is explained, in terms of its purpose, objectives, scope, approach, methodology, and limitations. Section 3 discusses some elements of Finnish development evaluation practice. Section 4 describes some basic characteristics of the 33 evaluation reports sampled. Section 5 presents and discusses the findings on the quality of the evaluation reports. Section 6 presents and discusses the findings concerning the quality of the aid, or development interventions, i.e., programmes and projects, assessed in these reports. Section 7 con-siders what the information in this sample suggests regarding the implementation of the current development policy. Section 8 provides overall conclusions, and Section 9 provides recommendations.

1.1 Finnish Development Policies and Development Cooperation

Finnish Development Cooperation Policies Finland has had an active development cooperation programme for more than four decades. The Government has supported Official Development Assistance (ODA) since 1961, and since 1965 MFA has had offices dealing with development coopera-tion issues (OECD 2007).

Over the years, the Government of Finland has periodically revised its development cooperation policies and strategies. In 1993, the Government issued the first com-prehensive development policy, Finland’s Development Cooperation in the 1990s (Ministry for Foreign Affairs of Finland 1993). Subsequently, this policy was updated with the 1996 Government Decision-in-Principle on Finland’s Development Cooperation (Government of Finland 1996), the 1998 Government Decision-in-Principle on Finland’s Policy on Relations with Developing Countries (Government of Finland 1998), the 2001 Operationalisation of Development Policy Objectives in Finland’s International Development Cooperation (Government


of Finland 2001), and the 2004 Development Policy, Government Resolution (Ministry for Foreign Affairs of Finland 2004a). In October 2007, the Government of Finland adopted the current development policy, Development Policy Programme 2007: Towards a Sustainable and Just World Community (Ministry for Foreign Affairs 2007a). In addition to these major policy documents, the Government has also issued a number of more detailed policy and strategy documents to address specific themes, cross-cutting issu-es, and sectors. As several reviews and commentaries have noted, each successive de-velopment policy has built upon the previous ones (e.g., Ministry for Foreign Affairs of Finland 2004b).

The 1993 policy declared Finland’s commitment to (1) poverty reduction, (2) com-bating global environmental threats, and (3) promotion of social equality, democracy, and human rights in developing countries (Ministry for Foreign Affairs of Finland 1993). The 1996 policy reaffirmed these goals, and reconfirmed Finland’s commit-ment to gradually increase its support for development cooperation to 0.7 percent of gross national income (GNI). The 1996 policy also emphasized Finland’s support for the May 1996 Organisation for Economic Co-operation and Development (OECD) development cooperation strategy (Government of Finland 1996). The 1998 policy identified its policy goals as increasing global security, reducing poverty, promoting respect for human rights and democracy, preventing global environmental problems, and increasing economic interaction. It also discussed Finland’s role as a global ac-tor, with its support for the United Nations (UN) and its Global Agenda, as well as Finland’s development policy within the context of its membership in the European Union (EU) (Government of Finland 1998).

The 2001 policy document provided guidance for implementation of the previous policies. Establishing criteria for selection of long-term partner countries, as well as other partners, it identified 11 long-term partners: Egypt, Ethiopia, Kenya, Mozam-bique, Namibia, Nepal, Nicaragua, Peru, Zambia, Tanzania, and Viet Nam. It also discussed operational implications of its support for the UN systems, international financing institutions (IFI), and development cooperation with the European Com-munity (Government of Finland 2001).

With the 2004 development policy (Ministry for Foreign Affairs of Finland 2004a), Finland expressed its agreement with international goals for development, and stres-sed the importance of having a coherent development policy to ensure quality and ef-fectiveness in its development cooperation programme. It placed major emphasis on Finland’s support for the UN Millennium Declaration and Millennium Development Goals (MDGs), and stated that the main goal of Finnish development cooperation is to contribute to the eradication of extreme poverty, emphasizing the human rights-based approach (HRBA). The number of long-term partner countries was reduced to eight: Egypt, Namibia, and Peru were determined to be eligible for more diversified cooperation rather than grant aid, and therefore were dropped from the list. The po-licy stated that in each long-term partner country, cooperation would focus on a ma-ximum of three sectors, with the possibility of additional budget support for national


poverty reduction strategies. It also discussed need for greater policy coherence.

The 2007 policy emphasizes sustainable development and poverty reduction as major goals, and notes that sustainability contributes to overall poverty reduction and achie-vement of the MDGs. Sustainable development is seen as having three interrelated dimensions - ecological, economic, and social. It also addresses issues of regional security and political stability. The policy is built on three key principles – policy co-herence, complementarity, and effectiveness. It furthermore specifies the long-term partner countries, and other countries to be priorities for assistance, and outlines the different funding modalities, or instruments, through which the programmes can be implemented. In terms of promoting effectiveness, the policy notes the importance of intervening where Finland can “add value”, because of its comparative advanta-ges, in terms of expertise or other factors (Ministry for Foreign Affairs of Finland 2007a).

The 2007 development policy differs from the earlier ones in a number of significant ways. The current policy places increased emphasis on ecological sustainability, ma-nagement of natural resources, and responding to issues of climate change. Thus, whereas environmental issues had been perceived as a cross-cutting issue in the 2004 policy, they are now elevated to a more prominent position.

In the 2004 policy (Ministry for Foreign Affairs of Finland 2004a), the cross-cutting issues were:

• Promotion of the rights and the status of women and girls, and promotion of gender and social equity;

• Promotion of the rights of groups that are easily marginalized, particularly those of children, the disabled, indigenous peoples and ethnic minorities, and promotion of equal participation opportunities for them;

• Consideration of environmental issues.

In comparison, the 2007 policy’s (Ministry for Foreign Affairs of Finland 2007a) cross-cutting themes were:

• Promotion of the rights and status of women and girls, and promotion of gen-der and social equity;

• Promotion of the rights of groups that are easily excluded, particularly children, people with disabilities, indigenous peoples and ethnic minorities, and the pro-motion of equal opportunities for participation;

• Combating HIV/AIDS, as a health problem and as a social problem.

The concept of sustainability as applied in Finnish development cooperation has evolved over the years. In 1999, the MFA’s Guidelines for Programme Design, Monito-ring and Evaluation defined eight key elements of sustainable development: 1) policy environment; 2) economic and financial feasibility; 3) institutional capacity; 4) so-cio-cultural aspects; 5) participation and ownership; 6) gender; 7) environment; and 8) appropriate technology. One version of the key elements also added: 9) human resource development (Ministry for Foreign Affairs of Finland 1999). These ele-ments, or dimensions, of sustainability have been seen as integral to the success of


development interventions. Thus, over the years, many appraisals and evaluations conducted for MFA have specified in their Terms of Reference (TOR) that these elements should all be assessed.

In 2004 the Development Policy identified “sustainable development” as a key prin-ciple. It noted Finland’s support for the UN Millennium Declaration and the eight MDGs, one of which is to “ensure environmental sustainability.” Under this goal are three targets, one of which is to: “integrate the principles of sustainable development into country policies and programmes and reverse the loss of environmental resour-ces.” (Ministry for Foreign Affairs of Finland 2004a)

In the previous development policy programmes (2001 and 2004), Finnish value ad-ded had been, at least implicitly, considered a question of values, mostly shared with the like-minded Nordic+ donor community, i.e., Finland, Sweden, Norway, Denmark, the Netherlands, the United Kingdom and Ireland. In the operationalisation of the 2007 Development Policy Programme the emphasis has shifted towards increased participation by Finnish personnel and utilisation of Finnish know-how in Finnish-funded projects or programmes. In the 2007 policy, this issue of Finnish added value is seen as one of highlighting areas where Finland has comparative advantages, there-by contributing to increased effectiveness of development cooperation. It also cont-ributes to the objectives of aid harmonisation, wherein there is a division of labour among the donor community (Ministry for Foreign Affairs of Finland 2007a).

Finnish Development Cooperation The Government of Finland provides substantial development cooperation sup-port through a number of channels. The majority is handled by the MFA, although some funds go through other Ministries, such as the Ministry of Finance, Ministry of Employment and the Economy and others. In 2007 the Government of Finland’s disbursements for development cooperation amounted to € 547 million, and € 600 million in 2008, with a budget of € 731 million for 2009 (Ministry for Foreign Affairs 2009c).

In 1970 Finland made a commitment to increase its development cooperation fun-ding to 0.7 percent of GNI. This goal was achieved in 1991 (OECD 2007), but then funding had to be cut due to the severe economic recession in Finland. Finland reaffirmed in the 1996 development policy, that “the ultimate national target is still 0.7 percent of GNI” (Government of Finland 1996, p.1). In the 2004 development policy, the Government expressed its desire to reach this target by 2010 (Ministry for Foreign Affairs of Finland 2004a). The Government subsequently committed to increase its development support to 0.51 percent of GNI by 2010, and 0.7 percent by 2015 (Ministry for Foreign Affairs of Finland 2008b, 2009c). In early September 2009, MFA estimates suggested that Finland is on track to achieving the 2010 target.

Finnish development assistance is implemented through a wide range of different funding modalities, or instruments. Finland supports bilateral programmes and pro-


jects. It also provides bilateral support through small grants, called Local Coopera-tion Funds (LCFs), managed by the respective Embassies. Finland supports a range of different interventions by multilateral organisations, such as UN organisations, and multilateral international financial institutions (IFI), or development banks, both “centrally” from MFA headquarters, and in some cases, at the local embassy level. Finland also actively participates in the EU’s development activities. In many count-ries, Finland has engaged in multi-donor partnerships or funds, and general budget or sector support programmes. Finland supports Finnish non-governmental organisa-tions (NGOs) and NGO Foundations for activities in developing countries, as well as providing support to international non-governmental organisations (INGOs). MFA supports institutional collaboration with institutional cooperation instruments (ICI) and a new funding instrument that has been designed to support collaboration with educational institutions from 2010 onwards. The Finnish government also provides some private-sector funding through the Finnish Fund for Industrial Cooperation Ltd. (Finnfund), the business partnership programme, and concessional credits.

1.2 Evaluation of Finnish Development Cooperation

MFA is a member of OECD/DAC and its Evaluation Network. The MFA aims to follow international best practices in its development cooperation programme, inclu-ding its appraisals and evaluations.

MFA conducts a variety of different types of assessments and evaluations. The-se include various programme preparation reviews, pre-appraisals and appraisals of proposed activities, mid-term reviews (MTRs, or mid-term evaluations) of ongoing activities, other implementation reviews, and final as well as ex post evaluations of completed activities.

In the MFA the evaluation function is divided between (1) large centralised thematic evaluations, meaning those carried out by Development Evaluation (EVA-11), which is attached to the office of the Under-Secretary of State for Development, and (2) project- or programme-specific decentralised evaluations, which are done by the re-gional departments and units or in some cases by the embassies. In addition to the annual average of 5-7 central evaluations commissioned by EVA-11, the MFA com-missions a large number of evaluations, MTRs, appraisals and other assessments of aid, which target specific projects or topics. In 2008, 0.2 percent of Finnish develop-ment cooperation disbursements went to evaluation activities (Ministry for Foreign Affairs of Finland 2009c).

MFA partners with a variety of multilateral development organizations, thus it may participate in joint evaluations, or receive evaluation reports conducted by one or more of its development partners on behalf of the co-financing partners. Under the 2007 Evaluation Guidelines (Ministry for Foreign Affairs of Finland 2007b), the Department for Global Affairs was normally responsible for MFA’s participation in multilateral


evaluations, although in some cases it was delegated to the respective Embassy, such as for the evaluation of jointly-financed programmes or sector support. Following the MFA’s internal reorganisation in 2008, however, the former responsibilities of the Unit for Global Affairs were transferred to the Development Policy Department.

Concerned country (or region) programme staff, comprised of Embassy staff, desk officers, sectoral advisors, and other MFA headquarters staff, are responsible for ap-praising and evaluating development interventions for their respective geographical areas. Other MFA staff members are responsible for certain funding instruments, such as NGO grants, concessional credits, multilateral and EU interventions, and other global or international initiatives.

In December 2007, the MFA issued its Evaluation Guidelines: Between Past and Future (Ministry for Foreign Affairs of Finland 2007b), for test application. This document had been preceded by the 1985 Project Evaluation: Concept and Guidelines (Ministry for Foreign Affairs of Finland 1985) and the 1999 Guidelines for Programme Design, Monito-ring and Evaluation (Ministry for Foreign Affairs of Finland 1999). The new evaluation guidelines were developed and commented in 2007. Now, after about two years of testing, the final revised version will be prepared. These guidelines seek to harmonize MFA practices with those of the OECD/DAC and with the EU. They also incorpo-rate the principles expressed in the 2004 Finnish Development Cooperation Policy (Ministry for Foreign Affairs of Finland 2004a) and the 2005 Paris Declaration on Aid Effectiveness (High-Level Forum on Aid Effectiveness 2005).

The MFA is currently preparing a new programme and project preparation and imple-mentation manual, which will include guidance on appraisals and evolving current practices. Among the issues that have been recognized are the significant resource requirements (human resources, time, funding, etc.) required to undertake full ap-praisals. Therefore, provisions will be made to authorize and undertake simplified appraisals under certain conditions.

1.3 Comparative Analyses of Development Evaluations

Previous Meta-analyses Over the years, the MFA has undertaken a number of different assessments that re-view samples of development evaluation reports, such as sectoral evaluations, thema-tic evaluations, evaluations of specific aid instruments, country programme studies, or other synthesis reports. Two recent analyses have been done on the entire evaluati-on portfolio of both centralised and decentralised evaluations in 1994-95 and 2007.

Effects or Impacts? Synthesis Study on Evaluations and Reviews 1988 to Mid-1995 (Koponen J & Mattila-Wiro P 1996) examined 150 reports prepared during 1988-1995, but li-mited the analysis to 62 reports. This synthesis study was a major effort, with work conducted between July 1994 and September 1995. In addition to analyzing the do-


cumentation, the study undertook a large number of interviews and workshops with MFA personnel, and field trips to Tanzania and Nepal. As part of the study, the aut-hors rated the evaluation reports on the basis of relevance, effectiveness, efficiency, sustainability, cross-cutting issues, general approach, and structure of the evaluation reports, but they did not provide the ratings in their study report. The study focused primarily on the development projects and programmes, rather than on the evaluation reports themselves. In other words, the evaluation reports were used as source materi-als to assess the projects and programmes, but the authors also consulted a wide range of other project, programme and development documentation. The authors did not formally rate or score the development interventions, as they found it “impossible” to do so. The study report was a lengthy book of over 400 pages. The authors had no-ted that an even earlier comparative analysis had been done in the Review of Evaluation Reports on Finnish Development Projects in the 1980s (Hirstiö-Snellman P 1991).

The Meta-Analysis of Development Evaluations in 2006 (White P & Stenbäck T 2007) covered 29 reports done in 2006. Three were from the centralized Evaluation Unit, the remaining 26 were from the geographical units of the MFA. This assessment exa-mined the characteristics of the evaluation reports and their quality, reviewed the de-velopment projects or programmes rated in the reports, and examined how both the interventions and the evaluation reports conformed with the Finnish development assistance policy. The authors rated these evaluation reports using eight European Commission (EC) quality assessment criteria: 1) relevance; 2) appropriate design; 3) reliable data; 4) sound analysis; 5) credible findings; 6) valid conclusions; 7) helpful recommendations; and 8) clarity. They then rated the overall quality of the reports using a five-point scale. They discussed the evaluation reports and their ratings, as well as some common findings concerning the projects and programmes, and proposed recommendations concerning possible improvements to the evaluations.

In reference to the above reports, the TOR for this current Meta-analysis (Annex 1) states, “When the conclusions of these two meta-analyses are compared, fairly little improvement in terms of sustainability and impact can be observed.” In deciding to undertake a Meta-analysis of Development Evaluations in 2007 and 2008, the MFA wished to see whether or not any changes had occurred, as compared with the pre-vious two meta-analyses.

Purpose, Objectives and Scope of this Meta-analysis The purpose of this Meta-analysis is to draw out lessons learned from recent evaluati-on experience, identify good practices, and make those lessons more widely available, and to help MFA to better plan its evaluation activities and use the results of evalu-ations for system-wide and organisational learning and thus improve the quality of development assistance. The intended audience for this Meta-analysis includes MFA personnel and collaborators, other development practitioners, and other interested stakeholders.

This Meta-analysis has three specific objectives. First, it analyses the quality of evalua-


tion reports prepared in 2007-2008. Second, it assesses the overall quality of aid based of the findings of the evaluations. Third, it analyses how the 2007 Development Policy Programme (Ministry for Foreign Affairs of Finland 2007a) has been operationalised, with a particular focus on effectiveness and sustainability of results.

2 APPROACH, METHODOLOGY AND LIMITATIONS

2.1 Approach

Figure 1 presents an overview of the approach to the Meta-analysis. The analysis was conducted in three stages, first examining the quality of the reports, then examining the quality of the programme(s) or project(s) evaluated in each report, and finally analysing the entire set of reports, programmes and projects covered by this Meta-analysis, regarding the implementation of the 2007 development policy.

Figure 1 Key elements of the Meta-analysis.

A major focus of the evaluation has been on policy issues, i.e., examining how well the evaluations and projects and programmes conform with Finnish deve-lopment policies. In this regard, the Evaluation Team reviewed the TORs and evaluation reports to see how, for instance, the new policy is reflected in the TORs and resulting evaluation reports. The evaluation also explored how the policy principles were dealt with in the TORs and evaluation reports. Figure 2 outlines the approach to the analysis of the inclusion / recognition of the policy programme in the evaluations

This Meta-analysis has three specific objectives. First, it analyses the quality of evaluation reports prepared in 2007-2008. Second, it assesses the overall quality of aid based of the findings of the evaluations. Third, it analyses how the 2007 Development Policy Programme (Ministry for Foreign Affairs of Finland 2007a) has been operationalised, with a particular focus on effectiveness and sustainability of results.

2 APPROACH, METHODOLOGY AND LIMITATIONS

2.1 Approach Figure 1 presents an overview of the approach to the Meta-analysis. The analysis was conducted in three stages, first examining the quality of the reports, then examining the quality of the programme(s) or project(s) evaluated in each report, and finally analysing the entire set of reports, programmes and projects covered by this Meta-analysis, regarding the implementation of the 2007 development policy.

Figure 1 Key elements of the Meta-analysis. A major focus of the evaluation has been on policy issues, i.e., examining how well the evaluations and projects and programmes conform with Finnish development policies. In this regard, the Evaluation Team reviewed the TORs and evaluation reports to see how, for instance, the new policy is reflected in the TORs and resulting evaluation reports. The evaluation also explored how the policy principles were dealt with in the TORs and evaluation reports. Figure 2 outlines the approach to the analysis of the inclusion / recognition of the policy programme in the evaluations:

I. Quality of each Evaluation and its TOR • Does it respond to its TOR? • How does it address the 10 OECD/DAC evaluation (report) quality standards, i.e.,

relevance, purpose and objectives; scope; context; methodology; information sources; independence; ethics; quality assurance; relevance of evaluation results; and completeness?

II. Quality of each of the Development Interventions • What is the quality of the development intervention in terms of the five OECD/DAC

evaluation criteria, i.e., relevance, effectiveness, efficiency, impact and sustainability? • What is the quality of the intervention with respect to the following three additional

criteria: policy coherence and complementarity; Finnish added value; and data, indicators and monitoring

• Other assessments of intervention’s relative “success” and “failure”? • How the intervention implements the 2007 Development Policy?

III. Comparative analysis of set of development interventions analysed • How is the 2007 Development Policy being operationalised in these projects and

programmes? • Are there discernible common factors of successful interventions? of failures?

.


Figure 2 Analysis of realization of policy principles.

2.2 Methodology

The assessment was primarily as a desk review of the documentation provided (Annex 3). The Team reviewed the documentation, assessing each evaluation report against a number of key questions and criteria, which were based upon the Meta-analysis TOR, the OECD/DAC evaluation quality standards, the five OECD/DAC evaluation crite-ria and three additional criteria. The Team also interviewed a number of MFA staff and other key resource persons (Annex 2). Prior to the interviews, a general interview guide was prepared (Annex 4), but during the actual interviews the questions were adapted to fit the circumstances of each interview. The Team also identified supple-mentary information on the Internet. Further information regarding the Evaluation Team’s work is provided in Annex 8.

The Team prepared three evaluation tools. First, basic comparative data on all the reports was summarized and entered into an Excel spreadsheet. The data (Excel spreadsheet) included brief notes on items such as date of report, date of the pro-gramme or project document (if such information was available), type of report, country or regional, sector, whether the report was available in electronic form or only hard copy, whether the report TOR were available (as an annex to the report or a separate document), which organization managed the evaluation exercise (if not MFA), etc.

Second, the Team designed a form for recording qualitative and narrative observa-tions on each report (Annex 5). This form provided a consistent format for sum-marizing some key qualitative information on each evaluation report and the project or programme being appraised or evaluated. This form complements, and provides

Figure 1 Key elements of the Meta-analysis. A major focus of the evaluation has been on policy issues, i.e., examining how well the evaluations and projects and programmes conform with Finnish development policies. In this regard, the Evaluation Team reviewed the TORs and evaluation reports to see how, for instance, the new policy is reflected in the TORs and resulting evaluation reports. The evaluation also explored how the policy principles were dealt with in the TORs and evaluation reports. Figure 2 outlines the approach to the analysis of the inclusion / recognition of the policy programme in the evaluations:

I. Quality of each Evaluation and its TOR • Does it respond to its TOR? • How does it address the 10 OECD/DAC evaluation (report) quality standards, i.e.,

relevance, purpose and objectives; scope; context; methodology; information sources; independence; ethics; quality assurance; relevance of evaluation results; and completeness?

II. Quality of each of the Development Interventions • What is the quality of the development intervention in terms of the five OECD/DAC

evaluation criteria, i.e., relevance, effectiveness, efficiency, impact and sustainability? • What is the quality of the intervention with respect to the following three additional

criteria: policy coherence and complementarity; Finnish added value; and data, indicators and monitoring

• Other assessments of intervention’s relative “success” and “failure”? • How the intervention implements the 2007 Development Policy?

III. Comparative analysis of set of development interventions analysed • How is the 2007 Development Policy being operationalised in these projects and

programmes? • Are there discernible common factors of successful interventions? of failures?

Are these issues reflected in the reports ? What methods and sources of verification have been used? What has been achieved?

How sustainable development, principles and effectiveness have been addressed in the TORs and evaluation reports?

Achievement of Millennium

Development Goals and

Poverty Reduction

• Coherence with the 2007 Finnish policy;

• Consideration of the cross-cutting themes;

• Coherence with the policy of the recipient country

• Complementarities of Finnish aid within the overall development context of the partner country

• Coordination of the Finnish aid with other donors

Utilization of findings and recommendations


additional information concerning issues identified in the quality assessment form.

Third, the Team developed a quality assessment form (Annex 6) to rate the report against the OECD/DAC evaluation quality standards and rate the project or programme against the OECD/DAC and additional evaluation criteria. The quality assessment form has three sec-tions. The first section provides some basic data on the report and programme or project.

The second section, on the second page of the form, rates the report against the OECD/DAC evaluation (report) quality standards; on the basis of these ratings, an overall rating is made for the evaluation report quality. The Meta-analysis Team had developed this form on the basis of the OECD/DAC evaluation quality standards, and the supporting document that describes the standards in detail (OECD 2006). The rating form uses the ten key standards, as well as two to five additional items for further detail on each standard.

Figure 3 Summary example of evaluation report quality rating.

A summary example of the rating is provided in Figure 3. In the above example, “completeness” and “overall report quality” were both scored between “good” and “very good.” The rating scale used was based on the one used by the EC for its evalu-ation quality ratings (European Commission 2006). In some cases, the Meta-analysis Team reviewers gave an intermediate rating, i.e., good to very good. On the basis of the ratings for the individual criteria, the Team then assigned an overall rating for the indivi-dual report. Thus, as suggested in the example above, if ratings for the standards were split between good and very good, although some items could not or were not assessed, the overall quality of the report was considered to be “good to very good.”

The second section, on the second page of the form, rates the report against the OECD/DAC evaluation (report) quality standards; on the basis of these ratings, an overall rating is made for the evaluation report quality. The Meta-analysis Team had developed this form on the basis of the OECD/DAC evaluation quality standards, and the supporting document that describes the standards in detail (OECD 2006). The rating form uses the ten key standards, as well as two to five additional items for further detail on each standard.

Evaluation Report Quality

OECD/DAC Evaluation Quality Standards

Poor

Good

Very Good

NA

1. Rationale, purpose and objectives of the evaluation

X

2. Evaluation scope X 3. Context X 4. Evaluation methodology

X

5. Information sources X 6. Independence X 7. Evaluation ethics X 8. Quality assurance X 9. Relevance of the evaluation results

X

10. Completeness X X OVERALL REPORT QUALITY

X X

Figure 3 Summary example of evaluation report quality rating. A summary example of the rating is provided in Figure 3. In the above example, “completeness” and “overall report quality” were both scored between “good” and “very good.” The rating scale used was based on the one used by the EC for its evaluation quality ratings (European Commission 2006). In some cases, the Meta-analysis Team reviewers gave an intermediate rating, i.e., good to very good. On the basis of the ratings for the individual criteria, the Team then assigned an overall rating for the individual report. Thus, as suggested in the example above, if ratings for the standards were split between good and very good, although some items could not or were not assessed, the overall quality of the report was considered to be “good to very good.” When the overall results were summarized to discuss the set of reports, then individual overall ratings by report were converted to numerical scores, where very good = 3, good = 2, and poor = 1. The third section, on the third and fourth pages of the form, rates the quality of the aid intervention, and focuses on five key OECD/DAC evaluation criteria – relevance, effectiveness, efficiency, impact, and sustainability – as well as three other key criteria, which included (1) policy coherence and complementarity, (2) Finnish added value, and (3) data, indicators, and monitoring. Definitions of these criteria are provided in Annex 7. A summary example is shown in Figure 4.

Quality of the Aid Interventions

Aid Evaluation Criteria

Poor

Good

Very Good

NA

Relevance X Efficiency X Effectiveness X

Ratings: Poor = standard only partially fulfilled Good = standard mostly fulfilled Very good = standard entirely fulfilled NA = information not available, not applicable, or not assessed

Ratings: Poor = criteria only partially fulfilled Good = criteria mostly fulfilled Very good = criteria entirely fulfilled NA = information not available, not applicable, or not assessed


When the overall results were summarized to discuss the set of reports, then indivi-dual overall ratings by report were converted to numerical scores, which were very good = 3, good = 2, and poor = 1.

The third section, on the third and fourth pages of the form, rates the quality of the aid intervention, and focuses on five key OECD/DAC evaluation criteria – relevance, effectiveness, efficiency, impact, and sustainability – as well as three other key criteria, which included (1) policy coherence and complementarity, (2) Finnish added value, and (3) data, indicators, and monitoring. Definitions of these criteria are provided in Annex 7. A summary example is shown in Figure 4.

Figure 4 Summary example of project or programme quality rating.

The second section of the assessment form also noted whether or not these criteria had been specified in the evaluation TOR, and included questions such as did the report consider the Paris Declaration, did it identify assumptions regarding sustaina-bility, did it have baseline data or data disaggregated by gender, etc. to aid in rating the project or programme on the criteria.

2.3 Limitations of this Meta-analysis

The analysis aims to look at the overall patterns of the evaluations, programmes, and projects examined and provide illustrative examples of good practice and areas needing improvement. As the Meta-analysis Team does not know the total portfolio of appraisals and evaluations undertaken by the MFA, it is not possible to assess how representative is this sample. Nonetheless, the Meta-analysis Team believes that the sample provides a sufficient diversity in the reports, projects, and programmes to permit a useful analysis of some key issues. This work is limited by the information that the Meta-analysis Team had at its disposal for the analysis, in terms of the original documentation provided and additional supplementary information obtained, through interviews and other means.

In very few cases were individual evaluation reports accompanied by the respective project or programme document, or other documentation. As will be further discus-

Evaluation Report Quality OECD/DAC Evaluation Quality Standards

Poor

Good

Very Good

NA

1. Rationale, purpose and objectives of the evaluation

X

2. Evaluation scope X 3. Context X

4. Evaluation methodology

X

5. Information sources X

6. Independence X

7. Evaluation ethics X 8. Quality assurance X

9. Relevance of the evaluation results

X

10. Completeness X X

OVERALL REPORT QUALITY

X X

Figure 3 Summary example of evaluation report quality rating. A summary example of the rating is provided in Figure 3. In the above example, “completeness” and “overall report quality” were both scored between “good” and “very good.” The rating scale used was based on the one used by the EC for its evaluation quality ratings (European Commission 2006). In some cases, the Meta-analysis Team reviewers gave an intermediate rating, i.e., good to very good. On the basis of the ratings for the individual criteria, the Team then assigned an overall rating for the individual report. Thus, as suggested in the example above, if ratings for the standards were split between good and very good, although some items could not or were not assessed, the overall quality of the report was considered to be “good to very good.” When the overall results were summarized to discuss the set of reports, then individual overall ratings by report were converted to numerical scores, where very good = 3, good = 2, and poor = 1. The third section, on the third and fourth pages of the form, rates the quality of the aid intervention, and focuses on five key OECD/DAC evaluation criteria – relevance, effectiveness, efficiency, impact, and sustainability – as well as three other key criteria, which included (1) policy coherence and complementarity, (2) Finnish added value, and (3) data, indicators, and monitoring. Definitions of these criteria are provided in Annex 7. A summary example is shown in Figure 4.

Quality of the Aid Interventions

Aid Evaluation Criteria

Poor Good

Very Good

NA

Relevance X

Efficiency X Effectiveness X

Impact X

Sustainability X Coherence X

Finnish added value X

Monitoring X OVERALL AID QUALITY

X

Figure 4 Summary example of project or programme quality rating.

Ratings: Poor = standard only partially fulfilled Good = standard mostly fulfilled Very good = standard entirely fulfilled NA = information not available, not applicable, or not assessed

Ratings: Poor = criteria only partially fulfilled Good = criteria mostly fulfilled Very good = criteria entirely fulfilled NA = information not available, not applicable, or not assessed


sed in Section 5.1, for nine of the 33 reports, TOR for the evaluations were unavai-lable, so it was not possible to assess every report against its own TOR. Presumably the TOR had been prepared for the consultants, but they were not located in the document filing system and/or were not available to the Meta-analysis Team.

The availability and quality of documents and other information also limited the ability to assess the compliance with the development policy. All of the evaluation reports had various weaknesses and omissions – in terms of the evaluation quality standards and evaluation criteria for this Meta-analysis – thus all the information needed for a thorough analysis of the quality of the aid and the implementation of the development policy was not available. The two previous meta-analyses had noted the same problem.

Another significant limitation is that the sample of reports is relatively small yet quite varied, in terms of countries, sectors, aid instruments, and types of evaluations. As a result, it is difficult to draw any meaningful conclusions regarding possible differences in the quality of the evaluation reports and the quality of the aid, i.e., the projects or programmes by sector and/or by country, on the basis of such factors.

3 CURRENT PRACTICE IN FINNISH DEVELOPMENT EVALUATION

When considering the evaluations assessed in this Meta-analysis, it was noticed that not all reports live up to the ideal practice, as recommended in the established guide-lines. As further discussed below, the focus in the decentralised evaluations tends to be primarily on addressing management issues and decisions that need to be made. Broader functions of evaluations, such as promoting learning -- from both successes and failures -- and accountability, while strategically important in the long run may be less urgent in the short term.

Evaluations for different types of activities have different areas of emphasis. For appraisals and pre-appraisals, the focus is often on issues of relevance, feasibi-lity, and sustainability, or more narrowly on institutional arrangements and financial mechanisms. Among the reports assessed, there was clearly a difference in the scope of the TOR and subsequent appraisal reports, depending upon the type of project preparation and institutional arrangements. For certain aid instruments, such as co-financing with an international financial institution, there may be periodic prepara-tion reviews, done prior to formal appraisals, to keep the intervention active in the pipeline as the design and country negotiations are developed.

In the case of ex ante evaluations, such appraisals evaluate the intervention in advance, using standard OECD/DAC evaluation criteria – relevance, effectiveness, efficiency, impact, and sustainability. Furthermore, they should anticipate data needs for future evaluations, such as mid-term, final, ex post and/or impact evaluations, and look at whether or not the intervention has or will develop baseline data, defined indicators, and other plans for measuring the performance, achievements, and other impacts of


the intervention. None of the appraisals in this sample met these rigorous require-ments – and indeed, for none of them was the term “ex ante evaluation” used.

For appraisals of proposed projects or programmes where the MFA has been invol-ved in the formulation process, the appraisal often focuses on a final check of project feasibility, and perhaps finalizes institutional arrangements or financing mechanisms. For bilateral projects, the host government and Finland will normally have already agreed to collaborate on the particular intervention. If the project or programme preparation is not considered to be adequate, then generally steps are taken to rectify the weaknesses. In some cases, appraisal teams may be asked to assist in finalising the programme or project document, for example, preparing or revising a logical framework. For those interventions being undertaken with other donor institutions, there may be some degree of uncertainty still remaining, as to whether or not Finland should go ahead and participate. Then the assessment of issues such as relevance and coherence with Finnish Development Policy, or coherence within the country and sector in question, may be more important.

A very different situation arises when MFA has not been actively involved in the process, such as when an outside party prepares a funding proposal and submits it to MFA for consideration; examples include concessional credit projects, Finnish NGO and international NGO projects, or perhaps other multilateral or multi-donor interventions where Finland has not been involved in the programme preparation. In such cases, then the issues are much more open – not only should it be a well-de-signed proposal, but also how well does it fit with the development policy objectives and country programme? Do the proponents have the institutional capacity to carry out the project or programme being proposed?

For MTRs or evaluations, there are likely to be one or more focal areas. The MTR may focus on implementation, progress and performance to date, and identifying whether or not any improvements can be made in implementation to better ensure achievement of the project objectives. In some cases, the MTR Team may be tasked to assess whether the project may need an extension. If the project has had a slow start, then a no-cost extension may be considered, to give more time to achieve the objectives and disburse the funding. In some cases, additional funding may be con-sidered, or the MTR mission may be asked to consider whether another phase of the project is warranted. If so, then the MTR report may serve to guide the planning of the next phase. At times a more narrow and focused implementation review may be undertaken, which does not cover all the issues normally addressed in a MTR.

Although many projects anticipate a final evaluation, in practice final evaluations of bilateral projects seem to be somewhat optional. They seem to occur most often when the existing project may influence future programming, whether for a follow-on phase or project, or perhaps for replication elsewhere or on a regional scale. If a project has not been successful, or there is no interest in continuing in the sector or in the particular country, then a final evaluation is much less likely to take place. Similar


considerations may be applied to ex post evaluations, which seem to be relatively rare. In such cases, perhaps there are lessons that could otherwise have been learned, but they may go undetected and undocumented.

Evaluations may be further complicated when Finland is partnered with other donors to co-finance programmes or projects, and evaluation standards are not the same. Some donors, UN agencies for example, undertake annual reviews (but not MTRs) and only do an external final evaluation, following preparation of an internal project completion report.

It should be noted that although the MFA has project information databases and an archiving system, not all key documentation on development interventions is included or entered into the system. One problem cited by some staff members is difficulty in sending large electronic files from the Embassies to headquarters via the Internet. Another issue seems to be difficulty in retrieving documentation from the system, because of the way that it is coded into the system.

4 OVERVIEW OF THE EVALUATION REPORTS

Aid Instruments and Types of EvaluationsThe development evaluations studied are categorized as shown in Table 2. In this stu-dy, bilateral projects or programmes were the most common type of aid instrument evaluated, followed by multi-donor interventions and INGO proposals.

In terms of the types of evaluation reports in the sample, most of the reports had already been classified by the MFA as MTRs, final evaluations, ex post evaluations, or appraisals, or pre-appraisals. A few reports were classified by the Meta-analysis Team. In distinguishing between a final and ex post evaluation, the Meta-analysis Team con-sidered that an evaluation done within six months before or after the project comp-letion date to be a final evaluation, whereas a report done more than six months after completion to be an ex post evaluation. Thus, the evaluation of the biogas projects in Ethiopia (Report VIII) was considered to be an ex post evaluation. The implemen-tation reviews of the FOMEVIDAS project in Nicaragua (I) and the Palestine land administration project (XIII) were narrower than MTRs, so these two reports were classified as “other evaluations.”


Table 2 Aid instruments and types of evaluations.

Timing of Projects and ProgrammesThe current Finnish development policy was adopted in October 2007. Nine evalua-tions (of 11 projects) and two appraisals were completed prior to early October 2007 (36.1% of the 36 projects evaluated) (see Table 1). Although 22 reports were pre-pared after the adoption of 2007 Development Policy Programme, most concerned interventions were designed prior to the policy. Only for six of the appraised pro-grammes or projects had the design process begun after October 2007, i.e., the Sup-port to National Forestry Programme in Mozambique (XXVII) and the five INGO proposals for the Western Balkans (XXIX to XXXIII). All twenty of the evaluation reports (covering 23 projects and programmes) covered activities that were designed and approved prior to 2007 policy. Thus, 30 of the 36 interventions were designed before the approval of the 2007 current development policy.

Countries or Geographical RegionsThe 33 evaluations included five regional programmes – 1) a bilingual education pro-gramme for Andean countries (Bolivia, Ecuador, and Peru) (V), 2) a regional project on forest policy, economics-education and research in the Western Balkans (Albania, Bosnia and Herzegovina, Croatia, and Macedonia) (XVI), two regional INGO pro-jects for the Western Balkans – 3) one working in Albania, Bosnia and Herzegovina, Kosovo, Macedonia, Montenegro, and Serbia (XXIX), 4) the other in Albania, Bosnia and Herzegovina, Croatia, Montenegro, Serbia and Slovenia) (XXX) – and 5) the Lake Victoria Basin Commission partnership (Kenya, Tanzania, and Uganda) (XXVIII). One report (XVII) undertook a regional comparative analysis, and assessed and com-pared three different interventions in three Western Balkan countries (Serbia, Bosnia and Herzegovina, and Montenegro). The other projects and programmes assessed under this review covered 14 different individual countries. Thus altogether, this assessment looked at interventions in 24 countries.

be relatively rare. In such cases, perhaps there are lessons that could otherwise have been learned, but they may go undetected and undocumented. Evaluations may be further complicated when Finland is partnered with other donors to co-finance programmes or projects, and evaluation standards are not the same. Some donors, UN agencies for example, undertake annual reviews (but not MTRs) and only do an external final evaluation, following preparation of an internal project completion report.

It should be noted that although the MFA has project information databases and an archiving system, not all key documentation on development interventions is included or entered into the system. One problem cited by some staff members is difficulty in sending large electronic files from the Embassies to headquarters via the Internet. Another issue seems to be difficulty in retrieving documentation from the system, because of the way that it is coded into the system.

4 OVERVIEW OF THE EVALUATION REPORTS

Aid Instruments and Types of Evaluations The development evaluations studied are categorized as shown in Table 2. In this study, bilateral projects or programmes were the most common type of aid instrument evaluated, followed by multi-donor interventions and INGO proposals. In terms of the types of evaluation reports in the sample, most of the reports had already been classified by the MFA as MTRs, final evaluations, ex post evaluations, or appraisals, or pre-appraisals. A few reports were classified by the Meta-analysis Team. In distinguishing between a final and ex post evaluation, the Meta-analysis Team considered that an evaluation done within six months before or after the project completion date to be a final evaluation, whereas a report done more than six months after completion to be an ex post evaluation. Thus, the evaluation of the biogas projects in Ethiopia (Report VIII) was considered to be an ex post evaluation. The implementation reviews of the FOMEVIDAS project in Nicaragua (I) and the Palestine land administration project (XIII) were narrower than MTRs, so these two reports were classified as “other evaluations.” Table 2 Aid instruments and types of evaluations.

Evaluation Type Aid Instruments

Mid

-Ter

m

Rev

iew

Oth

er

evalu

ati

on

Fin

al

evalu

ati

on

Ex

po

st

evalu

ati

on

Pre

- ap

pra

isal

Ap

pra

isal

To

tal

P

roje

cts

or

Pro

gra

mm

es

Bilateral project or programme 8 1 7 1 3 20 Local Cooperation Funds (LCF) 1 1 Concessional credit 1 1 International non-governmental organisations (INGOs)

5 5

Multilateral: international development financing institution

1 1 1 3

Multilateral: UN agency 1 1 Multi-donor partnership / basket fund / sectoral support

2 1 2 5

Total 10 2 9 2 1 12 36

Timing of Projects and Programmes The current Finnish development policy was adopted in October 2007. Nine evaluations (of 11 projects) and two appraisals were completed prior to early October 2007 (36.1% of the 36 projects


It should be noted that some other linkages exist among the reports. Two “sister pro-jects” were included in the sample: both are part of the Integrated Development of the Eastern Nile watershed management initiative, one in Ethiopia (XXIV) and one in Sudan (XXV). For the five INGO proposals in the Western Balkans and Kosovo (XXIX to XXXIII), they were all submitted in response to one call for proposals, and through a framework agreement, one consultant consortia appraised all five pro-posals.

Figure 5 Reports by country, region and partner status.

Half (17 out of 33 reports, or 52%) of the evaluation and appraisal reports were for Finland’s long-term partner countries – Ethiopia, Kenya, Mozambique, Nepal, Nicaragua, Tanzania, and Viet Nam. This set included one appraisal for a regional project (Lake Victoria Basin Partnership, Report XXVIII) that includes two long-term partner countries – Kenya and Tanzania – and Uganda. None of the evaluations or appraisals in this assessment covered Zambia, the only long-term partner not included in the sample.

Just over one-third (14 out of 33 reports, or 42%) were for partner countries (or regions) recovering from (or still undergoing) violent crises, i.e., Western Bal-kans, Sudan, and the Palestine Territories. Afghanistan, a partner country recovering from violent crises, had no evaluations or appraisals in this 2007-2008 sample. The remainder of the evaluations and appraisals were for other countries or regions.

evaluated) (see Table 1). Although 22 reports were prepared after the adoption of 2007 Development Policy Programme, most concerned interventions were designed prior to the policy. Only for six of the appraised programmes or projects had the design process begun after October 2007, i.e., the Support to National Forestry Programme in Mozambique (XXVII) and the five INGO proposals for the Western Balkans (XXIX to XXXIII). All twenty of the evaluation reports (covering 23 projects and programmes) covered activities that were designed and approved prior to 2007 policy. Thus, 30 of the 36 interventions were designed before the approval of the 2007 current development policy. Countries or Geographical Regions The 33 evaluations included five regional programmes – 1) a bilingual education programme for Andean countries (Bolivia, Ecuador, and Peru) (V), 2) a regional project on forest policy, economics-education and research in the Western Balkans (Albania, Bosnia and Herzegovina, Croatia, and Macedonia) (XVI), two regional INGO projects for the Western Balkans – 3) one working in Albania, Bosnia and Herzegovina, Kosovo, Macedonia, Montenegro, and Serbia (XXIX), 4) the other in Albania, Bosnia and Herzegovina, Croatia, Montenegro, Serbia and Slovenia) (XXX) – and 5) the Lake Victoria Basin Commission partnership (Kenya, Tanzania, and Uganda) (XXVIII). One report (XVII) undertook a regional comparative analysis, and assessed and compared three different interventions in three Western Balkan countries (Serbia, Bosnia and Herzegovina, and Montenegro). The other projects and programmes assessed under this review covered 14 different individual countries. Thus altogether, this assessment looked at interventions in 24 countries. It should be noted that some other linkages exist among the reports. Two “sister projects” were included in the sample: both are part of the Integrated Development of the Eastern Nile watershed management initiative, one in Ethiopia (XXIV) and one in Sudan (XXV). For the five INGO proposals in the Western Balkans and Kosovo (XXIX to XXXIII), they were all submitted in response to one call for proposals, and through a framework agreement, one consultant consortia appraised all five proposals.

Figure 5 Reports by country, region and partner status.


SectorsThe sectoral breakdown of the projects and programmes evaluated is shown in Figu-re 3. Ten (28%), deal with education, and one (3%) deals with research, or innovati-on systems, development. Four (11%), have been classified as environmental sector (including a biogas project). Another nine (26%), are in other sectors (meteorology, water, watershed management, forestry), which also could be considered broadly to be environmental projects. The remainder were in other sectors. If one examines the interventions more closely, however, it is clear that several projects or program-mes overlap in different sectors, for example watershed management projects with rural development components.

Figure 6 Sectors of development interventions.

If one examines the sectoral breakdown of the evaluations and appraisals separately, then it is clear that the appraisals (of new activities) reflect the shift in the overall development policy programme to emphasize ecologically sustainable deve-lopment: 8 out of the 11 appraisals conducted after October 2007 are for projects or programmes in the environment or related sectors (meteorology, water, watershed management, or forestry). For the projects involving education, research (innovati-on systems), health, rural development, enterprise development, land administration, and vulnerable groups, all were ongoing or completed and evaluated in 2007 or 2008. One environmental sector project was evaluated in mid-2007. Regarding the two

Half (17 out of 33 reports, or 52%) of the evaluation and appraisal reports were for Finland’s long-term partner countries – Ethiopia, Kenya, Mozambique, Nepal, Nicaragua, Tanzania, and Viet Nam. This set included one appraisal for a regional project (Lake Victoria Basin Partnership, Report XXVIII) that includes two long-term partner countries – Kenya and Tanzania – and Uganda. None of the evaluations or appraisals in this assessment covered Zambia, the only long-term partner not included in the sample. Just over one-third (14 out of 33 reports, or 42%) were for partner countries (or regions) recovering from (or still undergoing) violent crises, i.e., Western Balkans, Sudan, and the Palestine Territories. Afghanistan, a partner country recovering from violent crises, had no evaluations or appraisals in this 2007-2008 sample. The remainder of the evaluations and appraisals were for other countries or regions. Sectors The sectoral breakdown of the projects and programmes evaluated is shown in Figure 3. Ten (28%), deal with education, and one (3%) deals with research, or innovation systems, development. Four (11%), have been classified as environmental sector (including a biogas project). Another nine (26%), are in other sectors (meteorology, water, watershed management, forestry), which also could be considered broadly to be environmental projects. The remainder were in other sectors. If one examines the interventions more closely, however, it is clear that several projects or programmes overlap in different sectors, for example watershed management projects with rural development components.

Figure 6 Sectors of development interventions.


meteorology projects, one completed in Mozambique underwent an ex post evaluation in 2008 and another one, proposed for Peru, was appraised in 2008. Similarly, for projects focusing on gender and governance issues, one was evaluated in 2008 and one appraised in mid-2007.

5 QUALITY OF THE EVALUATIONS

5.1 Quality of the Terms of References of the Evaluation Reports

The starting point for any evaluation is its TOR, which constitute the substantive – and contractual – basis for the work. The TOR constitutes the agreement between the client, in this case the MFA, and the consultant for the work to be done, in terms of the scope, key issues to be addressed, and the approach and methodology to be followed. It is extremely important to have high-quality TOR to achieve a high-quality evaluation.

Normally, the TOR for the evaluation report is provided with the report itself, as an annex. In this sample, however, just over half (18) reports covering 19 interventions had TOR annexed, and for six reports covering eight projects and programmes, TOR were provided to the Meta-analysis team as a separate document.

For one-quarter (nine) of the programmes or projects, the TOR were not available to the Meta-analysis Team and thus it was not possible to compare the report with its TOR. Eight of the nine reports missing TORs were appraisals or pre-appraisals, and one was an evaluation. In four cases (Reports IV, XXIV, XXV, and XXVI), a series of reports were available for a particular project or programme, but the final report – which was the one assessed in this Meta-analysis – did not contain TOR. Two of these cases included pre-appraisal or appraisal missions conducted by the World Bank (WB), wherein Finland provided one consultant to join the team. For these four cases, presumably the TOR had been prepared for the consultants, but they were not located in the document filing system and/or were not provided to the Meta-analysis Team. Five INGO project proposals (XXIX, XXX, XXXI, XXXII, and XXXIII) were all appraised under one framework agreement and TOR, which was not provi-ded to the Meta-analysis Team.

The TOR for those Evaluations examined varied in their quality and comprehen-siveness. Different evaluations and appraisals had different purposes, and some were narrower in scope, and focused on only a few specific issues. Other TORs stood out as very well-written, with thorough background descriptions and very specific questions, demonstrating a very good knowledge of the intervention, whereas others did not provide much detail regarding the project or programme to be examined. Examples of very well-written TORs in our sample include those for the MTRs of the Andean Regional Project of Bilingual Intercultural Education (EIBAMAZ) (V) and the Support to Rural Development in Zambézia Province,


Mozambique (PRODEZA) (XI).

Often the TORs include a very long list of issues to be covered, with not only the general criteria, but specific questions under those criteria and other special issues to be addressed. One evaluation report noted, for example, that the team had been asked to address 35 questions. Given that evaluations are inevitably limited in time and resources, then it often becomes difficult to address all the issues adequately in the time available. When the TOR contains too many items, the evaluation team must then choose how to interpret the TOR and which issues to concentrate on.

The Meta-analysis Team examined the available TORs, to assess the evaluation crite-ria specified.

Table 3 Criteria specified in the TOR for assessing the projects and programmes.

*Note: the data above includes the 27 projects or programmes covered in the 24 reports for which the Meta-analysis Team obtained the TOR.

The OECD/DAC (2006) recommends that evaluations of development interven-tions focus on five key criteria: relevance, effectiveness, efficiency, impact, and sus-tainability. For 17 projects or programmes, the report TOR specified that all five OECD/DAC criteria be evaluated. Sixteen of these reports were evaluations, only one was an appraisal. In this Meta-analysis sample, appraisals focused most often on issues of relevance, feasibility, and sustainability.

In terms of sustainability, it was most common to find reference in the evaluation report TOR, if not to all, to at least some of the eight dimensions of sustainability that were outlined in the 1999 MFA Guidelines for Programme Design, Monitoring and Evaluation (Ministry for Foreign Affairs of Finland 1999):

1) policy environment; 2) economic and financial feasibility;

The Meta-analysis Team examined the available TORs, to assess the evaluation criteria specified. Table 3 Criteria specified in the TOR for assessing the projects and programmes.

Types of Development Evaluations Criteria specified in Report TOR

Mid-term evaluations

Other reports

Final or ex post evaluations

Appraisals or pre-appraisals

TOTAL

Relevance 10 0 8 3 21 (78%) Effectiveness 10 2 9 1 22 (81%) Efficiency 10 2 8 2 21 (78%) Impact 9 0 7 2 18 (67%) Sustainability 10 1 9 4 24 (89%) Coherence 8 1 5 4 18 (67%) Finnish added value 1 0 2 1 4 (15%) Data, indicators, monitoring

2 1 5 2 10 (37%)

Total projects and programmes with report TOR*

10 2 10 5 27 (100%)

Total projects and programmes

10 2 11 13 36

*Note: the data above includes the 27 projects or programmes covered in the 24 reports for which the Meta-analysis Team obtained the TOR. The OECD/DAC (2006) recommends that evaluations of development interventions focus on five key criteria: relevance, effectiveness, efficiency, impact, and sustainability. For 17 projects or programmes, the report TOR specified that all five OECD/DAC criteria be evaluated. Sixteen of these reports were evaluations, only one was an appraisal. In this Meta-analysis sample, appraisals focused most often on issues of relevance, feasibility, and sustainability. In terms of sustainability, it was most common to find reference in the evaluation report TOR, if not to all, to at least some of the eight dimensions of sustainability that were outlined in the 1999 MFA Guidelines for Programme Design, Monitoring and Evaluation (Ministry for Foreign Affairs of Finland 1999): 1) policy environment; 2) economic and financial feasibility; 3) institutional capacity; 4) socio-cultural aspects; 5) participation and ownership; 6) gender; 7) environment; and 8) appropriate technology. In terms of specifying guidelines for the evaluation reports, TOR referred most frequently to the MFA’s 1999 Guidelines for Programme Design, Monitoring and Evaluation (Ministry for Foreign Affairs of Finland 1999). Relatively few of the TORs for the later reports referred to the MFA’s 2007 Evaluation Guidelines: Between Past and Future (Ministry for Foreign Affairs of Finland 2007b). Similarly, there were few references made to the EU or OECD/DAC evaluation guidelines or criteria. 5.2 Quality of the Evaluation Reports


3) institutional capacity; 4) socio-cultural aspects; 5) participation and ownership; 6) gender; 7) environment; and 8) appropriate technology.

In terms of specifying guidelines for the evaluation reports, TOR referred most fre-quently to the MFA’s 1999 Guidelines for Programme Design, Monitoring and Evaluation (Ministry for Foreign Affairs of Finland 1999). Relatively few of the TORs for the later reports referred to the MFA’s 2007 Evaluation Guidelines: Between Past and Future (Ministry for Foreign Affairs of Finland 2007b). Similarly, there were few references made to the EU or OECD/DAC evaluation guidelines or criteria.

5.2 Quality of the Evaluation Reports

In rating the quality of the evaluation reports, the Meta-analysis Team relied primarily upon the information contained in the reports themselves and in their TORs. In some cases, MFA personnel had rated the report more highly, which could be due to the report addressing certain specific needs of the MFA. In some cases the Meta-analysis Team found that a report was missing important information, so the Team did not feel that the report provided enough information to assess certain issues, and thus did not rate it as highly as did MFA colleagues. Such information, however, was often available or known to the MFA staff, either through other documentation or through their knowledge of the intervention.

OECD/DAC Evaluation Quality Standards The Meta-analysis Team endeavoured to assess the evaluation reports against the ten OECD/DAC evaluation quality standards: 1) rationale, purpose and objectives of the evaluation; 2) evaluation scope; 3) context; 4) evaluation methodology; 5) information sources; 6) independence; 7) evaluation ethics; 8) quality assurance; 9) relevance of the evaluation results; and 10) completeness (OECD 2006). This effort proved to be very difficult. The information needed to assess these issues was not in the evaluation reports. Some of the information may be known by the evaluation manager, but was not available to the Meta-analysis Team.

Rationale, Purpose and Objectives of the Evaluation In terms of the first criterion, most evaluations have a brief statement of the ratio-nale, purpose and objectives, typically taken from the TOR, but occasionally such information is not even present – the report simply jumps directly into a presentation of the findings and analysis. More thorough discussions, such as indications of the audience for the report, were not common.

Evaluation Scope For the second criterion, the evaluation scope, intervention logic, evaluation criteria and evaluation questions, generally information was available on the overall scope of


the evaluation. Some reports assessed the intervention logic, especially the logical framework and indicators, whereas other did not mention it. As will be discussed in Section 7, the evaluation reports had varying coverage of the five key OECD/DAC evaluation criteria – relevance, effectiveness, efficiency, impact, and sustainability. As for any specific questions for the evaluation, generally the evaluation reports dealt with them. The major factor concerning the scope of the evaluation seemed to be whether or not OECD/DAC evaluation criteria or specific issues had been explicitly mentioned in the TOR. If so, they were generally addressed.

Context Information on the third criterion, overall context for the intervention – in terms of development and policy context, institutional context, socio-political context, and implementation arrangements – was variable. Most often the issues were dealt with not in terms of context, but rather in terms of assessments elsewhere in the report. For example, the development and policy context may be considered in terms of relevance and policy coherence, rather than in terms of the overall context. Relati-vely few evaluations considered the overall context for development, the overall aid situation in the sector or the ability of government to finance its own development objectives. Similarly, institutional issues and implementation issues are often discussed in the reports, but specifically focused on the intervention, rather than the broader context. The socio-political context is often only mentioned when it is problematic, such as political disturbances influencing project implementation, i.e., in Palestine (XII), or political elections leading to a change in government, thereby possibly thre-atening sustainability due to a change in MFA personnel, i.e., in Bosnia and Herzego-vina (XVII-B). One report that dealt with this issue in depth was the evaluation of the Kenya Gender and Governance Programme (IX) – but that programme aimed to influence the political sphere by, among other goals, promoting the election of more women to political office.

Evaluation Methodology The fourth criterion, evaluation methodology, examines whether or not the report has explained its evaluation methodology, means of assessment of results, consulta-tion with relevant stakeholders, sampling, and the evaluation team itself. Here again, the reports had differing results. For a number of reports, it was difficult to ascertain when the evaluation had been carried out, and the level of effort, i.e., total number of person-days. While some reports explained their methodology, usually they did not discuss how the results were assessed, nor did they explain any sampling, such as choice of field sites to visit, if it occurred. In only a few of the reports were there discussions of methodological issues, approaches, difficulties, or limitations. Some re-ports mentioned stakeholders consulted, in terms of a list of persons contacted and/or a mission itinerary or agenda. For most of the evaluations, time was extremely limited, and that limited the opportunities for field trips, stakeholder workshops, or beneficiary surveys. For example, the MTR team for the Andean bilingual education project (V) was asked to assess the project with only two weeks for field visits in three different countries.


While normally the reports mentioned the evaluation team, often by providing the names of the author(s) of the report, in one-third (11 out of 33) of the reports, the team members or authors were not identified – only the consulting company that provided the personnel was named. In one report (XX), the author’s relevant backg-round for undertaking the assignment was mentioned, which provided the reader with confidence that the evaluator was well-qualified to undertake the assignment. Another report, an appraisal of the Finnish Support to the National Forest Program in Mozambique (XXVII), did not give the author’s name, but did mention some of the author’s recent assignments in terms of how they contributed to his understan-ding of the sector. According to a MFA sector advisor, however, this consultant had previously served as the Director of Forestry for Mozambique, which undoubtedly explains, in part, his comprehensive view of the sector. In report concerning the education sector in South Africa (VI), it was mentioned that the team leader had par-ticipated in a parallel mission for Sweden, after undertaking the evaluation for MFA, so some additional information from that assignment had been incorporated into the final version of the report.

Information Sources As for the fifth criterion, transparency, reliability, and accuracy of information sour-ces, most reports contained a list of references or documentation reviewed, and a list of persons met or consulted during the mission, but a surprising number did not. One report (XIX) questioned the accuracy of some of the government stake-holders’ comments regarding the project, noting that a number of them had worked for the project as consultants and thus had a vested interest in providing a positive assessment of the project. Some reports commented on the poor quality of data for assessing the project, or sector, performance. Quite a number of the reports, howe-ver, provided analysis and drew conclusions without providing supporting data. For example, a report may have mentioned that the project contributed substantially to human resource development, through on-the-job, in-service and other training, but may not have mentioned how many person-days of training were provided, what pro-portion of the staff or other beneficiaries were trained, whether or not the trainees included women as well as men, and so forth.

Independence, Evaluation Ethics and Quality Assurance The next three criteria – independence, evaluation ethics, and quality assurance – were virtually impossible to assess on the basis of the evaluation reports. The standard specifies that evaluators be free and independent vis-à-vis the stakeholders, and eva-luations be conducted as a free and open process. Normally there is no information on these issues in the reports. In one case (XVII), however, the TOR mentioned that the evaluation team leader had been involved in the early stages of implementation of one of the three projects being evaluated, and described the steps taken to ensure no conflict of interest in the evaluation. In that evaluation, all interviews were done by and discussed among the entire team, to avoid any possible bias.


It is difficult to assess whether or not the evaluations were done in a professional and ethical manner, from the information in the reports. For one report that evaluated biogas projects in Ethiopia (VIII), however, the overall report quality was poor, and thus it could be concluded that the evaluator did not do the work in a professional manner, and perhaps was totally inexperienced in doing evaluations of development interventions. In this particular case, comments from a stakeholder meeting at which the draft report was discussed were appended, and several participants commented on the poor quality of the evaluation. None of the reports mentioned whether or not any disagreements among the team had occurred, although one report noted that the two team members agreed on the report contents.

Virtually no information was available on quality control for the reports, beyond some reports commenting that stakeholder feedback on the draft report had been incorpo-rated into the final report. Only a few reports annexed stakeholder comments on the draft report, such as comments of a stakeholder workshop on the Ethiopia biogas evaluation (VIII). For the appraisal of the gender programme in Nicaragua (XXI), the authors annexed MFA comments on their draft report and the authors’ reply to those comments. One final evaluation report in Viet Nam (XIV) was commissioned by the Evaluation Group of the United Nations Industrial Development Organizati-on (UNIDO), thus one assumes that the Evaluation Group provided quality control on the final contents of the report before it was printed, but no information on this process is mentioned in the report. For some of the larger multi-donor programmes or funds, such as the FED common NGO pool fund in Nicaragua (II), the rural sec-tor programme in Nicaragua (III), the Gender and Governance Programme in Kenya (IX), or the Nepal Education for All Programme (XV), the Finnish Embassy repre-sentatives in the donor groups were involved in management of the evaluation pro-cess and quality control. In discussions with MFA desk officers, they mentioned that for a couple of the evaluations, they had needed to provide considerable guidance to the evaluation team. None of the reports mentioned whether or not the consultant companies themselves had any internal processes for quality control.

Relevance of the Evaluation Results The ninth criterion pertains to relevance of the evaluation results, in terms of formulati-on of the findings, recommendations and lessons learned, completion of the evaluation within the allotted time and budget, and subsequent use(s) of the evaluation findings. Most of the reports presented findings and recommendations, but relatively few discus-sed lessons learned. For one appraisal of a gender project in Nicaragua (XXI), however, the appraisal team had been asked to not only appraise the programme document, but also to work on revising and finalising it. That team did not provide any conclusions or recommendations its short appraisal report, but rather incorporated these issues into the revision of the programme document (which the Meta-analysis Team did not re-ceive). In some reports it was apparent that the evaluation had been conducted in the allotted time, and in others it was apparent that the report was finalised some time after the evaluation mission had been conducted. No information was available on whether or not the evaluations were completed within their original budgets.


In terms of subsequent use of the evaluation report, after it was completed – this information is not provided in any of the evaluation reports, and it would be difficult for the original authors to address the issue, other than to make recommendations. In discussing various reports with a number of MFA desk officers and sectoral advisors, however, it was clear that some of the reports had been very important in reorienting the programme or project being evaluated, or provided important guidance for the design of the subsequent phase. For MTRs, follow-up to the report is normally done by the Supervisory Board, Steering Committee, or other relevant body overseeing the project or programme. Often they may accept part, but not necessarily all, of the evaluation report recommendations. Clearly issues on how the evaluation mission was conducted (in terms of time and budget) and subsequent use and follow-up of the evaluation report are matters that the Evaluation Managers could better assess.

Completeness The final criterion concerns the completeness of the report, i.e., whether or not the evaluation questions were answered, distinctions among conclusions, lessons learned and recommendations, clarity of the analysis, and clarity of the summary. Generally the evaluation reports were considered to be of good quality in this regard.

In fairness to the evaluation report authors, it must be reiterated that they had not been specifically asked to address the OECD/DAC evaluation quality criteria. No-netheless, it is difficult to comprehend how some evaluation teams would omit some very basic elements that should be part of any report, such as the authors’ names, terms of reference for the evaluation, references consulted, or persons met during the mission.

Overall Quality of the Evaluation Reports The Meta-analysis Team rated the overall quality of the evaluation reports as generally good (see Table 4). In most cases, the reports responded reasonably well to their res-pective TOR. In some cases, the evaluation reports went beyond their TOR, such as assessing the intervention according to some or all of the five OECD/DAC criteria even when they were not all mentioned in the TOR.

Table 4 Overall evaluation report quality. (Ratings based on OECD/DAC evaluati-on quality standards).

cases, the evaluation reports went beyond their TOR, such as assessing the intervention according to some or all of the five OECD/DAC criteria even when they were not all mentioned in the TOR. Table 4 Overall evaluation report quality. (Ratings based on OECD/DAC evaluation quality standards).

In terms of the OECD/DAC evaluation quality standards, the following reports were rated most highly: Very good:

• the MTR of the Multi-donor Fund for Equality and Sexual and Reproductive Rights (FED) in Nicaragua (II);

• the final evaluation of the Kenya Gender and Governance Programme ) in Kenya (IX); • the final evaluation of the Entrepreneurship Development Programme for Women in Food

Processing in Central Viet Nam (XIV); and • the appraisal of the Hydro-Meteorology Project in Peru (XXII).

Good to very good: • the MTR of the Sector Wide Support Programme for Rural Development (PRORURAL) in

Nicaragua (III); • the MTR of the Support to Rural Development in Zambézia Province, Mozambique

(PRODEZA) (XI); • the MTR of the Nepal Education for All (EFA) Programme (XV); • the MTR of the Forest Policy and Economics Education and Research (FOPER) Programme in

the Western Balkans (XVI); • the evaluations (two final and one mid-term) of three education projects in the Western

Balkans (XVII); • the final evaluation of the Environmental GIS project in Montenegro (XVIII); and • the appraisal of the Finnish Support to National Forest Programme (SNFP) in Mozambique

(XXVII). When considering the 11 best evaluation reports, several key factors stand out. Four reports were major evaluations conducted of multi-donor funds or sector programmes, i.e., the Multi-donor Fund for Equality and Sexual and Reproductive Rights (FED) in Nicaragua (II), the Kenya Gender and Governance Programme, Phase II (IX), the Sector Wide Support Programme for Productive Rural Development (PRORURAL) in Nicaragua (III), and the Nepal Education for All (EFA) Programme (XV). In these cases, while generally the evaluation was managed by one donor on behalf of the group, all the donors had contributed to the development of the TOR, the selection of consultants, and review of the draft reports. Thus, the donors were quite engaged in the management of the evaluation. Furthermore, as these programmes or funds were substantial in size, the evaluations had been appropriately resourced, with a larger-sized team undertaking the mission for anywhere from one to two and a half months, and substantial time available for document review, stakeholder consultations, and field visits. It should be noted, however, that although larger programmes tend to have better resourced evaluations, programme size alone is not necessarily a guarantee of quality of the evaluation report. The largest programme evaluated was the Nepal Education for All (EFA) Programme (2004-09), a multi-donor programme that had five donors and initial commitments of US$ 664 million, of which

Report Quality (Number of Reports Rated) Type of Report

Total Number of Reports

Poor (1.0 to 1.5)

Good (2.0)

Good to Very Good (2.5 to 2.75)

Very Good (3.0)

Average Rating

Evaluations 20 2 9 6 3 2.2 Appraisals 13 11 1 1 2.1 Total 33 2 20 7 4 2.2


In terms of the OECD/DAC evaluation quality standards, the following reports were rated most highly:

Very good:• the MTR of the Multi-donor Fund for Equality and Sexual and Reproductive

Rights (FED) in Nicaragua (II);• the final evaluation of the Kenya Gender and Governance Programme ) in

Kenya (IX);• the final evaluation of the Entrepreneurship Development Programme for Wo-

men in Food Processing in Central Viet Nam (XIV); and • the appraisal of the Hydro-Meteorology Project in Peru (XXII).

Good to very good:• the MTR of the Sector Wide Support Programme for Rural Development

(PRORURAL) in Nicaragua (III); • the MTR of the Support to Rural Development in Zambézia Province, Mo-

zambique (PRODEZA) (XI);• the MTR of the Nepal Education for All (EFA) Programme (XV);• the MTR of the Forest Policy and Economics Education and Research (FO-

PER) Programme in the Western Balkans (XVI);• the evaluations (two final and one mid-term) of three education projects in the

Western Balkans (XVII); • the final evaluation of the Environmental GIS project in Montenegro (XVIII);

and • the appraisal of the Finnish Support to National Forest Programme (SNFP) in

Mozambique (XXVII).

When considering the 11 best evaluation reports, several key factors stand out. Four reports were major evaluations conducted of multi-donor funds or sector program-mes, i.e., the Multi-donor Fund for Equality and Sexual and Reproductive Rights (FED) in Nicaragua (II), the Kenya Gender and Governance Programme, Phase II (IX), the Sector Wide Support Programme for Productive Rural Development (PRO-RURAL) in Nicaragua (III), and the Nepal Education for All (EFA) Programme (XV). In these cases, while generally the evaluation was managed by one donor on behalf of the group, all the donors had contributed to the development of the TOR, the selection of consultants, and review of the draft reports. Thus, the donors were quite engaged in the management of the evaluation. Furthermore, as these programmes or funds were substantial in size, the evaluations had been appropriately resourced, with a larger-sized team undertaking the mission for anywhere from one to two and a half months, and substantial time available for document review, stakeholder con-sultations, and field visits.

It should be noted, however, that although larger programmes tend to have better resourced evaluations, programme size alone is not necessarily a guarantee of quality of the evaluation report. The largest programme evaluated was the Nepal Educa-

cases, the evaluation reports went beyond their TOR, such as assessing the intervention according to some or all of the five OECD/DAC criteria even when they were not all mentioned in the TOR. Table 4 Overall evaluation report quality. (Ratings based on OECD/DAC evaluation quality standards).

In terms of the OECD/DAC evaluation quality standards, the following reports were rated most highly: Very good:

• the MTR of the Multi-donor Fund for Equality and Sexual and Reproductive Rights (FED) in Nicaragua (II);

• the final evaluation of the Kenya Gender and Governance Programme ) in Kenya (IX); • the final evaluation of the Entrepreneurship Development Programme for Women in Food

Processing in Central Viet Nam (XIV); and • the appraisal of the Hydro-Meteorology Project in Peru (XXII).

Good to very good: • the MTR of the Sector Wide Support Programme for Rural Development (PRORURAL) in

Nicaragua (III); • the MTR of the Support to Rural Development in Zambézia Province, Mozambique

(PRODEZA) (XI); • the MTR of the Nepal Education for All (EFA) Programme (XV); • the MTR of the Forest Policy and Economics Education and Research (FOPER) Programme in

the Western Balkans (XVI); • the evaluations (two final and one mid-term) of three education projects in the Western

Balkans (XVII); • the final evaluation of the Environmental GIS project in Montenegro (XVIII); and • the appraisal of the Finnish Support to National Forest Programme (SNFP) in Mozambique

(XXVII). When considering the 11 best evaluation reports, several key factors stand out. Four reports were major evaluations conducted of multi-donor funds or sector programmes, i.e., the Multi-donor Fund for Equality and Sexual and Reproductive Rights (FED) in Nicaragua (II), the Kenya Gender and Governance Programme, Phase II (IX), the Sector Wide Support Programme for Productive Rural Development (PRORURAL) in Nicaragua (III), and the Nepal Education for All (EFA) Programme (XV). In these cases, while generally the evaluation was managed by one donor on behalf of the group, all the donors had contributed to the development of the TOR, the selection of consultants, and review of the draft reports. Thus, the donors were quite engaged in the management of the evaluation. Furthermore, as these programmes or funds were substantial in size, the evaluations had been appropriately resourced, with a larger-sized team undertaking the mission for anywhere from one to two and a half months, and substantial time available for document review, stakeholder consultations, and field visits. It should be noted, however, that although larger programmes tend to have better resourced evaluations, programme size alone is not necessarily a guarantee of quality of the evaluation report. The largest programme evaluated was the Nepal Education for All (EFA) Programme (2004-09), a multi-donor programme that had five donors and initial commitments of US$ 664 million, of which

Report Quality (Number of Reports Rated) Type of Report

Total Number of Reports

Poor (1.0 to 1.5)

Good (2.0)


Very Good (3.0)

Average Rating

Evaluations 20 2 9 6 3 2.2 Appraisals 13 11 1 1 2.1 Total 33 2 20 7 4 2.2


tion for All (EFA) Programme (2004-09), a multi-donor programme that had five donors and initial commitments of US$ 664 million, of which donors provided US$ 150 million including US$ 14 million from Finland (World Bank 2009). One of the MFA sectoral advisors noted that for the Nepal EFA Programme, whereas the MTR (XV), which the Embassy of Finland had managed on behalf of the donor group, was of very high quality, the subsequent final evaluation report was disappointing in comparison.

Another noteworthy report was on the project supporting small-scale women entrep-reneurs in Viet Nam, commissioned by UNIDO (XIV). This project had been imple-mented by UNIDO, with Finland serving as the sole donor. As is standard practice with UN agencies, the project had annual reviews, then project management prepared a draft project completion report prior to the arrival of the evaluation team to under-take the final evaluation. Quality control was provided by the UNIDO Evaluation Group. The report also contained six specific performance rating criteria – relevance, effectiveness, efficiency, impact, ownership and sustainability – in terms of project design, and then separately for implementation. Such a performance rating approach is, again, common practice in UN evaluation reports. This report was very compre-hensive, with adequate information on the project, such that the report can be fully understood as a stand-alone document. It also included a glossary of terms and maps of the project sites, and was published in UNIDO’s series of evaluation reports.

Some other very good reports were smaller ones, done of bilateral programmes or projects, where the evaluation team members were highly knowledgeable about the issues and/or countries concerned, and provided a good overview of the project within its overall context, and had a thorough knowledge of Finnish development policies as well as local policies. For example, a very short evaluation, conducted in one week’s time, which was nonetheless well done, was that conducted of the Envi-ronmental GIS Project in Montenegro (XVIII). In this particular case, one of the report co-authors was a MFA sectoral advisor, who was already quite familiar with the project.

One appraisal report of a water supply, sanitation and hygiene (WASH) project in Ethiopia (XXVI) was very provocative. It considered the proposed intervention in a much broader context, looking at how it could serve as a pilot model for the entire country, and suggested some rather major restructuring of the project, such as dec-reasing the amount of long-term technical assistance (TA), increasing short-term TA and increasing the geographical coverage of the project. This particular appraisal was done by two consultants who seemed to be highly knowledgeable about the sector and activities of other development partners in the country – and the international networks, but less knowledgeable about Finnish development cooperation.

One important issue affecting the quality of the evaluation reports is their timing, with respect to the interventions being assessed. For example, in two cases, South Africa (VI) and Nepal (XV), evaluations were made of education programmes while


the schools were on vacation. Therefore, when the evaluation teams went to the field, they were not able to visit any schools in session, but only to talk with local officials and other stakeholders.

Concerning the two evaluation reports rated as poor, it is seems likely that the most important factors were that the evaluators were inexperienced and the TOR did not provide adequate guidance. As previously discussed, the biogas evaluation (VIII) aut-hor seemed inexperienced in preparing an evaluation of a development intervention. In the final evaluation of innovation systems in South Africa (VII), the TOR provided limited guidance on issues to be addressed. Rather, it specified that the consultant was to propose an evaluation plan, but the Meta-analysis Team had no information on what type of plan, if any, was provided. In this case, the TOR mentioned that the evaluation itself was to be done with a Finnish consultant and a local government team, to develop local capacity in conducting evaluations, but there was no informati-on in the report as to whether this approach had actually been attempted or not.

When the issue of quality of evaluation reports in general (not specifically this samp-le) was discussed with several MFA staff members, one noted that the best ones are those that “raise issues that MFA has not thought of, and thus are worth outsour-cing.” With respect to this particular sample of evaluation reports, MFA staff noted that several of the reports had been very important and influential, in some cases even controversial, as they had recommended some major changes needed in the pro-ject or programme design, approach, ongoing implementation, or issues to be consi-dered in design of the next phase. Other staff noted, however, that sometimes MTRs can raise a lot of issues that are difficult to resolve, or can create tensions among the project management, government, and donor. In such cases, it might be “easier just to go ahead, or else to stop the project entirely.”

As previously noted, it must be remembered that all these ratings of the reports are based on the OECD/DAC evaluation quality standards. Some of the reports asses-sed, however, had been asked to focus on very specific analyses, which may have not covered all the OECD/DAC evaluation quality standards. For example, the appraisal of the Water Supply, Sanitation and Hygiene (WASH) Capacity Building Pool Fund (CBPF) for Ethiopia (XXIII) was well done, in response to its own specific TOR. In terms of the OECD/DAC standards, it was only rated as good, but in terms of the specific analysis done, the report was well appreciated by the respective donors.

6 QUALITY OF THE AID INTERVENTIONS

6.1 Overall Rating of the Aid Interventions

A major part of the assessment was aiming to rate the development interventions in terms of the eight key evaluation criteria used for this Meta-analysis: the five OECD/DAC evaluation criteria – relevance, effectiveness, efficiency, impact, and sustainabi-


lity, and three additional criteria – policy coherence, Finnish added value, and baseli-ne data, indicators, and monitoring. This undertaking was based primarily upon the information available in the project and programme evaluation reports. Thus, it is likely that the evaluation reports do not present a full picture of the interventions, and had the assessments been based on additional information, perhaps some of the ratings might have differed. The overall ratings are indicated in Table 5, and further discussed in the following sections.

Table 5 Ratings of aid interventions by evaluation criteria. (Ratings based on five OECD/DAC evaluation criteria and three additional evaluation criteria).

6.2 Relevance, Policy Coherence and Complementarity

Overall, relevance received the highest average ratings of the eight evaluation deve-lopment criteria assessed in this Meta-analysis. Reports varied, however, in terms of whether they assessed relevance vis-à-vis the project beneficiaries, and/or the host government’s policies, and/or the development policies of Finland. In some cases, the policy issues were discussed as a matter of policy coherence and complemen-tarity, not in terms of relevance, and thus the relevance assessment focused on the beneficiaries. This conceptual confusion makes it difficult to do a straightforward comparison of the reports.

in the project or programme design, approach, ongoing implementation, or issues to be considered in design of the next phase. Other staff noted, however, that sometimes MTRs can raise a lot of issues that are difficult to resolve, or can create tensions among the project management, government, and donor. In such cases, it might be “easier just to go ahead, or else to stop the project entirely.” As previously noted, it must be remembered that all these ratings of the reports are based on the OECD/DAC evaluation quality standards. Some of the reports assessed, however, had been asked to focus on very specific analyses, which may have not covered all the OECD/DAC evaluation quality standards. For example, the appraisal of the Water Supply, Sanitation and Hygiene (WASH) Capacity Building Pool Fund (CBPF) for Ethiopia (XXIII) was well done, in response to its own specific TOR. In terms of the OECD/DAC standards, it was only rated as good, but in terms of the specific analysis done, the report was well appreciated by the respective donors.

6 QUALITY OF THE AID INTERVENTIONS 6.1 Overall Rating of the Aid Interventions

A major part of the assessment was aiming to rate the development interventions in terms of the eight key evaluation criteria used for this Meta-analysis: the five OECD/DAC evaluation criteria – relevance, effectiveness, efficiency, impact, and sustainability, and three additional criteria – policy coherence, Finnish added value, and baseline data, indicators, and monitoring. This undertaking was based primarily upon the information available in the project and programme evaluation reports. Thus, it is likely that the evaluation reports do not present a full picture of the interventions, and had the assessments been based on additional information, perhaps some of the ratings might have differed. The overall ratings are indicated in Table 5, and further discussed in the following sections.

Table 5 Ratings of aid interventions by evaluation criteria. (Ratings based on five OECD/DAC evaluation criteria and three additional evaluation criteria).

Evaluation Criteria for Aid Interventions Evaluations and Appraisals Assessed R

ele

van

ce

Eff

icie

ncy

Eff

ect

iven

ess

Imp

act

Su

stain

ab

ilit

y

Po

licy

co

her

ence

Fin

nis

h a

dd

ed

valu

e

Dat

a, i

nd

icato

rs

and

mo

nit

ori

ng

Over

all

qu

ali

ty o

f aid

in

terv

enti

on

EVALUATIONS Average rating for Interventions (# rated)

2.8 (19)

1.7 (20)

1.9 (23)

2.1 (15)

1.9 (20)

2.3 (17)

2.0 (5)

1.4 (11)

1.9 (23)

APPRAISALS Average rating for Interventions (#rated)

2.5 (13)

2.2 (4)

2.0 (6)

2.1 (7)

2.1 (12)

2.2 (13)

2.0 (3)

2.1 (8)

2.1 (13)

TOTAL SAMPLE Average rating 2.7

(32) 1.8 (24)

1.9 (29)

2.1 (22)

1.9 (32)

2.3 (30)

2 (8)

1.7 (19)

2.0 (36)

Note: Ratings: 1 = poor ; 2 = good; and 3 = very good


Table 6 Ratings of aid interventions by relevance and policy coherence.

6.2.1 Relevance

For the evaluations and appraisals assessed, two-thirds of the evaluation and appraisal missions judged the projects and programmes being assessed as very relevant. It was most common to find that the reports discussed the interventions as being relevant with respect to the policies of the partner government and the Finnish policies. In some cases, the reports mentioned relevance with respect to the target groups and be-neficiaries. This finding is not surprising. It is assumed that the partner government and the MFA would consider the intervention to be a priority and relevant, otherwise they would be focusing their cooperation on other matters.

In one case (XIX), however, the evaluators reported that some of the partner govern-ment stakeholders’ opinions on the project relevance were undoubtedly biased, as they had benefited from the project by being employed as consultants.

Most reports considered the overall relevance of the project or programme, rather than considering various elements of the interventions. One report, on the Nepal Education for All Programme (XV), had assessed 16 programme strategies separately, and concluded that 15 were of high relevance and 1 of medium relevance.

One report on the rural development programme in Nicaragua (III) had found that the relevance of the programme had changed between its design and the MTR, as the programme had become more relevant to the beneficiaries, but in so doing had become less relevant to the implementing institutions and policy context. But most reports did not comment on any changes in relevance between design and implemen-tation.

In the appraisal of Support to the National Forestry Programme in Mozambique (XXVI), the evaluator had assessed that the intervention would be relevant to benefi-ciaries, but noted that these beneficiaries needed more information and consultation, and recommended that steps be taken to improve their involvement as the design was finalised. Most other reports did not comment on this issue of how or whether the target groups, beneficiaries, or other stakeholders were consulted as to the relevance of the intervention.

For a few of the proposed interventions appraised, such as the gender mainstreaming

6.2 Relevance, Policy Coherence and Complementarity Overall, relevance received the highest average ratings of the eight evaluation development criteria assessed in this Meta-analysis. Reports varied, however, in terms of whether they assessed relevance vis-à-vis the project beneficiaries, and/or the host government’s policies, and/or the development policies of Finland. In some cases, the policy issues were discussed as a matter of policy coherence and complementarity, not in terms of relevance, and thus the relevance assessment focused on the beneficiaries. This conceptual confusion makes it difficult to do a straightforward comparison of the reports. Table 6 Ratings of aid interventions by relevance and policy coherence.

6.2.1 Relevance

For the evaluations and appraisals assessed, two-thirds of the evaluation and appraisal missions judged the projects and programmes being assessed as very relevant. It was most common to find that the reports discussed the interventions as being relevant with respect to the policies of the partner government and the Finnish policies. In some cases, the reports mentioned relevance with respect to the target groups and beneficiaries. This finding is not surprising. It is assumed that the partner government and the MFA would consider the intervention to be a priority and relevant, otherwise they would be focusing their cooperation on other matters. In one case (XIX), however, the evaluators reported that some of the partner government stakeholders’ opinions on the project relevance were undoubtedly biased, as they had benefited from the project by being employed as consultants. Most reports considered the overall relevance of the project or programme, rather than considering various elements of the interventions. One report, on the Nepal Education for All Programme (XV), had assessed 16 programme strategies separately, and concluded that 15 were of high relevance and 1 of medium relevance. One report on the rural development programme in Nicaragua (III) had found that the relevance of the programme had changed between its design and the MTR, as the programme had become more relevant to the beneficiaries, but in so doing had become less relevant to the implementing institutions and policy context. But most reports did not comment on any changes in relevance between design and implementation. In the appraisal of Support to the National Forestry Programme in Mozambique (XXVI), the evaluator had assessed that the intervention would be relevant to beneficiaries, but noted that these beneficiaries needed more information and consultation, and recommended that steps be taken to improve their involvement as the design was finalised. Most other reports did not comment on this issue of how or whether the target groups, beneficiaries, or other stakeholders were consulted as to the relevance of the intervention. For a few of the proposed interventions appraised, such as the gender mainstreaming programme for Nicaragua (XXI) or the capacity building pool fund for the water, sanitation and hygiene sector in Ethiopia (XXIII), the appraisal missions had judged the proposals highly relevant in terms of the

Report Quality (Number of Reports Rated) Criteria

Number of Interventions Rated on Criteria

Poor (1.0 to 1.5)

Good (2.0)


Very Good (3.0)

Average Rating

Relevance 32 10 1 21 2.7 Policy Coherence

30 1 20 9 2.3


programme for Nicaragua (XXI) or the capacity building pool fund for the water, sanitation and hygiene sector in Ethiopia (XXIII), the appraisal missions had judged the proposals highly relevant in terms of the policy priorities of both the partner go-vernment and the donor(s), but then later the partner government decided that it did not want to proceed with the proposed cooperation.

6.2.2 Policy Coherence and Complementarity

In terms of policy coherence, the Meta-analysis Team looked at a number of factors, including coherence with Finland’s development policy, other sectoral interventions, activities of other donors, the Paris Declaration on Aid Effectiveness, the MDGs, poverty reduction, and mainstreaming of cross-cutting issues.

Overall, for the criterion of policy coherence received the second highest rating. It was most common to find an assessment of coherence with Finnish development policies and with the host government’s development policies. The issue of MDGs is one that is addressed on a national level, although assuredly a programme or pro-ject can make contributions towards achieving these MDG targets, or contributions towards poverty reduction. But this issue was rarely considered in the evaluation reports.

Consideration of the Paris Declaration, or efforts of a programme or project to har-monize and align with government procedures and work together with other donors, was also not much stressed in the reports of these projects or programmes. Refe-rence was made to harmonisation in the PRORURAL Project (III) and proposed PRO-GENERO Project (XXI), both in Nicaragua -- where it is a significant issue for the government.

The issue of mainstreaming of cross-cutting issues was rarely mentioned, although for a few projects or programmes it was a priority, such as gender issues (IX, XXI) or providing inclusive education for persons with disabilities (VI-A, XVII-B). HIV/AIDS was only mentioned in some reports for Mozambique (XI, XVII), where the issue is highly salient.

Often, however, apart from the issue of policy coherence with Finnish development policies and with the host government’s policies, these other elements of policy cohe-rence were not mentioned in the evaluation report’s TOR, and thus it is little surprise that they were often not mentioned in the reports.

It was difficult for the Meta-analysis Team to judge the complementarity of the inter-ventions to those of other donors and the host country government, or even other Finnish-assisted interventions within the country, as this topic was rarely mentioned in the reports. In one final sector evaluation, of two education programmes in South Africa (VI), the TOR asked for – and the report supplied – an overview of other interventions in the sector, for consideration for possible future programming. In


this particular case, future support may be provided through other modalities, such as institutional collaboration between educational institutions.

6.3 Effectiveness and Efficiency

The issues of effectiveness and efficiency concern the performance and achievements of the project or programme. Effectiveness is an issue of whether or not the inter-vention is achieving its objectives, whereas efficiency concerns how such results are achieved, in terms of use of time, personnel, and other resources.

Table 7 Ratings of aid interventions by effectiveness and efficiency.

6.3.1 Effectiveness

The issue of effectiveness was examined, in varying degrees, in 29 of the 36 projects. The terminology used, however, varied – in some cases, referred to as performance or achievements, or in the case of appraisals, as feasibility. The average rating was 1.9. In a few cases, the evaluators stated that a project had been highly effective or successful.

One relatively small project that was considered to be quite effective was the Envi-ronmental GIS project in Montenegro (XVIII), which had developed many concrete results, in GIS information development, capacity building and sector applications, and had been quite responsive to requests from various partner organisations for GIS information. Another was the educational programme in Bosnia and Herzegovina (XVII-B), which had supported education policy, administration and management, as well as inclusive education. This project had strong local ownership and leadership. Trainees came from all over the country, and both training course and M.Sc. partici-pants were able to use their new skills and information in immediate applications in their day-to-day work.

The FINAM project to rebuild and develop the meteorological services in Mozambi-que (XX) was very widely recognized to be effective. Prior to these efforts, disastrous flooding had occurred in 2000, with almost 700 casualties and major damage. Fol-lowing improvements in the meteorological services, when major flooding occurred in 2007, at least 55 000 people were affected, but there were no casualties due to effective early warnings. The FINAM project had contributed substantially to these efforts, but there had also been parallel support provided by several other develop-

policy priorities of both the partner government and the donor(s), but then later the partner government decided that it did not want to proceed with the proposed cooperation. 6.2.2 Policy Coherence and Complementarity In terms of policy coherence, the Meta-analysis Team looked at a number of factors, including coherence with Finland’s development policy, other sectoral interventions, activities of other donors, the Paris Declaration on Aid Effectiveness, the MDGs, poverty reduction, and mainstreaming of cross-cutting issues. Overall, for the criterion of policy coherence received the second highest rating. It was most common to find an assessment of coherence with Finnish development policies and with the host government’s development policies. The issue of MDGs is one that is addressed on a national level, although assuredly a programme or project can make contributions towards achieving these MDG targets, or contributions towards poverty reduction. But this issue was rarely considered in the evaluation reports. Consideration of the Paris Declaration, or efforts of a programme or project to harmonize and align with government procedures and work together with other donors, was also not much stressed in the reports of these projects or programmes. Reference was made to harmonisation in the PRORURAL Project (III) and proposed PRO-G�NERO Project (XXI), both in Nicaragua -- where it is a significant issue for the government. The issue of mainstreaming of cross-cutting issues was rarely mentioned, although for a few projects or programmes it was a priority, such as gender issues (IX, XXI) or providing inclusive education for persons with disabilities (VI-A, XVII-B). HIV/AIDS was only mentioned in some reports for Mozambique (XI, XVII), where the issue is highly salient. Often, however, apart from the issue of policy coherence with Finnish development policies and with the host government’s policies, these other elements of policy coherence were not mentioned in the evaluation report’s TOR, and thus it is little surprise that they were often not mentioned in the reports. It was difficult for the Meta-analysis Team to judge the complementarity of the interventions to those

of other donors and the host country government, or even other Finnish-assisted interventions within

the country, as this topic was rarely mentioned in the reports. In one final sector evaluation, of two

education programmes in South Africa (VI), the TOR asked for – and the report supplied – an

overview of other interventions in the sector, for consideration for possible future programming. In

this particular case, future support may be provided through other modalities, such as institutional

collaboration between educational institutions. 6.3 Effectiveness and Efficiency The issues of effectiveness and efficiency concern the performance and achievements of the project or programme. Effectiveness is an issue of whether or not the intervention is achieving its objectives, whereas efficiency concerns how such results are achieved, in terms of use of time, personnel, and other resources. Table 7 Ratings of aid interventions by effectiveness and efficiency.



Poor (1.0 to 1.5)

Good (2.0)


Very Good (3.0)

Average Rating

Effectiveness 29 6 19 4 1.9 Efficiency 24 10 10 4 1.8


ment partners, such as Spain, the EU, the Food and Agriculture Organization of the United Nations (FAO), and the United States Agency for International Development (USAID).

But it was often quite difficult to assess overall effectiveness, as the interventions are often “a mixed bag.” As one report (XV) noted, the programme in question had some notable achievements, but also many areas needing improvement. In the case of some major sectoral programmes or other interventions, such as the Kenya Gen-der and Governance Programme (IX), the activities had been effective in certain sites, i.e., provinces that had good capacity, and not in other sites with weaker capacity, or with with certain implementing partners but less so with other partners. Sometimes an intervention was successful in certain components but not in others – perhaps due to overly ambitious design, such that there were too much to implement at the same time. This situation of mixed effectiveness was typical for most of the interventions. For example, the MTR determined that the Forest Education and Policy Research (FOPER) project for the Western Balkans (XVI) was quite effective in its pre-service and in-service (professional) training, particularly the M.Sc. programme, but it had not really begun activities related to its policy research component.

Common problems with effectiveness (and also efficiency) were delays in project implementation for various reasons. In some cases, such as with the educational pro-gramme in Palestine, these delays could be easily understood, due to political prob-lems that had affected everyone’s daily life. The evaluators had, in fact, commended the TA team for keeping the project running, even when other donors had pulled out.

Two of the evaluations were highly critical of the implementation approaches used by their respective projects, and proposed that it was important to look at the overall conceptual issues being addressed by a programme. One was a rural development programme in Mozambique (XI), which was criticized as being ineffective for its scat-tered interventions, and not focusing on building a “value chain” from the producers to the markets.

The second examined the Kenya Gender and Governance Programme, Phase II (IX), which had focused (during an election period) primarily on trying to help more wo-men get elected to political office. More women had stood for office, but there had not been any noticeable increase in the number of women elected. The evaluation team questioned why the programme had not put more focus on its component for supporting policy development, and questioned whether simply electing more women to political office was enough. They suggested that instead the focus should have been on building an effective women’s movement, to better represent women’s issues in the political process.

One small ex post evaluation (VIII), undertaken in 2007, examined 22 of 24 biogas digesters that had been built between 2001 and 2004 in Ethiopia, with support from


the Embassy’s Local Cooperation Funds (LCF). This study had found that many of the units had not been completed, or else had not been properly maintained and thus were no longer in use. There had, however, been some successes, with a unit in a prison that had been very successful, and thus had been replicated by the prison administration (without Finnish assistance) in other prisons.

6.3.2 Efficiency

The efficiency issue was considered much less frequently, only being examined for two-thirds (24 out of 36) of the interventions. This criterion received a rating of 1.8, which was the next to lowest average rating.

In a few notable cases, the intervention was praised as being highly efficient. For example, the Environmental GIS project in Montenegro had achieved cost savings, through acquisition of software at reduced rates (or for free), and thus was able to operate on its budget for two years, instead of the originally planned 18 months. The UNIDO project for women entrepreneurs in Viet Nam (XIV) was lauded for its use of intermittent international TA, instead of having a permanent long-term advisor in place, which resulted in greater cost-effectiveness. This project also adapted training materials and approaches from a similar project elsewhere in Viet Nam, which also contributed to cost-effectiveness.

Conversely, one project working on improving policies for persons with disabilities in Bosnia and Herzegovina (XIX) was severely criticized for its highly inefficient use of government civil servants as consultants. In another programme, Nepal Educati-on for All (XV), it was noted that many of the block grants provided to schools for capital improvements were being used instead to pay for teacher salaries. In many projects, the most common problem with efficiency was delays in implementation, thereby leading to lower disbursements of the budget.

6.4 Sustainability and Impact

Sustainability refers to the likelihood that the benefits of a project or programme will continue after donor funding is phased out. Impacts concern the long-term effects of the intervention. Both of these criteria refer to what is likely to happen after the intervention ends. Some of the evaluation reports commented on the intrinsic dif-ficulties of assessing the probabilities of future likelihood of sustainability or future impacts.


Table 8 Ratings of aid interventions by sustainability and impact.

6.4.1 Sustainability

The issue of sustainability of the activities or benefits of the development interventi-on was assessed for 32 out of 36 interventions, with an average rating of 1.9. Sustai-nability was examined more often in appraisal reports and final, or ex post evaluation reports, than in MTRs.

Sustainability for this group of interventions is most often viewed in terms of eit-her institutional sustainability (15 interventions) or financial terms (mentioned for 11 interventions). Sustainability was also often assessed in terms of whether or not there was any local ownership for the activity. One report drew a distinction between sustainability of results for the beneficiaries versus institutional and financial sustai-nability of the government institution supporting the sector.

Only five projects or programmes were reviewed using the three dimensions mentio-ned in the 2007 policy – ecological, social, and economic sustainability. Two other evaluation reports had considered the eight dimensions of sustainability in the 1999 MFA Guidelines for Programme Design, Monitoring and Evaluation (Ministry for Foreign Affairs of Finland 1999) , which include social, economic, and environmental factors, whereas other reports considered a subset of the eight dimensions.

For example, a Finnish consultant on the appraisal of a watershed management project in Ethiopia, co-financed by the WB, Finland, and Government of Ethiopia (XXIV), noted that the project’s prospects for sustainability were “better than average,” as it was planning to use government systems and promote watershed management prac-tices that had been used throughout Ethiopia over the past 20 years.

The proposed hydro-meteorological project in Peru (XXII) was rated as having very good sustainability potential. This project would build upon the already existing natio-nal meteorological systems, upgrading their technical and human resource capabilities. The report very thoroughly discusses economic, social and ecological issues and antici-pated benefits of the proposed intervention. In terms of the project’s direct objectives, i.e., improved meteorological services, the major sustainability issues are institutional and financial. The appraisal team also recommended that the project consider developing a wider range of meteorological services and more commercially-oriented services, as well as an information and advocacy campaign to broaden the impacts of the project.

put more focus on its component for supporting policy development, and questioned whether simply electing more women to political office was enough. They suggested that instead the focus should have been on building an effective women’s movement, to better represent women’s issues in the political process.

One small ex post evaluation (VIII), undertaken in 2007, examined 22 of 24 biogas digesters that had been built between 2001 and 2004 in Ethiopia, with support from the Embassy’s Local Cooperation Funds (LCF). This study had found that many of the units had not been completed, or else had not been properly maintained and thus were no longer in use. There had, however, been some successes, with a unit in a prison that had been very successful, and thus had been replicated by the prison administration (without Finnish assistance) in other prisons. 6.3.2 Efficiency The efficiency issue was considered much less frequently, only being examined for two-thirds (24 out of 36) of the interventions. This criterion received a rating of 1.8, which was the next to lowest average rating. In a few notable cases, the intervention was praised as being highly efficient. For example, the Environmental GIS project in Montenegro had achieved cost savings, through acquisition of software at reduced rates (or for free), and thus was able to operate on its budget for two years, instead of the originally planned 18 months. The UNIDO project for women entrepreneurs in Viet Nam (XIV) was lauded for its use of intermittent international TA, instead of having a permanent long-term advisor in place, which resulted in greater cost-effectiveness. This project also adapted training materials and approaches from a similar project elsewhere in Viet Nam, which also contributed to cost-effectiveness. Conversely, one project working on improving policies for persons with disabilities in Bosnia and Herzegovina (XIX) was severely criticized for its highly inefficient use of government civil servants as consultants. In another programme, Nepal Education for All (XV), it was noted that many of the block grants provided to schools for capital improvements were being used instead to pay for teacher salaries. In many projects, the most common problem with efficiency was delays in implementation, thereby leading to lower disbursements of the budget. 6.4 Sustainability and Impact Sustainability refers to the likelihood that the benefits of a project or programme will continue after donor funding is phased out. Impacts concern the long-term effects of the intervention. Both of these criteria refer to what is likely to happen after the intervention ends. Some of the evaluation reports commented on the intrinsic difficulties of assessing the probabilities of future likelihood of sustainability or future impacts. Table 8 Ratings of aid interventions by sustainability and impact.

6.4.1 Sustainability



Poor (1.0 to 1.5)

Good (2.0)


Very Good (3.0)

Average Rating

Sustainability 32 5 24 3 1.9 Impact 22 1 18 3 2.1


The MTR team for the Kenya Gender and Governance Programme (IX) rated the prospects for sustainability as poor, due to financial and institutional challenges, poli-tical competition, ethnic divisions, and a lack of a unified broader women’s moment and networks. The programme had aimed not only to get more women elected to po-litical office, but also to change the government’s policies to better address women’s issues. These various factors were seen as major challenges to be addressed. For this programme, it was assumed that changes in attitudes and changes in the national Con-stitution are long-term issues, and efforts to bring about these changes would require long-term support.

Major Factors contributing to Good SustainabilityIt was commonly agreed that major factors contributing to good sustainability are local ownership, political will, government commitment, participation and empower-ment, capacity development, building appropriate policy frameworks, and continuing financing. A particularly notable case of government commitment had been the efforts in the Palestine Territories to keep the educational system running, despite major ongoing political problems that had severely impacted daily life (XII). Like-wise, ownership by the local indigenous peoples had been highlighted as a factor for sustainability in the Andean bilingual education programme (V). It was often noted that use of local systems, the institutionalisation of systems introduced through the development intervention or replication of successful pilots were key elements of sustainability.

Human resource capacity – either existing or developed through donor interventions – was another key factor, as well as a potential risk in some cases of “brain drain,” or the eventual loss of trained personnel. This issue was especially highlighted with res-pect to personnel training information and communications technology (ICT). On the other hand, in a project supporting the national meteorology services in Mozam-bique (XX), the potential risk of “brain drain” was judged unlikely – because there are no other potential employers. In one project in Bosnia and Herzegovina (XVII-B), it was noted that the training in inclusive education had been very successful, as it could be immediately applied in the participants’ work situations, and as participants had come from all over the country, had a large impact.

In some cases, interventions have been judged successful in raising awareness, and/or developing a “moral consensus” regarding the need for change, such as in the Ke-nya Gender and Governance Programme (IX), or the case of creating a demand for “early childhood education,” which had occurred with the Nepal Education for All Programme (XV).

The issue of appropriate technology was mentioned in a few cases. For example, in the Mozambique meteorology project (XX), the evaluator thought that the project had chosen appropriate technology that was adequate to meet the required standards, rather than considerably more expensive technology as recommended by another donor team, which would have been very difficult to sustain due to high operating costs.


Another key element is adequate planning for sustainability. In the UNIDO project promoting small-scale women entrepreneurs in Viet Nam (XIV), as the project closed, “way forward” stakeholder workshops were planned for each province, where partici-pating stakeholders would prepare two-year Sustainability Action Plans for follow-up after the project ends. The final evaluation team commended this action, although they noted that it would have been preferable had such plans been developed earlier in the project implementation, rather than at the very end. The idea of developing Sus-tainability Action Plans was also proposed by a MTR team, for the Forest Policy and Economics Education and Research (FOPER) project in the Western Balkans (XVI). For one multi-donor fund that was financing small-scale initiatives (II), the evaluation had noted that some grantees had developed exit strategies from the very beginning.

Obstacles to SustainabilityThe evaluation reports identified a wide range of issues seen as obstacles to sus-tainability. In some cases there had been inadequate leadership and support from government, in others limited staff capacities, staff overloaded with work, or facing bureaucratic bottlenecks or inter-departmental competition. In one case, where colla-boration had otherwise been very good, the lack of involvement of a key stakeholder agency was cited as an obstacle. The issue of limited capacity and human resource development was a factor in a few cases. In one rural development programme in Mozambique (XI), over-reliance on outsourcing to consultants was criticized for not adequately building government capacity, thereby endangering future sustainability.

Political issues, such as political changes, political conflict, rivalries, ethnic divisions and patronage politics were mentioned in several reports. Such political problems were cited in Kenya, Bolivia, and Palestine. In a couple of projects, it was noted that a perceived lack of neutrality by the programme management, regarding grants or scholarships, threatened to undermine confidence in and sustainability of the pro-gramme.

In the Nepal Education for All Programme (XV), it was noted that despite many achievements, there was still a great need for improvement, especially “getting those responsible for delivery of education to do things as they are meant to be doing as set out in the strategies” (Copenhagen Development Consulting & Edburgh Consultants 2007, p. 81).

A common obstacle to sustainability for many interventions is inadequate funding, or insecure financial support. In one case (XX), the national policy for decentralisa-tion was cited as an obstacle, as the region did not have the finances to support an installation for the national weather service. In terms of developing disability policies in Bosnia and Herzegovina (XIX), concern was raised over the financial ability of localities to implement the policies. In the case of Palestine (XII), the educational sys-tem had been producing an integrated textbook/learner workbook, which was very expensive, as they had to be reprinted each year, if not each semester, and could not be sustained without donor support.


Another factor mentioned in some projects or programmes was inadequate local con-sultation and ownership. In the appraisal of a regional INGO project in the Western Balkans (XXIX), the evaluators had noted that the local people were already frustra-ted with repetitive planning processes and capacity building, so the project would have to demonstrate that it would translate such efforts into implementation.

Risks and Assumptions to SustainabilityMany of the future risks to sustainability were similar to current obstacles – such as uncertain future financial support, staff turnover, or risks in the political environ-ment. In some cases, there was concern about how to continue existing programme activities – due to uncertain commitment of local officials, or changes in government and thus changes in responsible parties, or uncertainty whether a particular govern-ment unit would continue. Other risks were mentioned, such as other stakeholders possibly feeling threatened. For example, in a project promoting disability policies (XIX), there was concern that disabled war veterans might feel that the project could possibly be a threat to their own benefits. An environmental project appraisal (XXIX) raised questions as to whether the programme approaches would be fully adopted and replicated. The evaluator of the Lake Victoria Basin Commission Partnership (XXVIII) voiced concerns that the activities might not be enough – or fast enough – to prevent environmental degradation, and also that the work could be jeopardized by any political disagreements, should they arise, among the member states. It was noted, however, that regional efforts were being undertaken to work in the opposite direction, towards regional integration.

In one appraisal of a multi-donor fund (XXIII), a risk analysis identified six key risks. In another report (XXI), it was noted that the risk analysis that had already been done needed to be updated. These two examples, however, were the only ones that speci-fically discussed risk analysis.

The Meta-analysis Team was also tasked to look at the issue of assumptions regarding sustainability. On this topic, most of the evaluation reports were silent, as they did not consider this issue. For the Kenya Gender and Governance Programme (IX), the evaluation team noted that it was assumed that political change, changing attitudes and even changes to the national Constitution were needed. They felt that empowe-ring women was a long-term challenge and process, and thus would require long-term support.

The evaluator examining the meteorological services in Mozambique (XX) noted that the project implementation team had assumed that promotion of market-based ap-proaches and selling meteorological information products, would be a way for the institute to raise enough funding to become a semi-autonomous agency. The evalu-ator judged this assumption unrealistic, given the state of the economy and level of poverty in Mozambique.


6.4.2 Impacts

Impact was examined less often than sustainability, for only 22 out of 36 interven-tions, with an average rating of 2.1. Some evaluators argued that it was “too early” during the MTR – or even in the final evaluation -- to assess long-term impacts, which would only become apparent after some years. Indeed, often impact studies are conducted some time after a project or programme finishes. The MTR team for the PRORURAL project in Nicaragua (III) documented 18 pages of project achie-vements, but said that it was too early to assess impacts. For a project in Viet Nam (XIV), however, a separate impact assessment had been planned to be conducted some months after the final project evaluation.

As previously discussed, the meteorological project in Mozambique (XX) was judged to have achieved very good impact , as it had helped with natural disaster manage-ment. Likewise, the meteorological project proposed for Peru (XXII) was seen to have very good potential impact.

The environmental GIS project in Montenegro (XVIII) was seen to have had very good impacts, not only in terms of producing GIS information and building capaci-ty, but also the availability of reliable environmental information for use in land use planning and allocation, to promote more sustainable development.

The sample had included an ongoing regional project in the Western Balkans, FO-PER (XVI), as well as appraisals of two proposed INGO regional environmental projects (XXIX, XXX). The regional projects in the Western Balkans seem to have very positive impacts, promoting professional collaboration and networking among people of different ethnicities and nationalities, thus helping to heal the social and political stresses from years of conflict in the region. Even for other projects opera-ting on a national level within this region, such as the education project in Bosnia and Herzegovina (XVII-B), similar impacts had been noted.

6.5 Finnish Added Value

The criteria of “Finnish added value” is explained in the 2007 MFA Evaluation Guideli-nes (Ministry for Foreign Affairs of Finland 2007b, p. 82) as “What is the added value provided by the Finnish support?” Many people with whom the Meta-analysis Team spoke had differing ideas as to what, exactly, constitutes Finnish added value.

This concept is difficult to assess, in terms of either identifying fields (or sectors) where Finland has comparative advantage, in terms of expertise, or even specifical-ly identifying whether a development intervention has Finnish nationals involved as long-term experts or short-term consultants, or Finnish institutions are partnered with institutions in the recipient country. Such information was not easily found in the reports being assessed. This section of this report presents, however, what infor-


mation was found in the evaluation reports.

If one takes a different approach, such as the EU does, one could consider what addi-tional (or incremental) value is added to the intervention through having donor sup-port. Thus, if one similarly tries to consider what value Finland adds by supporting an intervention, then one needs to consider the overall development context within the country and how the particular intervention complements other efforts. Unfortu-nately, however, the evaluation reports did not, in most cases, provide a broad enough overview of the development context in order to make such an assessment.

Table 9 Ratings of aid interventions by Finnish added value.

The issue of “Finnish added value” was only explicitly considered for eight out of 36, or one-quarter, of the interventions. Where it was assessed, the average rating was 2.0.

The two projects for which Finnish added value is rated very good are both meteo-rology projects. A completed project in Mozambique (XX) was found to have made good use of Finnish expertise and institutional linkages. The Finnish Chief Technical Advisor was cited as having been instrumental in working with the government not only to manage the project well, but also to coordinate donor activities in the sector and parallel financing. The other meteorology project (XXII) is a proposed conces-sional credit project for Peru. In the appraisal report, the team determined that if the project were to be supported with Finnish concessional credit, then approximately 76% of project investments for equipment and components would be spent on Fin-nish equipment (some of the needed equipment can only be sourced from Norway, and other equipment only from the United States).

The final evaluation of the education sector in South Africa (VI), comprised of two different programmes, was asked to specifically focus on the Finnish support to the-se programmes. Collaboration had been undertaken with educational institutes in Finland. In one case, a major difficulty noted was that there was large turn-over of Finnish personnel, three different teams had been fielded to South Africa, and each one had to develop relationships and an understanding of the situation. It was also noted that the exchange visits, both for Finns to South Africa, and South Africans to Finland, had been too short. Nonetheless, such exchanges had brought about a critical exchange of ideas and information for reflection. Another finding was that the South African institutions of higher education were very interested to collaborate with and learn from the Finnish polytechnics, but these institutions are not Finnish

reliable environmental information for use in land use planning and allocation, to promote more sustainable development.

The sample had included an ongoing regional project in the Western Balkans, FOPER (XVI), as well as appraisals of two proposed INGO regional environmental projects (XXIX, XXX). The regional projects in the Western Balkans seem to have very positive impacts, promoting professional collaboration and networking among people of different ethnicities and nationalities, thus helping to heal the social and political stresses from years of conflict in the region. Even for other projects operating on a national level within this region, such as the education project in Bosnia and Herzegovina (XVII-B), similar impacts had been noted. 6.5 Finnish Added Value

The criteria of “Finnish added value” is explained in the 2007 MFA Evaluation Guidelines (Ministry for Foreign Affairs of Finland 2007b, p. 82) as “What is the added value provided by the Finnish support?” Many people with whom the Meta-analysis Team spoke had differing ideas as to what, exactly, constitutes Finnish added value. This concept is difficult to assess, in terms of either identifying fields (or sectors) where Finland has comparative advantage, in terms of expertise, or even specifically identifying whether a development intervention has Finnish nationals involved as long-term experts or short-term consultants, or Finnish institutions are partnered with institutions in the recipient country. Such information was not easily found in the reports being assessed. This section of this report presents, however, what information was found in the evaluation reports.

If one takes a different approach, such as the EU does, one could consider what additional (or incremental) value is added to the intervention through having donor support. Thus, if one similarly tries to consider what value Finland adds by supporting an intervention, then one needs to consider the overall development context within the country and how the particular intervention complements other efforts. Unfortunately, however, the evaluation reports did not, in most cases, provide a broad enough overview of the development context in order to make such an assessment.

Table 9 Ratings of aid interventions by Finnish added value.

The issue of “Finnish added value” was only explicitly considered for eight out of 36, or one-quarter, of the interventions. Where it was assessed, the average rating was 2.0. The two projects for which Finnish added value is rated very good are both meteorology projects. A completed project in Mozambique (XX) was found to have made good use of Finnish expertise and institutional linkages. The Finnish Chief Technical Advisor was cited as having been instrumental in working with the government not only to manage the project well, but also to coordinate donor activities in the sector and parallel financing. The other meteorology project (XXII) is a proposed concessional credit project for Peru. In the appraisal report, the team determined that if the project were to be supported with Finnish concessional credit, then approximately 76% of project investments for equipment and components would be spent on Finnish equipment (some of the needed equipment can only be sourced from Norway, and other equipment only from the United States).



Poor (1.0 to 1.5)

Good (2.0)


Very Good (3.0)

Average Rating

Finnish added value

8 2 4 2 2.0


government institutions and thus not eligible to receive MFA support through the Institutional Cooperation Instrument (ICI). For the information and communica-tions technology (ICT) component of the higher education programme, however, the evaluation team found that support was primarily budgetary, and did not make use of potential Finnish technical expertise.

In the evaluation of the multi-donor Kenya Gender and Governance Programme (IX), the local stakeholders judged that the group of donors had added important value to the project, in addition to providing financial support. Their added value was appreciated most in terms of “technical support/expertise,” but also in terms of “gi-ving visibility to gender equity, monitoring of activities, harmonization/consolidation of efforts, high-level policy advocacy, [and] transparency and accountability” (Everatt D, Chesoni A, Nyamweya P & Kanyinga K 2008, p. 32).

The appraisal for the Ethiopian watershed management project (XXIV), to be funded by the WB and Finland, proposed that if the Finnish financing were split, with part of the funds being channelled bilaterally, instead of all going through the WB trust fund, then it might be easier to ensure synergy with another ongoing Finnish-assisted Rural Water Supply and Environmental Programme (RWSEP).

The appraisal of the Lake Victoria Basin Commission Partnership (LVBC) (XXVIII) suggested that Finnish experience in harmonizing regulations and establishing joint facilities, as with the EU and work on the Baltic Sea, might be of value to work with the LVBC. Moreover, through participation in this partnership, it was also proposed that other opportunities for using Finnish expertise might develop in the region. The author did not, however, make any reference to the already considerable Finnish ex-perience in working in the region, where Kenya and Tanzania have been long-term partner countries.

Although not specifically mentioned as “added value,” the Nepal Education for All MTR (XV) noted that, in addition to the multi-donor funding of the government’s education programme, Finland and other donors had also provided parallel TA sup-port. This report recommended that in the future, consideration be given to establish pooled TA support, to be jointly supported by the various donors.

In the support to disability policy development in Bosnia and Herzegovina (XIX), the evaluation team concluded that greater use could have been made of Finnish exper-tise on these issues.

In reading through all the reports, it is argued that Finland has considerable expertise in certain fields, – such as education (notably teacher training, inclusive education, and bilingual education), innovation systems, meteorology, and forestry. Others areas of expertise pertain to issues that Finland has long championed, such as gender and sexual and reproductive health, support for persons with disabilities, and rural development, including related land administration and small enterprise development efforts.


6.6 Baseline Data, Indicators and Monitoring

Efforts were made in conducting this analysis to assess whether or not the projects had baseline data, whether there were indicators to measure sustainability, what type of plan exists for recording data on sustainability, and whether or not the intervention has collected data disaggregated by gender. In reading through the reports, however, it became apparent that the reports had limited information or commentary on these issues, and that they were rarely specified in the TOR. Some reports, for example, made reference to the monitoring system, or to indicators – perhaps suggesting imp-rovements were needed, but not providing any detailed suggestions. The lack of ade-quate data makes it difficult to assess performance and achievements, and especially impacts.

Table 10 Ratings of aid interventions by baseline data, indicators and monitoring.

The rating for the criterion pertaining to baseline data, indicators, and monitoring had an average rating of 1.7, which was the lowest rating. This issue was only assessed for just over half, or 19 out of the 36, of the projects and programmes.

The issue of having baseline data was mentioned in only a few reports. For example, for the bilingual intercultural education programme (EIBAMAZ) in the Andean regi-on (V), baseline data had been collected in each of the three countries. In assessing the inclusive education programme in South Africa (VI-A), evaluators noted a lack of a systematic baseline and monitoring on learners (and potential learners) with disabilities.

The MTR of the Nepal Education for All Programme (XV) evaluators found that the national Educational Management Information System (EMIS) provided a good data base, “adequate… for annual monitoring and management of the education sector and can be utilized for medium and long-term educational policy decisions.” On the other hand, the data and reporting for the Education for All Programme was regar-ded by the MTR team as unsatisfactory, with “reports usually not in time, incomplete, inconsistent, unreliable and not transparent.” (Copenhagen Development Consulting & Edburgh Consultants 2007, p. 81) For the MTR of the second phase of the Kenya Gender and Governance Programme (IX), the evaluators were highly critical of the lack of baseline data. They noted that indicators were of variable quality, and too many. They advocated fewer indicators at the programme level, and more training and support to partners on their indicators

Efforts were made in conducting this analysis to assess whether or not the projects had baseline data, whether there were indicators to measure sustainability, what type of plan exists for recording data on sustainability, and whether or not the intervention has collected data disaggregated by gender. In reading through the reports, however, it became apparent that the reports had limited information or commentary on these issues, and that they were rarely specified in the TOR. Some reports, for example, made reference to the monitoring system, or to indicators – perhaps suggesting improvements were needed, but not providing any detailed suggestions. The lack of adequate data makes it difficult to assess performance and achievements, and especially impacts.

Table 10 Ratings of aid interventions by baseline data, indicators and monitoring.

The rating for the criterion pertaining to baseline data, indicators, and monitoring had an average rating of 1.7, which was the lowest rating. This issue was only assessed for just over half, or 19 out of the 36, of the projects and programmes. The issue of having baseline data was mentioned in only a few reports. For example, for the bilingual intercultural education programme (EIBAMAZ) in the Andean region (V), baseline data had been collected in each of the three countries. In assessing the inclusive education programme in South Africa (VI-A), evaluators noted a lack of a systematic baseline and monitoring on learners (and potential learners) with disabilities. The MTR of the Nepal Education for All Programme (XV) evaluators found that the national Educational Management Information System (EMIS) provided a good data base, “adequate… for annual monitoring and management of the education sector and can be utilized for medium and long-term educational policy decisions.” On the other hand, the data and reporting for the Education for All Programme was regarded by the MTR team as unsatisfactory, with “reports usually not in time, incomplete, inconsistent, unreliable and not transparent.” (Copenhagen Development Consulting & Edburgh Consultants 2007, p. 81) For the MTR of the second phase of the Kenya Gender and Governance Programme (IX), the evaluators were highly critical of the lack of baseline data. They noted that indicators were of variable quality, and too many. They advocated fewer indicators at the programme level, and more training and support to partners on their indicators and reporting. They noted that the programme had over 30 partner organisations (grant recipients). They found great differences in how (and how often) the partners reporting their activities, their understanding and use of indicators, etc. They judged that the overall programme had a weak logical framework, and a weak monitoring and evaluation system. Due to all these problems with the data, the evaluation team itself spent a considerable amount of time compiling primary data to assess project performance. They noted that it was quite difficult to assess impacts of the programme, due to data inadequacies as well as attribution issues – as other initiatives were ongoing, so changes were undoubtedly the result of many combined efforts, not just from this programme alone. For the National Institute for Educational Development (INDE) programme in Mozambique (X), evaluators noted that the project had good indicators, but they found no evidence that these were being used in the project reports to assess, or report upon, project performance and achievement.



Poor (1.0 to 1.5)

Good (2.0)


Very Good (3.0)

Average Rating

Baseline Data, Indicators and Monitoring

19 7 10 2 1.7


and reporting. They noted that the programme had over 30 partner organisations (grant recipients). They found great differences in how (and how often) the partners reporting their activities, their understanding and use of indicators, etc. They judged that the overall programme had a weak logical framework, and a weak monitoring and evaluation system. Due to all these problems with the data, the evaluation team itself spent a considerable amount of time compiling primary data to assess project performance. They noted that it was quite difficult to assess impacts of the program-me, due to data inadequacies as well as attribution issues – as other initiatives were ongoing, so changes were undoubtedly the result of many combined efforts, not just from this programme alone.

For the National Institute for Educational Development (INDE) programme in Mo-zambique (X), evaluators noted that the project had good indicators, but they found no evidence that these were being used in the project reports to assess, or report upon, project performance and achievement.

The evaluation of the PRORURAL programme in Nicaragua (III) found that it also relied upon national statistics, in this case for agriculture, and that individual projects under the programme had their own regular reporting of data. In between, at the programme level, however, the indicators were considered by the evaluation team to be poorly formulated and inadequately monitored.

In the project supporting disability policy development in Bosnia and Herzegovi-na (XIX), the MTR team found that the project was undertaking surveys, which were building up a valuable dataset of information on persons with disabilities. With respect to the project, however, they argued that many of the indicators were too long-term and not appropriate for the project. As a result, they felt that the project’s achievements were actually being under-reported.

The UNIDO project in Viet Nam (XIV) was cited as having comprehensive baseline data, and could document its achievements. The indicators were primarily quan-titative, although the evaluators argued that some qualitative indicators could have been useful. They noted that the project had provided regular monitoring and self-evaluation and progress reports – although in this case, the evaluators judged that the project management had done perhaps too much reporting.

In an appraisal of the bilateral Rural Water Supply, Sanitation and Hygiene Program-me in Benishangul-Gumz Region, Ethiopia (WASH-Beni) (XXVI), the evaluators had urged a reconsideration of the project design, so that the project could serve as a mo-del for others in the national WASH network elsewhere in Ethiopia. In this vein, they also advocated that the programme lead the way in developing model information technology systems, surveys and research, and monitoring and evaluation systems.


6.7 Overall Quality of the Aid Interventions

For the 36 aid interventions, their overall quality had an average rating of with an ave-rage rating of 2.0, with an average of 1.9 for ongoing or completed projects evaluated vs. an average of 2.1 for appraised interventions.

Table 11 Ratings of aid interventions by overall aid quality.

For a project for working with ethnic minorities in Kosovo (XXXIII), the appraisal judged the proposal poor. The proposal was, however, subsequently revised, appro-ved for funding, and is now under implementation.

In terms of the eight evaluation criteria, the following projects and programmes were rated most highly:

Very good: • the completed Post-Emergency Reconstruction in the Field of Meteoro logy Project in Mozambique (XX); • the proposed concessional credit Hydro-Meteorological Project for Peru (XXII); • the completed Programme for Finnish Cooperation in the Education Sector of Bosnia and Herzegovina (CES) (XVII-B); and • the appraised Watershed Management Sub-component of the Tana-Be les Sub-Basin Integrated Water Resources Development Project in Ethio pia (XXV).

Good to very good: • the completed Entrepreneurship Development Project for Women in Food Processing in Central Viet Nam (XIV); and • the appraised (but not funded) INGO Project Proposal on Sustainable Alliances for Education in Kosovo (XXXII).

Various aspects of these different interventions have already been mentioned in the preceding discussion. Many other projects and programmes had notable achieve-ments, but were only rated good overall because of many areas needing improvement. In some cases, they had made good progress in some components, but not in others, i.e., the Forest Policy and Economics Education and Research Project (FOPER) for the Western Balkans (XVI). Some large-scale programmes, such as the Kenya (IX)

The evaluation of the PRORURAL programme in Nicaragua (III) found that it also relied upon national statistics, in this case for agriculture, and that individual projects under the programme had their own regular reporting of data. In between, at the programme level, however, the indicators were considered by the evaluation team to be poorly formulated and inadequately monitored. In the project supporting disability policy development in Bosnia and Herzegovina (XIX), the MTR team found that the project was undertaking surveys, which were building up a valuable dataset of information on persons with disabilities. With respect to the project, however, they argued that many of the indicators were too long-term and not appropriate for the project. As a result, they felt that the project’s achievements were actually being under-reported. The UNIDO project in Viet Nam (XIV) was cited as having comprehensive baseline data, and could document its achievements. The indicators were primarily quantitative, although the evaluators argued that some qualitative indicators could have been useful. They noted that the project had provided regular monitoring and self-evaluation and progress reports – although in this case, the evaluators judged that the project management had done perhaps too much reporting. In an appraisal of the bilateral Rural Water Supply, Sanitation and Hygiene Programme in Benishangul-Gumz Region, Ethiopia (WASH-Beni) (XXVI), the evaluators had urged a reconsideration of the project design, so that the project could serve as a model for others in the national WASH network elsewhere in Ethiopia. In this vein, they also advocated that the programme lead the way in developing model information technology systems, surveys and research, and monitoring and evaluation systems. 6.7 Overall Quality of the Aid Interventions

For the 36 aid interventions, their overall quality had an average rating of with an average rating of 2.0, with an average of 1.9 for ongoing or completed projects evaluated vs. an average of 2.1 for appraised interventions. Table 11 Ratings of aid interventions by overall aid quality.

For a

project for working with ethnic minorities in Kosovo (XXXIII), the appraisal judged the proposal poor. The proposal was, however, subsequently revised, approved for funding, and is now under implementation. In terms of the eight evaluation criteria, the following projects and programmes were rated most highly: Very good:

• the completed Post-Emergency Reconstruction in the Field of Meteorology Project in Mozambique (XX);

• the proposed concessional credit Hydro-Meteorological Project for Peru (XXII); • the completed Programme for Finnish Cooperation in the Education Sector of Bosnia and

Herzegovina (CES) (XVII-B); and

Aid Quality Rating (Number of Interventions Rated)

Type of Report

Total Number of Interventions Poor

(1.0 to 1.5)

Good (2.0)


Very Good (3.0)

Average Rating

Evaluations 23 6 14 1 2 1.9 Appraisals 13 1 9 1 2 2.1 Total 36 7 23 2 4 2.0


and Nepal (XV) programmes, were implemented well in some regions, provinces, and sites, but not so well in others.

It should also be noted that despite some major problems noted in the evaluations, many projects and programmes continued into another phase. As the Meta-analysis Team does not have knowledge regarding what changes were made in response to the evaluation reports, it is impossible to comment further upon this issue.

6.8 Comparison by Region and Sector

The Meta-analysis Team also examined whether or not there were any notable diffe-rences in the aid quality, or the report quality, by geographical region or sector. As indicated in Table 12, there were some differences in report quality by region, with the two reports from Asia both scoring very high, and in aid quality, with the two projects in the Palestine Territories rated as poor. In terms of differences by sector, the ratings for the education and research reports and interventions were quite similar to those for the environment and natural resources sectors. The interventions in other sectors rated higher in terms of report quality, and lower in terms of aid quality. It is difficult, however, to consider these patterns meaningful, given the small sample size.

Table 12 Comparison of report quality and aid quality by geographical region and sector.

7 IMPLEMENTATION OF THE DEVELOPMENT POLICY PROGRAMME

7.1 Status of Implementation of Interventions in this Sample

Given the limited review sample of appraisals and evaluations conducted in 2007 and 2008, only limited observations can be made regarding the 2007 Finnish Develop-ment Policy Programme. As already noted, design of only six of the 36 interventions

• the appraised Watershed Management Sub-component of the Tana-Beles Sub-Basin Integrated Water Resources Development Project in Ethiopia (XXV).

Good to very good: • the completed Entrepreneurship Development Project for Women in Food Processing in

Central Viet Nam (XIV); and • the appraised (but not funded) INGO Project Proposal on Sustainable Alliances for Education

in Kosovo (XXXII). Various aspects of these different interventions have already been mentioned in the preceding discussion. Many other projects and programmes had notable achievements, but were only rated good overall because of many areas needing improvement. In some cases, they had made good progress in some components, but not in others, i.e., the Forest Policy and Economics Education and Research Project (FOPER) for the Western Balkans (XVI). Some large-scale programmes, such as the Kenya (IX) and Nepal (XV) programmes, were implemented well in some regions, provinces, and sites, but not so well in others. It should also be noted that despite some major problems noted in the evaluations, many projects and programmes continued into another phase. As the Meta-analysis Team does not have knowledge regarding what changes were made in response to the evaluation reports, it is impossible to comment further upon this issue. 6.8 Comparison by Region and Sector The Meta-analysis Team also examined whether or not there were any notable differences in the aid quality, or the report quality, by geographical region or sector. As indicated in Table 12, there were some differences in report quality by region, with the two reports from Asia both scoring very high, and in aid quality, with the two projects in the Palestine Territories rated as poor. In terms of differences by sector, the ratings for the education and research reports and interventions were quite similar to those for the environment and natural resources sectors. The interventions in other sectors rated higher in terms of report quality, and lower in terms of aid quality. It is difficult, however, to consider these patterns meaningful, given the small sample size. Table 12 Comparison of report quality and aid quality by geographical region and sector.

Geographical region

Number of reports

Average report quality

Number of projects or programmes

Average aid quality

Africa 13 2.0 14 2.0 Western Balkans 9 2.2 11 2.0 Latin America 7 2.4 7 2.0 Asia 2 2.7 2 2.2 Palestine Territories 2 2.0 2 1.2 Sectors Education and research 8 2.1 11 2.1 Environment and natural resources

14 2.0 14 2.0

Other sectors 12 2.4 12 1.7

7 IMPLEMENTATION OF THE DEVELOPMENT POLICY PROGRAMME 7.1 Status of Implementation of Interventions in this Sample


assessed after began in October 2007 or later, i.e., after the new development policy was formally adopted. Moreover, a number of the projects involve collaboration with other development partners, such as multi-donor initiatives, with other bilateral donors, with a multilateral development financing institution, or with a UN agency. As such, then, they may follow other development priorities and policies, which may overlap with Finnish priorities to varying degrees.

Certainly all the evaluations, whether done in 2007 or 2008, were of interventions that had been designed under a previous policy. (In addition, some of the projects or programmes evaluated in 2007 or 2008 have continued, or new phases or follow-up projects have started – but any incorporation of the 2007 policy into new phases or follow-up projects would not have been reflected in the reports examined.)

Thus, the 11 appraisals conducted after October 2007 really constitute the sample where one could expect to see the new development policy more clearly reflected. Nine of these 11 appraisals (seven (XXIV, XXVI, XXVII, XXVIII, XXIX, XXX, XXXIII) subsequently funded – and two more (XXII, XVII) still pending) were for projects in the environment or related sectors (meteorology, water, forestry).

Through interviews with MFA staff and other sources, the Meta-analysis team was able to learn some information regarding the follow-up to the completed project ap-praisals. Of the 13 projects and programmes appraised, seven have gone ahead and are now under implementation.

Only one project appraised was not recommended by the appraisal mission for sup-port, but in that case, additional work was done on that INGO proposal to work with ethnic minorities in Kosovo (XXXIII), and the project was ultimately appro-ved and funded. Two proposed interventions are still pending. For the concessional meteorological project (XXII) appraised, the project is still under consideration by the Peruvian government. For a large watershed management programme in Sudan (XXV), to be funded by Finland and the Global Environment Facility (through the WB), underwent a pre-appraisal that found further programme preparation was nee-ded, so that was ongoing.

In two cases, the appraisals had recommended approving interventions that had at least initial host government support, but that support eventually dissipated. The pro-posed bilateral programme for promotion of gender equality and women’s rights in Nicaragua (XXI) had been considered to be very much in line with both Finnish and Nicaraguan development policies, but evidently with the change in government in Nicaragua, adequate support was not forthcoming. In Ethiopia, a donor group had worked with the government to develop ideas for a capacity-building pool fund (CBPF), to support the water supply, sanitation, and hygiene (WASH) sector (XXIII). The government and donors already had a similar facility successfully working in another sector. In this case, the appraisal synthesized the key elements of the propo-sed approach, compared it with international experience, and made recommendations


to move forward. The appraisal report had been highly valued for its contributions. Ultimately, however, the Government of Ethiopia decided that it did not want to proceed.

Two INGO proposals for Kosovo (XXXI, XXXII) had been appraised and recom-mended for funding. In the end, however, the MFA determined that the two app-licants did not meet their definition of “international NGOs,” and thus were not eligible for support. This situation raises the question of why the applicants’ eligibility had not been screened prior to conducting appraisals of their proposals.

Of the ongoing or completed projects assessed in this Meta-analysis, several have continued to be active. In some cases, the evaluation was a MTR , and thus the pro-ject continued as originally expected, or in some cases received an extension. For other programmes, funding of a new phase has started, as was the case for the Kenya Gender and Governance Programme and the Forest Policy and Economics Educa-tion and Research Programme in the Western Balkans. In Nepal the School Sector Reform Programme (2009-2014) has recently started, which follows up Nepal’s Edu-cation for All Programme (2004-2009).

7.2 Key Elements of Implementation

The 2007 Development Policy Programme places major emphasis on promoting ef-fective and sustainable development, with sustainable development being understood to take into consideration the economic, social and ecological dimensions of sustai-nability. Other areas of emphasis are on policy coherence and complementarity, en-suring that cross-cutting issues are mainstreamed into all development interventions, and promoting regional integration and security. Another concern is to look at what difference does Finnish support make, what is Finland’s comparative advantage and added value.

With specific reference to the issue of sustainability, it should be noted that many eva-luation TORs and evaluation teams find it easiest to assess development interventions using the eight dimensions that have been stressed for many years: 1) correspondence with local policies; 2) development of local institutional capacity (including strengt-hening of local human resources); 3) economic/financial soundness; 4) suitability to local socio-cultural context; 5) participation of local stakeholders; 6) optimal partici-pation of men and women; 7) impacts on environment; and 8) appropriateness of technology (Ministry for Foreign Affairs of Finland 1999). Consideration should be given to how to integrate use of this approach vis-à-vis the current emphasis on the three broader elements, or pillars, of sustainability – ecological, social, and econo-mic.

It can be argued that the interventions assessed in this Meta-analysis have been ma-king some meaningful contributions to sustainable development. If one wishes to


see development that is clearly ecologically, socially, and economically sustainable, then the interventions in environmental and natural resource management sectors most clearly meet all these criteria. For example, support to a watershed management programme can have ecological, social, and economic benefits that could be sustained after the donor support phases out. For an education or health programme with clear social and economic benefits, it may be more difficult to identify sustainable ecologi-cal benefits.

But ultimately development is about improving human capabilities and options, pro-viding people with more freedom to not only meet their basic human needs, but to flourish as creative members of human societies. Therefore, interventions in other sectors, such as education, health, and governance are also vital for sustainable deve-lopment, even if their ecological linkages are less direct.

Finland is clearly adding value with its development cooperation. In some inter-ventions, Finnish expertise, or collaboration with Finnish institutions, has been well utilised. This issue, however, is one that bears more careful reflection and conside-ration, to see how it could be enhanced. It is of interest to note that the MFA has recently published a document on what Finland has to offer in one specific area of technical expertise – There’s a Map Behind all Development: Finnish Expertise on Mapping and Geographic Information Systems for Developing Countries (Ministry for Foreign Affairs of Finland 2009e).

In terms of appraising and evaluating individual interventions against the 2007 deve-lopment policy, further guidance is needed on how the policy is to be implemented. The MFA has already provided guidance in various key documents, such as, in 2008, its Guidelines: Budget Support Cooperation in Finland’s Development Policy (Ministry for Foreign Affairs of Finland 2008b), and in 2009, Africa in Finnish Development Policy: Finland’s De-velopment Policy Framework Programme (Ministry for Foreign Affairs of Finland 2009a), Development Policy Guidelines for Forest Sector (Ministry for Foreign Affairs of Finland 2009b), and Finnish Development Policy Guidelines for Environment Sector (Ministry for Foreign Affairs of Finland 2009d). Also in 2009, MFA has collaborated with the Finnish Ministry of Agriculture and Forestry and the Ministry of Environment in publishing an International Strategy for Finland’s Water Sector (Ministry for Foreign Affairs of Finland, Ministry of Agriculture and Forestry & Ministry of Environment 2009).

8 CONCLUSIONS

8.1 Trends over Time

When comparing the 2007-08 evaluation and appraisal reports with the Meta-analysis of 2006 development evaluations (White P & Stenbäck T 2007) and the synthesis study of the 1988-1995 evaluations (Koponen J & Mattila-Wiro P 1996), several key issues stand out. One must remember, however, that each analysis has its own spe-


cific focus, and a different sample of development evaluations, which makes it more difficult to compare the three analyses.

Terms of Reference for Evaluation ReportsFirst, in terms of the TOR for the evaluation reports, the synthesis study had noted that there was much variation in the TOR and the evaluation guidelines were out-dated. (Subsequently, the MFA issued guidelines in 1999, and then came out with a newer version for piloting in 2007.) The 2006 Meta-analysis found that TOR stan-dards, whether MFA, EU, or OECD/DAC, were often not followed. It also noted the need for TOR to be cognizant of what is manageable within the constraints and resources (time, personnel, etc.) available. It provided a number of recommendations for improvement of TOR. In this current Meta-analysis, again the quality of TOR was found to be variable, in terms of providing clear guidance on the intervention to be assessed and any special issues to be examined. As evaluation reports generally respond to their TOR, having well-written TOR is extremely important.

Quality of the Evaluation Reports The synthesis study found that report quality was variable, generally not very high. The synthesis study team rated the reports for their own purposes, but did not pub-lish the information. They found that the reports tended to focus on achievements, performance, and outputs. In many of the reports, they had observed a lack of inter-nal coherence in the argumentation. The study team considered the broader, thema-tic, or ex post evaluations to be better done than the evaluations of individual projects or programmes.

The 2006 Meta-analysis rated report quality using EC quality standards. The average report was considered to be good. The authors found that the weakest areas in the-se reports were appropriate design and valid conclusions. They recommended that evaluation reports focus more on future improvements, less on criticism of past per-formance, and should provide a limited, manageable number of recommendations. (White P & Stenbäck T 2007.)

Similar to the past two analyses, this Meta-analysis found the quality of the evaluati-on reports variable, but generally good. It used the OECD/DAC evaluation quality standards, which included some items that were extremely difficult, if not impossible, to rate on the basis of the limited information available in the evaluation reports. As with the previous Meta-analysis, this current exercise also found that it was difficult to obtain complete information.

All three analyses have concluded that improvements are needed in management of evaluations, use of guidelines, and providing more resources for evaluation (both consultant and MFA management time). The synthesis study had recommended that results need to be made more accessible, as in issuing compact and timely publica-tions, or through other innovative feedback mechanisms. The 2006 Meta-analysis mentions the need for more systematic follow-up on report recommendations and


dissemination (White P & Stenbäck T 2007). This current analysis agrees that these conclusions are also valid for the more recent sample of development evaluations. The information and lessons learned need to be more widely shared, both within the MFA and the larger development community.

Quality of the Aid and Implementation of the Development Policy The synthesis study primarily focused its work on the quality of the aid interventions, with careful consideration of the five OECD/DAC evaluation criteria. It noted that although the reports generally rated relevance of the projects and programmes highly, the evaluators usually took the issue of relevance “for granted,” and did not critically re-examine or question it. In terms of effectiveness, it considered that most of the projects or programmes had attained, or were likely to attain, their objectives. But it noted that the overall examination of effectiveness depends upon the quality of the design, and in particular, the logical framework, and this area was weak in many pro-jects. The synthesis study judged that increasingly, greater consideration was being given to sustainability, environmental issues and gender, over the time period (1988 to 1995) studied, but it argued that the requirements for sustainability need clearer and explicit specification. It found that most projects had a very poor knowledge base, with insufficient, poor or missing data, poor monitoring, and inadequate knowledge of social, cultural, and policy issues. Given the weak knowledge base, then it was no surprise that the issue of impact was little considered by either the projects or the evaluations. This issue was one requiring major improvement. Finally, in terms of ef-ficiency, this study found that efficiency had been given a low priority by both projects and evaluations. Nonetheless, they noted that some projects had high costs due to the number of Finnish personnel and procurement-sourcing requirements, but that this situation had been declining over the study period. The authors recommended that more focus be paid to economic efficiency.

The 2006 meta-analysis did not structure its discussion of aid quality around the five OECD/DAC evaluation criteria. Rather it focused more on the issue of how the interventions – and the evaluation reports – complied with the 2004 development policy (Ministry for Foreign Affairs of Finland 2004a), in terms of poverty reduction; protection of the environment, human rights, equality and democracy; sustainability; MDGs; Paris Declaration; and transparency and good governance. It also assessed the awareness of and treatment of cross-cutting issues in Finnish policy. With respect to sustainability, it observed that for many interventions, there were problems of sustainability, due to inadequate / poor local institutional ownership, and thus un-certainty regarding continuity after donor funding ended. The report recommended that the concept of “Finnish added value” be considered, in terms of how it could be evaluated in the future. (White P & Stenbäck T 2007.)

As with the synthesis study, the current Meta-analysis found that problems persist with inadequate knowledge bases, information systems, baseline data, indicators and monitoring. Also, development interventions tend to be weak in considering impact and efficiency. Although the issue of sustainability is certainly being considered more


than previously, it still needs further attention, in terms of how it is conceptualized, measured, and promoted in the implementation of projects and programmes. For this sample of interventions, relevance and policy coherence were rated highest, alt-hough many elements of policy coherence – such as the Paris Declaration, MDGs, or poverty reduction – were given scant attention. Finnish added value was only consi-dered in one-quarter of the reports, usually in terms of the use of Finnish expertise or Finnish personnel.The overall usefulness of the development evaluations is difficult to assess.

As noted in the synthesis study, the majority of the evaluations, especially MTR s, focused primarily on the management - rather than accountability - functions of eva-luations. In other words, they aimed to address the needs of managers - such as regarding informing management decisions for funding, implementation, possible ex-tension or replication. They were less concerned with the larger issues of accountabi-lity, and how the lessons learned in one programme or project could be more widely shared. This situation still holds true today. It is vital that the MFA consider how to broaden the use of the evaluations, to make them more effective as learning tools and to more widely share experiences, both of relative successes and relative failures.

8.2 Broadening the Use of Development Evaluations

The appraisals and evaluations fulfil several different functions. Project preparation reviews, pre-appraisals, and appraisals, generally focus on assessing the adequacy and feasibility of the overall design, with particular focus on institutional arrangements and financial mechanisms that need to be in place before the negotiations can go ahead. In terms of MTRs or other reviews during implementation, the emphasis is generally on implementation issues, as well as assessing performance and progress to date. Such reviews may also be tasked with assessing the possible need for an extension, or providing suggestions for a follow-on phase. Thus, for these reports, the overall emphasis in contracting consultants to undertake these assessments is on solving some practical problems for the MFA, to improve implementation. As such, the reports generally are only circulated among members of the country team (inclu-ding sectoral advisors) and within the country concerned, among relevant authorities, members of Supervisory Boards or Steering Committees, and the programme or project management team. In this regard, these reports generally addressed specific questions posed in their respective TOR, and thus were quite useful for the MFA.

For final or ex post evaluations, the purposes tend to be somewhat different. The emphasis is more on documenting what has been achieved. In many cases, where the idea is that the cooperation will continue, such assessments try to draw lessons learned for possible replication – perhaps regionally, in other areas, or through other modalities of development cooperation. In Finnish development cooperation, there seems to be considerable discretion in whether or not to undertake such assessments. This situation contrasts with that in some other development assistance organisations,


where efforts are made to undertake final, ex post or impact evaluations of all – or a much broader sample – of interventions.

To date, the MFA has not made broad use of the information developed in these appraisals and interventions, in terms of sharing lessons learned, or undertaking a comparative performance review of the overall development portfolio. If the MFA were to decide to pursue this idea, then efforts will be needed to undertake appraisals and evaluations in a manner in which they could be more easily compared. An overall performance review of development effectiveness would need to include not only evaluation reports, but a wider range of information. The Government of Australia, for example, is currently developing such an overall performance review system for its development assistance programme (AusAid 2008).

Many other development organisations have adopted standardised rating systems for projects and programmes. In this sample, for example, the UNIDO evaluation (XIV) provided such an assessment of the project. Such ratings are also used by other UN agencies, such as the United Nations Development Programme (UNDP), and then entered into databases for comparative purposes and overall performance review.

For more systematic follow-up of decentralised evaluations commissioned by regional units within the MFA, it may be important to consider development and adoption of a standardized management response format. Currently follow-up decisions to de-centralised appraisals and evaluations are documented and filed in the MFA archives. For example, a MTR would be considered by the project’s Steering Committee and/or Supervisory Board, and any decisions regarding follow-up would be documented. Such information, however, is difficult to retrieve for purposes of meta-analyses, or other systematic follow-up for improved management and learning. The MFA, ho-wever, is already using a standardized management response form for the broader and thematic evaluations commissioned by EVA-11.

For the evaluation reports themselves, there is also the possibility to develop stan-dardised ratings. It should be noted, however, that if the MFA wishes to apply the OECD/DAC evaluation quality standards, then the rating process will need to be done by the evaluation manager – not by the consultants. This Meta-analysis iden-tified certain difficulties in applying these standards as they are currently formula-ted. The OECD/DAC Network on Development Evaluation hosted an internatio-nal workshop in early 2009, which reviewed experience in using these OECD/DAC evaluation quality standards, and gave numerous suggestions for their improvement (OECD 2009).

In order to promote greater sharing of information generated by the various types of evaluation reports, the MFA could consider developing various means for sha-ring key information and highlights, which might include a communication strategy for dissemination of evaluation results. Some other development organisations have such approaches. Over the years, the WB’s Operations Evaluation Department has


produced various evaluation summaries, such as its two-page Evaluation Briefs, as well as its four- to six-page Précis series. The OECD/DAC Evaluation Feedback for Effective Learning and Accountability (OECD 2001) workshop proceedings and background pa-per review a number of other examples, citing especially the efforts of Denmark to develop a range of different evaluation products, e.g., full reports, popular 25-page summaries, 4-page summaries, videos or films, articles in newsletters, etc., to dissemi-nate results to different audiences.

Finland is supporting some notable development cooperation efforts. Such work de-serves to be more widely known, not only within the MFA, but also among the broa-der development community and general public.

As an overall conclusion, it can be noted that Finnish development cooperation is ma-king an important difference, but there remains room for improvement, both on the level of individual interventions and for the system as a whole. Evaluations play an important role in such quality improvement, and thus efforts are needed to upgrade the overall quality of evaluations, and use information from them more systematically and more strategically, to increase the overall quality of Finland’s development coope-ration programme.

9 RECOMMENDATIONS

On the basis of this Meta-analysis, the following are recommended:

1. To facilitate future meta-analyses of evaluations, or other means of sharing evaluation results, it would be important to agree upon the standards to be used, and then develop the tools and provide the training to assist the evalu-ators and evaluation managers to meet these standards. Although overall the evaluations have been well used as management tools, their potential use for broader learning and accountability could be further improved. MFA should establish minimal standards to be addressed in all appraisals and evaluations but allow flexibility for adding additional information (special issues) or ot-her modifications as needed for the individual assignment. Moreover, efforts need to be made to strengthen the MFA’s archiving and database systems, to ensure that more comprehensive records of such evaluations are accessible and available.

2. Although the evaluations were generally of a good quality, they could be further improved. Key elements for improving evaluations are improving the TOR, and providing more resources for the evaluation (including both more management time and more consultant time).

3. To improve the quality of development interventions, more attention needs to be focused on issues of baseline data, indicators, monitoring, intended im-


pacts, and issues of effectiveness and sustainability. This change will require much more emphasis on and training in managing-for-results.

4. Efforts should be made to document any follow-up and action based upon evaluation findings and recommendations, perhaps through wider use of tools such as management response formats.

5. Greater efforts are needed to share lessons learned from evaluations and ap-praisals, through various communication channels, both within and outside of the MFA. One possible tool might be a series of short evaluation briefing notes, 4-6 pages, which could be made available on the MFA web pages, as well as in printed format.

6. In terms of implementation of the Finnish Development Policy Program-me, it is important to recognize the time need to make significant changes in the portfolio, and the operational guidance needed to operationalise new key priorities and concepts.


REFERENCES

Annex 3 The Evaluated Reports by Author provides the list of documentation provided by MFA for the Meta-analysis.

Table 1 also provides a numbered list of the reports, projects or programmes inclu-ded in the Meta-analysis.

AusAid 2008 Annual Review of Development Effectiveness 2007. Australian Government, AusAID, Office of Development Effectiveness, Canberra, 60 p.

Copenhagen Development Consulting & Edburgh Consultants 2007 Mid-Term Review of Nepal Education for All (EFA) Programme 2004-2009, Final Report. Copenhagen De-velopment Consulting, Denmark and Edburgh Consultants, Netherlands, 114 p.

European Commission 2006 Evaluation Quality Assessment: Guidelines for filling in the quality assessment grid. European Commission, Brussels, 9 p. http://ec.europa.eu/euro-peaid/evaluation/methodology/guidelines/gui_qal_flr_en.htm

Everatt, D Chesoni A, Nyamweya P & Kanyinga K 2008 Evaluating the Gender & Go-vernance Programme Kenya 2008, Final Report. South Consultants, 131 p.

Government of Finland 1996 Decision-in-Principle on Finland’s Development Cooperation. Government of Finland, Helsinki, 12 September, 7 p.

Government of Finland 1998 Finland’s Policy on Relations with Developing Countries. Go-vernment of Finland, The Cabinet, Helsinki, 15 October, 19 p.

Government of Finland 2001 Operationalisation of Development Policy Objectives in Finland’s International Development Cooperation. Government Decision-in-Principle, Helsinki, 22 February, 8 p.

High-Level Forum on Aid Effectiveness 2005 Paris Declaration on Aid Effectiveness: Ow-nership, Harmonisation, Alignment, Results and Mutual Accountability. Paris, 2 March, 12 p.

Hirstiö-Snellman P 1991 Review of Evaluation Reports on Finnish Development Projects in the 1980s. Ulkoasiainministeriö, Kehitysyhteistyöosasto. ISBN 9514754980.

Koponen J & Mattila-Wiro P 1996 Effects or Impacts? Synthesis Study on Evaluations and Reviews Commissioned by Finnida 1988 to mid-1995. Evaluation report 1996:1. Ministry for Foreign Affairs of Finland, Hakapaino Oy, Helsinki, 408 p.

Ministry for Foreign Affairs of Finland 1985 Project Evaluation: Concept and Guidelines.


Ministry for Foreign Affairs of Finland, Finnish International Development Agency, Evaluation Section, Helsinki, 49 p.

Ministry for Foreign Affairs of Finland 1993 Finland’s Development Cooperation in the 1990s: Strategic Goals and Means. Ministry for Foreign Affairs of Finland, Department for International Development Cooperation, Helsinki, 18 p.

Ministry for Foreign Affairs of Finland 1999 Guidelines for Programme Design, Monitoring and Evaluation. The Ministry for Foreign Affairs of Finland, Helsinki, 69 p.

Ministry for Foreign Affairs of Finland 2004a Development Policy. Government Resolution 5.2.2004. The Ministry for Foreign Affairs of Finland, Helsinki, 39 p.

Ministry for Foreign Affairs of Finland 2004b Finland’s Report on the Millennium Develop-ment Goals 2004. The Ministry for Foreign Affairs of Finland, Helsinki, 32 p.

Ministry for Foreign Affairs of Finland 2007a Development Policy Programme 2007: To-wards a Sustainable and Just World Community, Government Decision-in-Principle. The Ministry for Foreign Affairs of Finland, Erweko Painotuote Oy, Helsinki, 39 p.

Ministry for Foreign Affairs of Finland 2007b Evaluation Guidelines: Between Past and Future. The Ministry for Foreign Affairs of Finland, Hakapaino Oy, Helsinki, 84 p. ISBN 978-951-724-624-8.

Ministry for Foreign Affairs of Finland 2008b Multilateral Cooperation in Finland’s De-velopment Policy. Policy Paper. The Ministry for Foreign Affairs of Finland, Helsinki, 27 p.

Ministry for Foreign Affairs of Finland 2009a Africa in Finnish Development Policy: Finland’s Development Policy Framework Programme. The Ministry for Foreign Affairs of Finland, Erweko Painotuote Oy, Helsinki, 44 p.

Ministry for Foreign Affairs of Finland 2009b Development Policy Guidelines for Forest Sector. The Ministry for Foreign Affairs of Finland, Erweko Painotuote Oy, Helsinki, 19 p.

Ministry for Foreign Affairs of Finland 2009c Finland’s Development Cooperation 08. The Ministry for Foreign Affairs of Finland, Erweko Painotuote Oy, Helsinki, 78 p. ISBN 978-951-724-735-1.

Ministry for Foreign Affairs of Finland 2009d Finnish Development Policy Guidelines for Environment Sector. The Ministry for Foreign Affairs of Finland, Erweko Painotuote Oy, Helsinki, 28 p.

Ministry for Foreign Affairs of Finland 2009e There’s a Map Behind all Development:


Finnish Expertise on Mapping and Geographic Information Systems for Developing Countries. The Ministry for Foreign Affairs of Finland, Erweko Painotuote Oy, Helsinki, 40 p. ISBN 978-951-724-768-9.

Ministry for Foreign Affairs of Finland, Ministry of Agriculture and Forestry, & Mi-nistry of the Environment 2009 International Strategy for Finland’s Water Sector. The Mi-nistry for Foreign Affairs of Finland, The Ministry of Agriculture and Forestry, and The Ministry of the Environment, Erweko Painotuote Oy, Helsinki, 29 p.

OECD 2001 Evaluation Feedback for Effective Learning and Accountability. Organisation for Economic Co-operation and Development, Development Assistance Committee, Paris, 117 p.

OECD 2006 DAC Evaluation Quality Standards (for test phase application). Organisation for Economic Co-operation and Development, Development Assistance Committee, Paris, 8 p.

OECD 2007 Finland: Peer Review. Organisation for Economic Co-operation and De-velopment, Development Assistance Committee, Paris, 90 p.

OECD 2009 DAC Evaluation Quality Standards Workshop: Workshop Report. New Zealand’s International Aid & Development Agency, Auckland, 9-10 February, 29 p.

White P & Stenbäck T 2007 Meta-analysis of Development Evaluations in 2006. Evaluati-on report 2007: 2. Ministry for Foreign Affairs of Finland, Hakapaino Oy, Helsinki, 88 p. ISBN 978-951-724-632-3.

World Bank 2009 http://web.worldbank.org/external/projects/main?Projectid=P074633 (Website for Nepal Education for All Programme, accessed 20 October 2009)


ANNEX 1 TERMS OF REFERENCE

Ministry for Foreign Affairs of Finland Office of the Under-Secretary of StateDevelopment Evaluation

Terms of Reference

META-ANALYSIS OF DEVELOPMENT EVALUATIONS IN 2007 AND 2008(89855701)

1 Background

In the Ministry for Foreign Affairs of Finland the evaluation function is divided into the central evaluations, meaning those carried out by the Development Evaluation attached to the office of the Under-Secretary of State for development, and into the decentralised evaluations, which are done by the regional departments and units and in some cases also by the embassies. Thus, in addition to the 5-7 evaluations per-formed by the central system, the Ministry for Foreign Affairs commissions a large number of evaluations, mid-term reviews (MTR) and other assessments of aid, which target specific projects or restricted topics. There has been two Meta-analysis done on the entire evaluation volume of both centralised and decentralised evaluations, one in 1996 (covering the years 1988-1995, 150 reports) and the other one in 2007 (co-vering the year 2006, 29 reports). When the conclusions of these two Meta-analysis are compared, fairly little improvement in terms of sustainability and impact can be observed.

However, in 2007 the Government of Finland adopted a new Development Policy Programme in which sustainability is one of the key dimensions contributing to the overall poverty reduction and achievement of the Millennium Development Goals. The new development policy has now been operationalised for about two years. Thus, it is interesting to see how much it has influenced implementation. The current Meta-analysis, covering the years 2007 and 2008 of ministry-wide evaluation activity, shall also contribute to information on the attainment of the objectives of the new po-licy. Particularly interesting angles are effectiveness and sustainability of results, and through sustainability the potential long-term impacts.

Lessons extracted from projects and programmes implemented in different countries and sectors can offer insights to the planning and management of other projects even if their fields are dissimilar.


2 Purpose and Objective

PurposeThe purpose of the meta-evaluation is to fulfil the need to widen the scope of eva-luation benefits and use of results in institutional learning, in other words, to offer system-wide opportunity to learn from past experience and to extract good practices for quality development of development assistance.

ObjectiveThe objective of this meta-evaluation is two-fold. Firstly to have an analysis of de-velopment assistance evaluations of the Ministry during 2007-2008 on the operatio-nalising of the new development policy (section 4 a below). The Meta-analysis will elucidate the effectiveness and sustainability dimensions in particular. Secondly, the objective is to assess the overall quality of aid and the quality of evaluation reports assessed against the OECD/DAC quality criteria (section 4 b below).

UsersThe users of the results of the meta-evaluation are the staff and decision-makers of the ministry and the wider constituency implementing development aid.

3 Scope

The meta-evaluation will assess and collate information contained in the evaluation reports collected ministry-wide. Evaluations carried out by the centralised Develop-ment Evaluation will be added to this analysis. The total number of reports to be ana-lysed shall not exceed 30. The bulk of the work is document analysis, but if necessary the evaluators shall make relevant contacts to the respective units and embassies for clarifications.

4 Evaluation Issues

4 a The Quality of Aid

The evaluation issues are largely geared to assessing the operationalisation of deve-lopment policy programme, in terms of effectiveness and results, and assessment of sustainability. Yet, also the other OECD/DAC development evaluation criteria, rele-vance and efficiency attribute to sustainability and effectiveness. Thus relevance and efficiency should be looked at in this particular context.

The following overall questions shall be addressed:

• Distribution of evaluations per sector and per country?• Share of major aid partner countries among evaluations?• Share of different aid instruments among evaluations?• Estimation of mainstreaming and consideration of the cross-cutting themes?


• Assessment of coherence to the Finnish policy and the policy of the recipi-ent country?

• Assessment of sectoral coherence of Finnish development interventions in a country?

• Assessment of complementarity of Finnish aid within the overall develop-ment context of the partner country?

• Coordination of the Finnish aid with other donors?• How is the Paris Declaration principles featured in the implementation? • Achievement of Millennium Development Goals? Achievement of poverty

reduction?

Assessment of effectiveness and results:

• Are there concrete results discernible; at output/outcome/result/impact le-vel?

• What types of projects have produced best results (local implementation, multilateral cooperation, bilateral cooperation or other)?

• Which sectors / themes have produced discernible results (water, forestry, education etc.) in reasonable time span, at what level?

• What is the Finnish value added in the cases studied?• What has been the method of implementation to attain best Finnish value

added, if any? • Is there a connection between effectiveness and Finnish expertise areas or

presence of Finnish personnel?

Assessment of sustainability:

OECD/DAC defines sustainability as follows:

“Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be envi-ronmentally as well as financially sustainable.”

• Is there proper baseline data available on the projects?• Are the indicators established for monitoring sustainability in the project do-

cuments? How has the monitoring and data on progress been recorded? Is it gender disaggregated? Have the cross-cutting themes been factored in in the monitoring indicators?

• Is there any particular circumstance discernible that stands out as a factor conclusive to good sustainability?

• What have been the obstacles to obtain sustainability?• What are the major risks to sustainability?

What are the assumptions of sustainability?


4 b The Quality of the Evaluation Reports

The following overall questions shall be addressed:

• Have the OECD/DAC evaluation criteria been implemented in evaluation documents?

• Is sustainability covered in ToRs? • Is there coherence between ToRs and reports?• How is knowledge and data management described as having been carried

out?

5 Methodology

The Meta-analysis shall use multiple methods both qualitative and quantitative to draw findings and conclusions and to formulate recommendations. The more precise methodology of this Meta-analysis shall be defined in the inception report at the outset of the evaluation.

6 Working Plan

The evaluation team shall prepare a working plan, called inception report, at the outset of the evaluation. The working plan shall include the division of tasks between the evaluation team and time-tables. The inception report shall also elaborate on the approach and critical issues of the evaluation task as well as define the evaluation methodology. Outlines of the contents of the inception and other reports are available in the recent Evaluation Guidelines - Between Past and Future 2007 of MFA (ISBN 978-951-724-624-8).

7 Expertise Required

The specific requirements of the Evaluation Team are contained in the Instructions to Tender Document, which constitutes Annex A to the Invitation to Tender, this Terms of Reference being Annex B.

The overall requirements towards the evaluators include relevant experience in per-forming different types of evaluations, including Meta-analysis, and knowledge of the methodologies in evaluation and research. Thorough knowledge of the Finnish development policy and the organisational set-up of the Finnish aid system is also required. The qualifications of the Team Members should be mutually reinforcing and complementary. The Team should be gender balanced.

8 Reporting

The report will be written in the English language, and include an abstract of no more than 250 words in Finnish, Swedish, and English and also a summary in Finnish, Swedish, and English. The language should be carefully checked, the text proof-read


and be ready to print.

The following reports shall be submitted:- Inception report / work plan, after two weeks from signature of the contract. It will describe the division of work between the two experts, the approach, critical issues, methodology and time-table.

- Draft Final Report which is nearly the final one, shall be subjected to a round of comments of the stakeholders. The Draft Final Report should be ready by mid-Sep-tember. After receipt of comments, precisions or corrections, the report is finalised. Commenting time is about two weeks.

- Final Report is submitted no later than two weeks after receipt of the comments, with a tentative time table of the first week of October 2009. The final report discusses all evaluation issues and questions, and is organized in separate sections of findings, conclusions and recommendations. It will also feature the methodology used and de-fine the limitations to the evaluation. The report’s editorial shape shall follow detailed instructions provided by MFA, be proof-red, copy-edited, and ready to print.

- Oral presentation of results is organized after completion of the evaluation and submission of the Final Report. The team leader is required to do a power point -supported presentation either in a public event or in an event organized particularly to the stakeholders of this evaluation. Preferably also the other member of the team attends the final presentation.

The Final Report shall be clear and concise, with language that is easily comprehensible by ordinary readers. The body-text should not exceed 30 pages. Annex 1 is the terms of refe-rence, and Annex 2 the people interviewed, other annexes can be added as need arises.

9 Time Schedule

The duration of the assignment is estimated to be around 70 working days total for the two experts. The work shall be completed so that the final report is finished no later than the first week of October 2009.

10 Budget

The budget of this evaluation shall not exceed 60.000 Euros. The availability of the entire budget is subject to approval of the final report by the ministry.

11 Mandate

The consultants are expected to make relevant contacts but they are not authorised to make any commitments on behalf of the Government of Finland.

Aira PäivökeDirectorDevelopment Evaluation























T REPORT 2004:4 Refocusing Finland’s Cooperation with NamibiaISBN: 955-724-477-0, ISSN: 1235-7618












































T

REPORT 2008:7 ISBN: 978-951-724-716-0 (printed), ISBN: 978-951-724-717-7 (pdf), ISSN: 1235-7618Kosovo ountry rogramme C P REPORT 2004:4 Refocusing Finland’s Cooperation with Namibia

ISBN: 955-724-477-0, ISSN: 1235-7618











































T


















TREPORT 2004:4 Refocusing Finland’s Cooperation with Namibia

ISBN: 955-724-477-0, ISSN: 1235-7618

















Evaluation of the Bilateral Development Co-operation between Finland and Zambia

REPORT 2005:1 Ex-Ante Evaluation of Finnish Development Cooperation in the Mekong RegionISBN: 955-742-478-9, ISSN: 1235-7618REPORT 2008:7

ISBN: 978-951-724-716-0 (printed), ISBN: 978-951-724-717-7 (pdf), ISSN: 1235-7618Kosovo ountry rogramme C P


ISBN: 978-951-724-7 - (printed), ISBN: 978-951-724-7 - (pdf), ISSN: 1235-7618Finland´s Development Cooperation in Central Asia and South Caucasus

28 3 29 0REPORT 2009 :1

ISBN: 978-951-724-7 - (printed), ISBN: 978-951-724- - (pdf), ISSN: 1235-7618Agriculture and Rural Development. Preliminary Study

46 7 747 4REPORT 2009 :2





Support to REPORT 2009 :3

ISBN: 978-951-724-7 - (printed), ISBN: 978-951-724- - (pdf), ISSN: 1235-7618Agriculture and Rural Development. A Preliminary Study


ISBN: 955-724-477-0, ISSN: 1235-7618





















REPORT 2009 :2













































T



Support to REPORT 2004:4 Refocusing Finland’s Cooperation with NamibiaISBN: 955-724-477-0, ISSN: 1235-7618





















REPORT 2009 :3























T

ISBN: (printed), ISBN: 978-951-724- - (pdf), ISSN: 1235-7618Meta-analysis of Development Cooperation on HIV

978-951-724-769-6 770 2REPORT 2004:4 Refocusing Finland’s Cooperation with Namibia

ISBN: 955-724-477-0, ISSN: 1235-7618





















REPORT 2009 :4 / AIDS

ISBN: (printed), ISBN: 978-951-724- - (pdf), ISSN: 1235-7618978-951-724-7 - 7DEMO Finland Development Programme V


ISBN: 955-724-477-0, ISSN: 1235-7618





















REPORT 2009 : 6

REPORT 2009:5 Finnish Aid in Western Kenya; Impact and Lessons Learned ISBN: 978-951-724-783-2 (printed), ISBN: 978-951-724-786-3 (pdf), ISSN: 1235-7618

REPORT 2009:7 The North-South-South Higher Education Network Programme ISBN: 978-951-724-790-0 (printed), ISBN: 978-951-724-791-7 (pdf), ISSN: 1235-7618

REPORT 2009:8 Natural Disasters, Climate Change and Poverty ISBN: 978-951-724-807-5 (printed), ISBN: 978-951-724-808-2 (pdf), ISSN: 1235-7618













































T

REPORT 2009:9 Meta-Analysis of Development Evaluations in 2007 and 2008 ISBN: 978-951-724-809-9 (printed), ISBN: 978-951-724-810-5 (pdf), ISSN: 1235-7618

Documents

Fidida… 2/23.12.08. kansi 14.1.2009 13:34 Sivu 2REPORT 2008:6 The Cross-cutting hemes in the Finnish Development Cooperation ISBN: 978-951-224-714-6 (printed), ISBN: 978-951-224-715-3