Annual Review of Development Effectiveness Shared Global

THE WORLD BANK

THE WORLD BANK

Annual Review of Development Effectiveness

Shared Global Challenges

An

nu

al Review

of D

evelop

men

t Effectiven

ess 2008: Shared

Glo

bal C

hallen

gesT

he W

orld

Ban

k

2008

SKU 17714

ISBN 978-0-8213-7714-7

]|xjHIAICy13771tz]v*:+:!:+:!

ARDE2008_cover_9-12.qxd:ARDE08_cover_9-12.qxd 9/12/08 2:38 PM Page 1

2006 Annual Report on Operations Evaluation

Annual Review of Development Effectiveness 2006: Getting Results

Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank’s Approach to Global Programs

Assessing World Bank Support for Trade, 1987–2004: An IEG Evaluation

Books, Buildings, and Learning Outcomes: An Impact Evaluation of World Bank Support to Basic Education in Ghana

Brazil: Forging a Strategic Partnership for Results—An OED Evaluation of World Bank Assistance

Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy

Capacity Building in Africa: An OED Evaluation of World Bank Support

China: An Evaluation of World Bank Assistance

The CGIAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research

Committing to Results: Improving the Effectiveness of HIV/AIDS Assistance—An OED Evaluation of the World Bank’sAssistance for HIV/AIDS Control

Country Assistance Evaluation Retrospective: OED Self-Evaluation

Debt Relief for the Poorest: An Evaluation Update of the HIPC Initiative

A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995–2005

The Development Potential of Regional Programs: An Evaluation of World Bank Support of Multicountry Operations

Development Results in Middle-Income Countries: An Evaluation of the World Bank’s Support

Economies in Transition: An OED Evaluation of World Bank Assistance

Engaging with Fragile States: An IEG Review of World Bank Support to Low-Income Countries Under Stress

The Effectiveness of World Bank Support for Community-Based and –Driven Development: An OED Evaluation

Evaluating a Decade of World Bank Gender Policy: 1990–99

Evaluation of World Bank Assistance to Pacific Member Countries, 1992–2002

Extractive Industries and Sustainable Development: An Evaluation of World Bank Group Experience

Financial Sector Assessment Program: IEG Review of the Joint World Bank and IMF Initiative

From Schooling Access to Learning Outcomes: An Unfinished Agenda—An Evaluation of World Bank Support to PrimaryEducation

Hazards of Nature, Risks to Development: An IEG Evaluation of World Bank Assistance for Natural Disasters

How to Build M&E Systems to Support Better Government

IEG Review of World Bank Assistance for Financial Sector Reform

Improving Investment Climates: An Evaluation of World Bank Group Assistance

Improving the Lives of the Poor Through Investment in Cities

Improving the World Bank’s Development Assistance: What Does Evaluation Show?

Maintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health andNutrition Outcomes in Bangladesh

New Renewable Energy: A Review of the World Bank’s Assistance

Pakistan: An Evaluation of the World Bank’s Assistance

Pension Reform and the Development of Pension Systems: An Evaluation of World Bank Assistance

Poland Country Assistance Review: Partnership in a Transition Economy

The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank’s Support Through 2003

The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support

Power for Development: A Review of the World Bank Group’s Experience with Private Participation in the ElectricitySector

Putting Social Development to Work for the Poor: An OED Review of World Bank Activities

Small States: Making the Most of Development Assistance—A Synthesis of World Bank Findings

Social Funds: Assessing Effectiveness

Sourcebook for Evaluating Global and Regional Partnership Programs

Water Management in Agriculture: Ten Years of World Bank Assistance, 1994–2004

World Bank Assistance to the Financial Sector: A Synthesis of IEG Evaluations

The World Bank in Turkey: 1993–2004—An IEG Country Assistance Evaluation

World Bank Lending for Lines of Credit: An IEG Evaluation

IEG PUBLICATIONS

All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual selection, please visithttp://www.worldbank.org/ieg

WORKING FOR A WORLD FREE OF POVERTY

The World Bank Group consists of five institutions—the International Bank for Reconstruction and Development(IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), theMultilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of InvestmentDisputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their envi-ronment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public andprivate sectors.

THE WORLD BANK GROUP

ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION

The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses onassessment of IFC’s work toward private sector development, and IEG-MIGA evaluates the contributions of MIGAguarantee projects and services. IEG reports directly to the Bank’s Board of Directors through the Director-General,Evaluation.

The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of theBank Group’s work, and to provide accountability in the achievement of its objectives. It also improves Bank Groupwork by identifying and disseminating the lessons learned from experience and by framing recommendations drawnfrom evaluation findings.

THE INDEPENDENT EVALUATION GROUP


Annual Review of

Development Effectiveness 2008


2008The World Bank

Washington, D.C.http://www.worldbank.org/ieg

©2008 The International Bank for Reconstruction and Development / The World Bank1818 H Street NWWashington, DC 20433Telephone: 202-473-1000Internet: www.worldbank.orgE-mail: [email protected]

All rights reserved

1 2 3 4 5 11 10 09 08

This volume, except for the “Management Response” and the “Chairman's Summary,” is a product of the staff of the Independent EvaluationGroup of the World Bank Group. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views ofthe Executive Directors of The World Bank or the governments they represent. This volume does not support any general inferences beyond thescope of the evaluation, including any inferences about the World Bank Group's past, current, or prospective overall performance.

The World Bank Group does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and otherinformation shown on any map in this work do not imply any judgement on the part of The World Bank Group concerning the legal status of anyterritory or the endorsement or acceptance of such boundaries.

Rights and PermissionsThe material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation ofapplicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and willnormally grant permission to reproduce portions of the work promptly.

For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright ClearanceCenter Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; Internet: www.copyright.com.

All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e- mail: [email protected].

Cover: Indonesian elementary schoolchildren wear surgical masks in response to 2003 outbreak of SARS in China; photo ©Reuters/ Corbis,reproduced by permission. Satellite image of hurricane; photo ©Adastra/ Getty Images, reproduced by permission. Bengal tiger; photo©DLILLC/Corbis, reproduced by permission. Wind turbine farm; photo ©Frank Whitney/ Getty Images, reproduced by permission.

ISSN: 1520-9733ISBN-13: 978-0-8213-7714-7e-ISBN-13: 978-0-8213-7716-1DOI: 10.1596/978-0-8213-7716-1

Library of Congress Cataloging-in-Publication Data

Environmental sustainability : an evaluation of World Bank Group support.p. cm.

Includes bibliographical references.ISBN 978-0-8213-7670-61. Economic assistance—Developing countries—Environmental aspects. 2. Sustainable development—Developing countries. 3. World Bank—Developing countries—Evaluation. I. World Bank.HC60.E53 2008338.9'07091724—dc22

2008028078

Printed on Recycled Paper

World Bank InfoShop

E-mail: [email protected]

Telephone: 202-458-5454

Facsimile: 202-522-1500

Independent Evaluation Group

Knowledge Programs and Evaluation Capacity

Development (IEGKE)

E-mail: [email protected]

Telephone: 202-458-4497

Facsimile: 202-522-3125

i i i

vii Abbreviations

ix Acknowledgments

xi Foreword

xiii Executive Summary

xxi Management Comments: Summary

xxiii Chairman’s Summary: Committee on Development Effectiveness (CODE)

xxvii Evaluation Snapshot in Selected Languages

1 1 Introduction

5 PART I: TRACKING BANK PERFORMANCE

7 2 Development Outcomes: Indicators of Performance9 Measuring Project Performance: Trends from IEG Monitoring12 Project Outcomes in Fiscal 2007 Data14 How It Adds Up: Outcomes of Bank Country Programs

19 3 Underpinning Impact—M&E and Results Management21 Monitoring and Evaluation Systems at the Project Level24 Country-Level M&E: Early Evidence from Results-Based CASs26 Managing Global Programs and Partnerships: An Emerging Agenda28 Improving Our Understanding of Causality: The Use of Impact Evaluations30 Monitoring Institutional Effectiveness

33 4 Lessons and Opportunities35 Practical Lessons for the Near Term35 Directions for the Overall Bank Agenda

37 PART II: SHARED GLOBAL CHALLENGES—LESSONS FROM THE BANK’S EXPERIENCE

39 5 The Challenge of Global Public Goods

43 6 Using the Bank’s Country-Based Model to Foster Global Public Goods: Does It Work?45 How It Works in Theory—the Bank’s Strategic Setting for Fostering Global

Public Goods49 Country Programs in Practice—from Strategy to Action

57 7 The Bank’s Advocacy on Global Public Goods: What Has Worked and What Has Not?59 The Dimensions of Advocacy59 Advocacy at Its Best: The Bank’s Experience with Trade60 The Complex Challenge of Environmental Commons and Climate Change63 Creating a Unified Response: Learning from Avian Flu

Contents

65 Effective Advocacy Benefits from Voice and Representation68 New Dimensions of Advocacy: Innovation through Financial Capabilities

71 8 Improving the Bank’s Support for Global Public Goods: Lessons from Experience

75 Appendixes77 A: Project Performance Results91 B: Monitoring and Evaluation Overview95 C: IEG’s Self-Evaluation: Improving Effectiveness107 D: Features of Global Public Goods111 E: Management Comments

119 Endnotes

123 Bibliography

Boxes10 2.1 What Does a Satisfactory Project Look Like? Illustrations of

Development Impact16 2.2 What Does a Satisfactory Country Program Look Like?23 3.1 M&E Findings and Recommendations in Recent IEG Evaluations24 3.2 Armenia Joint Country Portfolio Performance Review26 3.3 Results Measurement and Monitoring in LICUS27 3.4 Use of CAS Results Frameworks in Country Program Management41 5.1 Key Characteristics of Global Public Goods47 6.1 Brazil: A Best Practice in Integrating GPG Themes in Country

Strategies48 6.2 Emerging Good Practice from the Regions53 6.3 Bank GPG and MIC Strategies: Fates Entwined61 7.1 Advocacy for Carbon Finance: The Bank’s Role in the Growth of a

World Market62 7.2 Clean and Dirty Energy—Can the Bank Do Both? 65 7.3 Importance of Collaboration on Advocacy: HIV/AIDS67 7.4 Governance as Institutional Experimentation: GEF67 7.5 From Shareholder to Stakeholder Model: CGIAR

Figures9 2.1 Project Performance Has Improved over the Medium Term11 2.2 Trends in Sectoral Performance12 2.3 Africa’s Projects Have Improved Substantially but Still Lag Behind

Other Regions15 2.4 CAEs Show Three-Fifths with Outcomes Moderately Satisfactory

or Better 17 2.5 CASCR Reviews Indicate That Bank Programs in MICs Outperform

Those in LICs22 3.1 Projects with Higher Outcome Ratings Have Better M&E Ratings22 3.2 M&E Is Rated Modest or Lower in Two-Thirds of ICR Reviews29 3.3 About 70 Percent of Ongoing Evaluations Are Clustered in Five Areas29 3.4 Nearly Two-Thirds of Ongoing Evaluations Are Located in

Two Regions49 6.1 Bank Expenditures on Main GPG Themes50 6.2 IBRD and IDA Lending for Main GPG Themes

i v

A N N U A L R E V I E W O F D E V E L O P M E N T E F F E C T I V E N E S S 2 0 0 8

55 6.3 Most Global Programs Do Not Focus on GPGs, but Most of the Bank’s GPP Resources Are Devoted to Those That Do

66 7.1 Less Than Half of Funds Committed to Integrated Country Plans Have Been Disbursed

Tables11 2.1 Distribution of Project Ratings Moved Up the Scale in FY03–0713 2.2 Disconnect between the Bank’s Self-Ratings and IEG Ratings

Increased Dramatically in FY0715 2.3 Summary of CAE Ratings, FY98–0817 2.4 Summary of CASCR Review Ratings, FY03–08

C O N T E N T S

v

v i i

Abbreviations

AfricaRMS Africa Results Monitoring SystemARDE Annual Review of Development EffectivenessCAE Country Assistance EvaluationCAS Country Assistance StrategyCASCR Country Assistance Strategy Completion ReportCGIAR Consultative Group on International Agriculture ResearchCODE Committee on Development EffectivenessCPS Country Partnership StrategyDGF Development Grant FacilityFY Fiscal yearGEF Global Environment FacilityGPG Global public goodGPP Global programs and partnershipsIBRD International Bank for Reconstruction and DevelopmentICR Implementation Completion and Results reportIDA International Development AssociationIEG Independent Evaluation GroupIFFIm International Financing Facility for Vaccines and ImmunizationISR Implementation Status and Results reportLIC Low- income countryLICUS Low- Income Countries Under StressM&E Monitoring and evaluationMAR Management Action RecordMIC Middle- income countryNGO Nongovernmental organizationPCF Prototype Carbon FundPRSC Poverty Reduction Support CreditQAG Quality Assurance GroupR&D Research and developmentRMS Results Measurement System

i x

Acknowledgments

This report was written by Thomas O’Brien, andprepared by a team under his leadership. Theteam’s core members were Brett Libresco, YoshiUchimura, Pamela Velez- Vega, Victoria Chang,and Jan Rielaender. The report draws uponsignificant research and contributions providedby Scott Barrett, Robin Broadfield, JohnEriksson, Nidhi Khattri, Shonar Lala, DomenicoLombardi, and Andrew Waxman. Marinella Yadaoand Yvette Jarencio- Lukban provided supportand assistance to the team during the project.William Hurlbut edited an earlier version of themanuscript. Helen Chin edited the report for publication.

Alan Gelb, Steve Radelet, and Salvatore Schiavo- Campo served as peer reviewers of the finalevaluation report.

The evaluation team greatly appreciates the timeand insights from many individuals inside andoutside the World Bank who were interviewedfor this report. The evaluation benefited substan-tially from the constructive advice and feedbackfrom many staff in the World Bank Group.

The evaluation was conducted under the guidanceof Mark Sundberg, Manager, IEG Corporate andGlobal Evaluation and Methods Unit.

Director- General, Evaluation: Vinod ThomasDirector, Independent Evaluation Group, World Bank: Cheryl Gray

Manager, Corporate and Global Evaluation and Methods: Mark SundbergTask Manager: Thomas O’Brien

Indonesian elementary schoolchildren wear surgical masks in response to 2003 outbreak of SARS in China; photo ©Reuters/Corbis, reproduced by permission.

x i

Foreword

A farmer in Kenya faces a hopeful future as heexports to growing markets opened up by globaltrading agreements. But another in Vietnamstares at ruin as her poultry is culled so that thethreat of avian flu can be curtailed. And others inthe Bay of Bengal face longerterm danger from agrowing risk of floods as the earth’s climatewarms. Living continents apart, these farmershave more in common than their efforts toescape poverty. Their futures are increasinglyinterlinked with the world’s response to sharedglobal challenges.

Such global public goods— a global tradingsystem, biodiversity— and the issue of combat-ing global public bads— climate change,transborder contagion— share the problem ofundersupply. To increase supply, and to avoid the“tragedy of the commons,” collective action isrequired, often with coordinated responses atthe local, regional, and global levels. Where thebenefits can be directly felt at the local level, suchas halting transmission of diseases, motivatingaction is less difficult. Where fully appropriatingthe benefits locally is not possible or takes a long time— such as climate change— collective actionis far more difficult and requires global efforts tohelp motivate local action.

The World Bank has set fostering global publicgoods as one of its six strategic priorities underits new president, and is deeply involved inefforts to strengthen their supply through invest-ments spanning local, regional, and global publicgoods. This goes hand- in- hand with the Bank’s

commitment to delivering development results,including through the use of the new round ofInternational Development Association fundingto which donors have committed a record ofnearly $42 billion.

This year’s Annual Review of DevelopmentEffectiveness focuses on assessing the WorldBank’s development effectiveness, with specialattention to global public goods. It notes someencouraging developments. Project perform-ance has improved over the medium term;country programs have worked relatively well inseveral large nations that house a majority of theworld’s poor; and the Bank has increasedattention to collective international action onglobal public goods and advocated effectively onsome of those important challenges. But work isrequired to remedy weaknesses. Notably there isa need to go beyond the Bank’s country- basedmodel when tackling issues where the perceivedlocal and national benefits of action do not matchglobal benefits from collective action. Attentionshould be paid to improving weak performanceof country programs in smaller states and thosewith extensive poverty, and redressing shortcom-ings in applying monitoring and evaluation inprojects and country programs.

Over the next decade and beyond, the success ofthe international community and the World BankGroup in rising to the shared global challenges ofour time will be crucial to reducing poverty and,indeed, to solving the looming challenges theworld collectively faces.

Vinod ThomasDirector- General, Evaluation

Bengal tiger; photo ©DLILLC/Corbis, reproduced by permission.

x i i i

For the World Bank and its partners, the ever-present test is to deliverresults—to lift people out of poverty and promote socially and envi-ronmentally sustainable development. Achieving such success in any in-

dividual country is increasingly intertwined with making progress on sharedglobal challenges. A fair and efficient international trade regime, for example,is a global public good that allows developing countries to trade more and growfaster. The increasing global threat of climate change—a “public bad,” by con-trast—particularly imperils the poor, who bear the brunt of more frequent nat-ural disasters and hazards to health and agriculture.

This year’s Annual Review of DevelopmentEffectiveness is in a new format and presentsevidence on the Bank’s efforts in two importantand connected areas. Part I, which is a standardsection of the new format, helps to track Bankperformance, notably trends in outcomes ofBank projects and country programs, theevolution of monitoring and evaluation (M&E),and the role of evaluation in the results agenda.Part II examines a special topic of greatrelevance to the results described in the firstpart: the Bank’s work in fostering global publicgoods, such as protecting the earth’s climateand preventing the spread of dangerouscommunicable diseases. Global public goodstend to be undersupplied, as are all publicgoods. Motivating local action is easier when thebenefits are captured locally: efforts to stop thetransborder spread of pandemic disease aremore easily motivated when the results directlybenefit local populations. By contrast, reducinggreenhouse gas emissions is harder to motivatebecause of a lack of perceived local benefits,particularly in the near term. The reportexamines both situations, but it is the latter—where global and local benefits diverge—inwhich the challenges are greatest and the role ofthe Bank is potentially pathbreaking.

Development outcomes from Bank lending haveimproved over the medium term, mostlythrough a rise in the share of projects rated asmoderately satisfactory in meeting theirobjectives. Achievements of country programs inmeeting their objectives—typically includinggrowth, poverty reduction, and environmentalsustainability—have been moderately satisfac-tory or better in three-fifths of cases, including inseveral large countries, home to the majority ofthe world’s poor. But too many other programs,particularly in impoverished countries, havebeen moderately unsatisfactory or worse. TheBank’s overall approach to M&E has manystrengths, including recent progress in updatingits policies on lending and country strategies toemphasize M&E. Yet significant overoptimism inthe Bank’s self-assessment of ongoing projectperformance and weaknesses in the use of M&Esystems are of concern. The quality of projectM&E is often quite low, and results frameworksin country assistance strategies need clearer andsimpler articulation with baseline indicators ifthey are to be effective as management tools.

The Bank has paid growing attention to globalpublic goods, which increasingly influencedevelopment outcomes. It has helped foster

Executive Summary

global public goods through country activities,and its country model has worked well whennational and global interests dovetail—oftenwith an agreed international framework foraction—and when grant finance supportscountry-based investments. The Bank has alsobeen a strong advocate for changes in globalsystems, such as international trade reform,where it has expertise and is willing to engage inpublic debate. But the greatest challenges arisewhere local, national, and global benefits—actualor perceived, immediate or for the next genera-tion—diverge significantly from each other. Forexample, the investments needed to protect theearth’s climate and environmental commonsvary considerably at the local, national, andglobal levels, as do the costs and benefits of suchactions. To more effectively bridge the gapbetween global needs and country concerns, theBank should consider: creating dedicatedbudgets and better incentives for country teamsto work on global public goods; deploying itsglobal knowledge networks more effectively;developing new financial instruments andsecuring additional resources, including grantfunds, to support country-level investments; andusing its standing more powerfully to givegreater voice to developing countries in thegovernance of global programs.

Part I: Tracking Bank Performance

Development outcomes from Bank lending haveimproved over the medium term. Over the threeyears to end-fiscal 2007, IEG’s evaluationsconfirm that 80 percent of projects weremoderately satisfactory or better in meeting theirdevelopment objectives. This meets the Bank’sown performance target and is a significantimprovement from the start of the decade. ABank-supported water project in Cambodia,which brought clean water to 750,000 people inPhnom Penh, illustrates such developmentoutcomes.

Project outcomes improved in most sectors, butaverage ratings slipped for projects in the fields ofhealth and public sector governance during fiscal2003–07, as compared with fiscal 1998–2002.

Project performance among the Bank’s Regionsimproved most in Africa—about three-quartersof projects, weighted by disbursement, duringfiscal 2003–07 were moderately satisfactory orbetter in meeting development objectives, ascompared with 60 percent during fiscal1998–2002. There is still a challenge for Africaprojects to improve further and get closer to theperformance in other Bank Regions.

Bank management should avoid overoptimism inassessing ongoing project performance to improvereal-time management for results. The considerableincrease, in fiscal 2007, in the difference betweenthe Bank’s self-ratings of project performanceand IEG’s final ratings of development outcomes(sometimes called the “disconnect”) illustratesthe point. In fiscal 2007, over two-thirds ofprojects rated moderately unsatisfactory orworse by IEG had been reported by the Bank asmoderately satisfactory or better just before theyclosed. Such a wide disconnect—about twice aslarge as in fiscal 2005 and fiscal 2006—meansmanagement is less likely to identify problemprojects and take timely remedial action.

Such management attention is important giventhat the share of projects with moderatelysatisfactory or better outcomes has fallen fromnearly 83 percent in fiscal 2006 to 76 percent infiscal 2007. A single year’s data is not, by itself, acause for alarm, but vigilance is needed to ensurethat it does not foreshadow a persistent decline.Excessively complex project design and overlyambitious assumptions on political ownershipand implementation capacity lay at the heart ofmany poorly performing projects that exited infiscal 2007.

Securing strong development outcomes at thecountry level has proved challenging. Over the past10 years, evaluations of 81 Bank countryprograms—incorporating projects, policy andtechnical advice, and other types of assistance—show that three-fifths of them were moderatelysatisfactory or better in meeting their develop-ment outcomes. Looking at specific grades onIEG’s ratings scale, the Bank succeeded insupporting satisfactory outcomes in 30 percent

x i v


of evaluated programs—including several largeand important countries such as Brazil andChina, which have made strides in reducingpoverty. A further 30 percent of countryprograms were rated moderately satisfactory. Butthe remaining 40 percent of programs—concen-trated in countries that are smaller or haveextensive poverty, such as Malawi—weremoderately unsatisfactory or worse in meetingtheir stated development objectives. Very fewcountry programs are producing best-practiceresults. Indeed, of 36 programs rated since fiscal2002, not one has been highly satisfactory. At thesame time, no program has ever been ratedhighly unsatisfactory.

How well is the Bank using and learning from goodmonitoring and evaluation (M&E) systems, which arekey to improving its effectiveness over the longerterm? The Bank’s overall approach to M&E hasmany strengths, and in recent years there hasbeen considerable progress in updating itspolicies on lending and country strategies toemphasize M&E. The introduction of results-based country assistance strategies has been aparticularly significant step. But considerableroom for improvement remains in putting all ofthis into practice.

At the project level, the overall quality of M&E hasbeen low—rated as modest or negligible in two-thirds of projects for which data are available—since fiscal 2006. Some of the factors contributingto low M&E quality assessments were poorlydesigned results frameworks, poorly articulatedresults chains linking outputs with outcomes,and performance indicators lacking baselinesand targets.

Effective results frameworks at the country level arekey to managing for results. While staff are gainingexperience with results frameworks, too often suchframeworks have been poorly formulated and hencetheir usefulness is undermined. In many cases,frameworks identify too many outcomes andmonitoring indicators and lack baselines andtargets. Their use for monitoring and managingthe country program, and for informing countryassistance evaluations, is very limited because of

poor design and the absence of incentives toconduct M&E. Even so, there are examples ofemerging good practice such as the “Moldovaresults scorecard,” which links country programmanagement and resource allocation.

The Bank has improved its approach to managingand monitoring global programs and partnerships.The Bank now has more robust systems to trackinvolvement in global programs and partner-ships, thus encouraging selectivity and quality atentry. All programs receiving Development GrantFacility funding of $300,000 or more, over the lifeof the program, are also subject to independentprogram-level evaluations. But an IEG assess-ment of a cross-section of such evaluations foundtheir quality frequently compromised by weakM&E systems, particularly a lack of systematicevidence on the achievement of programs’objectives at the outcome level. Therefore, it isdifficult to say whether the global programsreviewed—together accounting for about $100million of annual spending—ultimately had asubstantial effect on the ground.

Two recent developments may hold promise for theBank’s results agenda, although they are in theirearly days. The first is the use of impact evaluations:the number supported by the Bank has morethan doubled, to 158 over the past year. Impactevaluations are not a panacea but can createbetter understanding of the causal links andfactors contributing to the outcomes of projects,programs, and policies. However, these evalua-tions are concentrated in a few areas (education,health, and conditional cash transfers) and needto be managed more strategically to draw moreknowledge from them.

The second development is a new approach towardmeasuring and reporting on development results forthe International Development Association (IDA)—the Bank’s main source of concessional finance. Theresults management system for IDA—initiated inthe 14th replenishment of IDA, with commit-ments to enhance it in the 15th replenishment(IDA15)—tries, among other things, to spotlightchanges in indicators, including access to waterand measures of child health. It is premature to

E X E C UT I V E S U M M A RY

x v

assess how well this will work, but it is animportant step in corporate-level monitoringand evaluation. At the same time, there aredifficult questions as to whether and how a morecomprehensive results framework for the Bank,as a whole, could evolve. It continues to bedifficult to piece together the various M&Eindicators to form a view of the Bank’s overalldevelopment results.

There are two broad lessons for better tracking Bankperformance. First, practical steps are needed (a)at the project level, and in global and regionalprograms, to enhance the quality of the M&Esystems, especially by working to put in placegood baseline information and to elucidateclearly the link between project outputs andtargeted outcomes; (b) at the country level, tosimplify results frameworks and so make themmore useful in guiding and evaluating programs;and (c) at the institutional level, for the Bank andin partner countries, to manage and learn from agrowing number of impact evaluations, includ-ing by better integrating them into countryprograms and exploiting cross-country synergiesin conducting and sharing studies. Second, theBank and IEG should strengthen the evaluationknowledge base for the Bank’s corporate results.Progress on these two fronts will improve theprospects for greater development impact in theyears ahead.

Part II: Shared Global Challenges

The Challenge of Global Public Goods

Tackling global climate change and providing otherimportant global public goods present some of thegreatest challenges of our time. Indeed, manyglobal public goods are chronically undersup-plied. Why? Because it is difficult to secure collec-tive action among nations to provide a publicgood—such as keeping air clean—particularlywhen the costs are borne locally while thebenefits are largely captured nationally orglobally. Yet there is a growing interconnectionbetween the different types of investments andactions needed at various levels to foster globalpublic goods.

The World Bank Group has emphasized the needto foster global public goods as one of its mainpriorities in the future. The effective provision ofglobal public goods increasingly influencesdevelopment results (discussed in Part I above),especially addressing the many dimensions ofpoverty, including vulnerability. The Bank’sstrategic framework for its role with regard toglobal public goods notes that the Bank canconnect global concerns to country programsand advocate for collective international action.How can the Bank enhance its effectiveness inthis area?

Can the Bank’s Country-Based Model FosterGlobal Public Goods?

Relying on the country-based model as the platformfor the Bank’s work on global public goods is adouble-edged sword. The model works well whennational partners see an alignment betweendomestic and global benefits, and when the Bankhas an attractive instrument to help implementaction at the country level. For example, theBank’s successful work in client countries, tohelp phase out ozone-depleting substances,benefited from the existence of the MontrealProtocol—a binding agreement that committedcountries to globally agreed action—and theMultilateral Fund, which provided resources forinvestments. Global Environment Facility (GEF)grants have also been well integrated into Bankcountry programs, such as in China, where alarge GEF portfolio has buttressed growingattention to environmental issues. And inVietnam, the Bank has been able to use itsmultisectoral expertise, combined with conces-sional finance, to help the authorities cope withthe threat of avian flu, in part because there wasstrong national interest in averting economicfallout in the domestic food industry.

The country-based model, however, comes understrain, especially when global and countryinterests are seen to diverge significantly and theBank’s traditional tools, including its lending, donot gain traction with clients. This makes itdoubly difficult to secure progress with globalpublic goods. Tackling climate change requires

x v i


huge adjustments in various economicbehaviors, including reducing emissions andimproving economywide energy efficiency anduse. For many countries, the benefits of suchactions seem remote while the costs accrue inthe near term. To date, though, the Bank has notbeen able to call on an attractive large-scalefunding program or invoke an internationalframework to encourage comprehensive actionon climate change. It will be important to seehow the recently discussed Climate InvestmentFunds help improve this situation.

The Bank pays attention to fostering global publicgoods in its high-level corporate strategies and thetopic has been emphasized by the president as oneof the Bank’s six strategic pillars. However,attention wanes as one moves down the levelsfrom corporate strategies to sectoral or regionalstrategies, and then down one more level tocountry strategies. Both the Bank’s GPGFramework and Long-Term Strategic Exercisediscussed global public goods extensively butlacked specifics on how to translate corporatepriorities into country action. The treatment instrategies at the next level down—the Bank’snetworks and Regional units—varies signifi-cantly. Attention to global public goods is moreprominent in both sectoral and Regional strate-gies dealing with the environment than in thosedealing with the health sector. This may be dueto the type of intervention needed in healthsector global public goods—such as communi-cable disease control, which requires a strongnational focus that might not be explicitlyconnected to global action.

The systems for integrating global public goods intocountry strategies are underdeveloped. Environ-mental commons is frequently noted in countrystrategies (in part because GEF projects aremainstreamed in the Bank’s systems), but otherglobal public goods are less often emphasized.There is no evidence that over time thetreatment of global public goods in Bank countrystrategies has expanded, but very recentexamples of good practice—such as in Brazil—may pave the way for more thorough and consis-tent strategic planning.

The Bank has at least three levers for moving fromstrategy to action at the country level—budget andtrust fund allocation, financing instruments, andglobal programs. Each is discussed in turn below.

RESOURCE ALLOCATION

The Bank estimates its administrative expenditureon global public goods at around $110 million infiscal 2007, nearly half of which is from sources thatare outside the Bank’s core budget, such as trustfunds. At about 4 percent of its overall operatingbudget, this is one of the smaller allocations for theBank’s six strategic priorities. These estimatesshould be treated with some caution becausethey may vary significantly, depending on thedefinitions and data classifications used. Goingforward, a more precise definition and trackingof spending on global public goods would be auseful management tool.

A heavy reliance on trust funds for financingglobal public goods work may itself increase thedifficulties of mainstreaming such activityalongside long-standing work financed by theBank’s own budget. Spending on global publicgoods, as a whole, has risen rapidly over the pastfive years, with the biggest increase for work onenvironmental commons.

FINANCING INSTRUMENTS

Concessional finance is important to foster manyglobal public goods, and in recent years, the Bankhas committed substantial IDA funding to helpcountries in programs with clear global publicgoods dimensions, such as HIV/AIDS andenvironmental commons. Often, country-levelimplementation capacity is stretched, however,and national priorities may take precedence oversome global public goods considerations. Staffreport that there is great reluctance amongnational partners and Bank country teams toallow IDA allocations targeted for povertyreduction to be diverted to fostering globalpublic goods, which may not immediatelybenefit the poorest populations. A recent innova-tion in IDA is a specific allocation for regional(multicountry) projects. Although it is too earlyto assess how well this is working, it should bemonitored for lessons in mirroring this approach


x v i i

for some global public goods; great care wouldbe needed to avoid fragmenting IDA’s overallframework.

When the Bank has had a clear and viable instrumentto help its country partners take action on someglobal public goods, there has been progress—theGEF is a good example. Where the Bank has nothad an obviously attractive financial instru-ment—and/or where there has been a lack ofdemand from country partners—it is less easy tosee progress. Measures to protect and conserveimportant forest resources around the world, forexample, have produced a highly varied picture.In Indonesia, an evaluation of the Bank’s countryassistance program from 1999 to 2006 showedthat it covered forestry issues with large-scaleanalytical work but little lending. Over thatperiod, the traction achieved by the Bank wasvery limited, and deforestation continued at arapid clip.

There is often a mismatch between country needs(and resources) and global ambitions for globalpublic goods. In middle-income countries, theBank’s ability to influence (or persuade) acountry to take concrete action on some globalpublic goods is inherently limited, even thougheffective provision of those goods requires deepparticipation by these middle-income countries.The limits of nonconcessional finance are clear,for example, in the Bank’s work on avianinfluenza, where only 7 of the 50 projectsapproved are financed by the International Bankfor Reconstruction and Development (IBRD),and, to date, only $12 million of the $94 millionin IBRD loans have been disbursed.

GLOBAL PROGRAMS

The Bank is now a partner in some 160 globalprograms and partnerships, and about 90 percent ofthe total spending of these global programs andpartnerships, which is overseen by the Bank, isdirected at global public goods. A few large initia-tives account for most of this spending: the GlobalFund to Fight Aids, Tuberculosis, and Malaria; theGEF; and the Consultative Group for InternationalAgricultural Research (CGIAR). The Bank’sadministrative effort in global programs and

partnerships is not fully driven by global publicgoods concerns, however, since more than 100 ofthese programs are focused largely on nationalpublic goods, such as urban development orregulation of the markets for infrastructure.

Despite the Bank’s direct role as a partner in globalprograms, systematic linkages to country programshave been lacking at times. For example, many ofthe programs had only modest participation bymiddle-income countries. Task managers forglobal programs have not commonly beenrequired to demonstrate how such programshave added value to country programs and Bankoperations, and often lack the incentive oradministrative budget to do so.

Merely locating a global program in the Bank—thereare 57 such programs—does not guarantee effectivecountry linkages. For example, linkages wereweak in the Population Reproductive HealthCapacity Building Program, despite the potentialsynergies with Bank investment operations invarious countries. IEG evaluations have alsofound that greater legitimacy of a global programdoes appear to foster stronger linkages withcountry operations.

In the Bank’s efforts to provide regional publicgoods—and to link regional and country concernsand opportunities—it faces challenges similar tothose for global public goods. Regional programshave risen in importance in recent years, buttheir integration into country programs remainsthe exception rather than the rule, and they stillaccount for a modest share of Bank lending.

The Bank’s Advocacy on Global Public Goods:What Has Worked and What Has Not

Successful advocacy goes beyond encouragingaction at the country level. It also involves producingcollective global responses and promoting thedevelopment interests of the poor in internationalagreements and frameworks for action.

Promoting improvements in the global tradingframework is an example of the Bank’s advocacy atits best. Key ingredients included a long period of

x v i i i


working directly with partner countries, theassembly of first-rate intellectual and analyticalresearch capacity, proactive and highly visibledissemination, and the willingness to engage inpublic debate. These were combined to excellenteffect, and the Bank’s work also had an opportu-nity to gain traction in the context of “live”negotiations for the Doha round of a newinternational trade agreement.

The experience with avian flu also illustrates theBank’s strengths as an advocate and convener. TheBank’s contributions to a global response wasbuilt on robust economic analysis, conveningpower, fiduciary reputation, and multisectoralexpertise. It also helped that the ground wasfertile for the Bank’s advocacy, given that globaland national concerns aligned as country needswere urgently felt.

Advocacy on environmental commons has proved amore complex challenge. The Bank has played apositive advocacy role in some very practicalsettings, including the securing of resources forthe GEF, the launch of the Prototype CarbonFund (and subsequent carbon funds), andmethodologies to put the Clean DevelopmentMechanism into action. The extent to which theBank has been a leading influential advocate onclimate change is more debatable, but there isnow a platform on which to build futureadvocacy work, including the Bank’s new Strate-gic Framework for Climate Change.

Advocacy through global programs has become anincreasingly important channel for fostering globalpublic goods. Giving proper voice and representa-tion to developing countries in such programsimproves their responsiveness and long-termsustainability. Yet, developing country voicesremain underrepresented—not least in thegovernance of many global programs—andwhether the Bank could have pushed harder onthis issue remains a question. It is encouragingthat governance arrangements in severalprograms, including the GEF and CGIAR, haveimproved over time. For large new globalprograms aimed at climate change, it is critical toensure sound and equitable governance arrange-

ments that balance the interests of the keyparties involved.

Improving the Bank’s Support for Global PublicGoods: Lessons from Experience

The Bank’s country model has its place in fosteringglobal public goods. It has worked well whennational and global interests coincide—oftenwith an agreed international framework foraction, such as the Montreal Protocol—andwhen grant finance supports country-basedinvestments.

Looking ahead, some of the great shared globalchallenges arise where national and global benefitsdiverge significantly—most notably on climateprotection. In tackling these challenges, theBank—including through cooperation with theInternational Finance Corporation and Multilat-eral Investment Guarantee Agency—needs tofind a way to bridge the gap more effectivelybetween global needs and country preferences.Lessons from this review suggest some measuresin five areas that may help the Bank upgrade itsability to foster global public goods.

First, the Bank can create better incentives to deliverglobal public goods effectively at the country level.This would include new approaches to settingbudgets and recognizing the performance ofmanagers and staff. On budget setting, oneoption is to set aside, at the corporate level,significant administrative funding to be allocatedto country teams—transparently and possiblycompetitively—for high-priority global publicgoods work at the country level. Care would beneeded to make sure such funding was used as agenuine addition by teams and not simply todisplace other activity. To provide betterincentives to staff, managers at all levels need toconsider recognizing country- and global-levelwork on global public goods in performancemanagement systems.

Second, the Bank can consider clearer organiza-tional arrangements to best select, and indeed linktogether, responses at country, regional, and globallevels. Some Regions may want to have dedicated


x i x

staff advancing work on regional programs (andregional public goods), as has been done inAfrica, and perhaps expand their purview tocover global public goods as well. But this is nota one-size-fits-all prescription, and other Regionsmay have different arrangements suitable to theircircumstances.

Third, a more effective approach to the delivery ofthe Bank’s global knowledge and capacity to countryteams working on global public goods would bebeneficial. To this end, the way the Bank can bestdeploy its expertise, particularly that of itsspecialists located at the center of the institutionin the network anchors, should be reviewed.

Fourth, the Bank and its stakeholders could renewattention to ensuring that the perspective of develop-ing countries is connected effectively with globalresponses. The Bank might be able to use itsstanding more powerfully to give greater voice todeveloping countries in the governance of signif-icant global programs. It should take a moreproactive stance in advocating for developmentinterests—and developing country partners—ininternational forums (and agreements) dealing

with global public goods. That would include theBank’s continuing to secure additional develop-ment assistance and promote the design and useof market-based instruments to help developingcountries provide global public goods. The Bankcould also explore further ways to stimulateSouth-South exchange of knowledge—and thedevelopment and application of new technolo-gies designed with and for the South—tocontribute to global public goods, such asclimate-friendly energy production and use.

Finally, a firmer and more precise justification isneeded for the costs and benefits of actions beingproposed for the Bank’s work on fostering globalpublic goods, to ensure that such work isfinancially and institutionally sustainable over thelong term. Particularly for global programs, theBank must redouble its efforts to be moreselective in its engagement and more forthrightin its exiting programs whose benefits and cost-effectiveness are questionable. The Bank shouldalso be insistent about putting in place, andusing, sound results frameworks, underpinnedby realistic and cost-effective monitoring andevaluation systems.

x x


x x i

Management Comments: Summary

The 2008 Annual Review of Development Effectiveness (ARDE) tracksBank performance and examines a particular thematic topic, the Bank’swork in fostering global public goods (GPGs). Management values the

review of project and program outcomes and of monitoring and evaluation(M&E) practice, and we appreciate in particular the clarity of analysis, whichhelps the Bank learn from experience.

We very much welcome also the review ofcountry program support for GPGs and of theBank’s advocacy work on GPGs, particularly therecommendations on bridging the gap betweenglobal needs and country preferences. Manage-ment’s complete comments are included as anappendix; this note summarizes the main points,with a focus on recent actions.

Tracking Bank Performance

Project Outcomes. The ARDE confirms thatproject outcomes have significantly improvedover the medium term, exceeding the Bank’sperformance benchmarks in fiscal 2004–06.Management notes with particular satisfactionimpressive improvements in the projects of theAfrica Region and of the water supply and sanita-tion sector. Management shares IEG’s concernsabout the dip in project outcomes in fiscal 2007,notably in noninfrastructure sectors and low- income countries, and appreciates the need tobe vigilant so the dip does not become a trend.Overoptimism in self- ratings of operations closeto closure parallels findings in our more detailedreview of International Development Association(IDA) controls, reports by the Bank’s internalQuality Assurance Group (QAG), and the IndiaDetailed Implementation Review. In response,

Bank Regions are reviewing their portfolios andworking actively with staff on measures toimprove rating practices and strengthen ongoingoperations. A more fundamental issue, however,is that the traditional supervision model, whichwas geared to infrastructure projects in middle- income countries, needs to be adapted to thecircumstances of projects in softer sectors and infragile states, which call for more Bank engage-ment in project implementation, better riskreporting, and customized implementationsupport directed at capacity building. Manage-ment is addressing this issue in the context ofinvestment lending reform.

Country Program Outcomes. The ARDE reports thatoutcome ratings, averaged over a long period, areconsiderably lower for Bank- supported programsin low- income countries (LICs) than for programsin middle- income countries (MICs), and arelower for programs than for projects. Manage-ment is concerned about the gap between MICand LIC programs and would very much welcomeIEG’s deeper analysis of the underlying factors— notably an analysis of changes in outcomes overtime for subgroups of countries, taking intoaccount such developments as the introductionof Country Assistance Strategy (CAS) resultsframeworks and changes in evaluation methodol-

ogy. Management believes that the gap betweenprogram and project outcomes primarily reflectsa difference in performance standards, not afailure of programs to exploit synergies. The Bankonly recently introduced results- based CASs thatclearly distinguish between the country’sobjectives and CAS outcomes based on thoseobjectives, and include results chains setting outhow the Bank will contribute to those outcomes.Most of the programs evaluated by IEG are onbased on CASs in which this distinction was muchless clear. The upcoming CAS Retrospective willprovide a further opportunity for an informeddiscussion of program outcomes.

Monitoring and Evaluation. Management agreeswith the ARDE’s finding that the implementationof the Bank’s strong policy framework for M&Econtinues to face challenges in projects andprograms. Management appreciates and is actingon IEG’s recommendations to focus on theprovision of good baseline information, articu-late more clearly the link between projectoutputs and targeted outcomes, simplify CASresults frameworks, and use M&E moreeffectively for program management. To do thiseffectively will require not only Bank action butmeasures to address the issue of statisticalcapacity in member countries. As announcedduring the High-Level Forum on Aid Effective-ness in Accra, our joint efforts with the Nether-lands and the United Kingdom have succeededin establishing a new Statistics for Results Facilitythat will help countries implement their nationalplans for improving statistical systems.

Shared Global Challenges— Lessons from the Bank’s Experience

Bank Support at the Country Level. The ARDEobserves that GPGs other than environmentalcommons are not sufficiently emphasized inCASs and that the extent of Bank involvement inGPG issues varies widely among countries.

Management agrees that country- based supportfor GPGs will need to increase and be betterintegrated into CASs. We also believe that theappropriate role for the Bank is specific to theparticular GPG being supported and to thecountry context; hence variations in the extentof Bank involvement among GPGs and amongcountries are to be expected. Since the Bank isonly one player among many, our framework forsupport to GPGs calls for identifying where gapsare not being met by other agencies and thenhelping to fill the gaps where the Bank has thecapability and comparative advantage. Countryownership and response to client demandremain the primary principles of country- basedsupport for GPGs.

The Country Program Model. The ARDE highlightsthe challenges to country- based support for GPGsin cases where global and country interestsdiverge and there is no international frameworkfor collective action. Management believes thatsuch challenges can be addressed withoutearmarked funding at the corporate level forcountry work. We do not agree with the broadconclusion that relying on the country programmodel is a ”double- edged sword.” The Bank’sultimate clients are poor people. By using thecountry program model, the Bank is better able toprovide analysis that puts growth and povertyreduction at the core and relates GPG challengesto this goal, and to ensure that the actions itsupports are owned by the country. In Manage-ment’s view, the key challenge— discussed in theupcoming CAS Retrospective— is to morethoroughly integrate GPGs into the CAS diagnosisof country development challenges and thedialogue with the government, and on that basisdetermine the appropriate contribution of globalprograms and trust funds as part of the CASsupport program. We also agree that mobilizingconcessional funding is important for engage-ment on GPGs in MICs, and we note that efforts tothat effect are under way.

x x i i


x x i i i

2008 ARDE Part I of the report tracked the Bank’s perform-ance (including trends in outcomes of projectsand country programs), the evolution of monitor-ing and evaluation (M&E), and the role of evalua-tion in the results agenda. IEG found that overalldevelopment outcomes of Bank lending haveimproved over the medium term, albeit not overthe past year. Noting the growing “disconnect”between project supervision ratings and finalproject outcome ratings (which could point toweak incentives for accurate project reporting),IEG cautioned against overoptimism in assessingongoing project performance, and called forvigilance to ensure that the drop in projectperformance in fiscal 2007 does not foreshadow apersistent decline. IEG also noted opportunitiesto strengthen M& E.

Part II focused on a special theme, which this yearwas the Bank’s work in fostering global publicgoods (GPGs). IEG found that the country- basedmodel worked well when national and globalinterests coincided and when grant financesupports countrylevel investment, but thegreatest challenges arise when actual or perceivedlocal, national, and global benefits diverge signifi-cantly. IEG drew lessons for the Bank’s considera-tion: (i) strengthen incentives to deliver GPGs at

the country level; (ii) consider clearer organiza-tional arrangements to best select and linkresponses at country, regional, and global levels;(iii) enhance the delivery of global knowledge andcapacity to country teams working on GPGs; (iv)ensure that the perspectives of developingcountries are effectively connected with globalresponses; and (v) improve the justifications forthe costs and benefits of actions being proposedto foster GPGs.

Draft Management Response Management welcomed the insightful reviewand agreed that vigilance is warranted withrespect to the weakening developmentoutcomes of exiting projects and the increasing“disconnect” between the Bank’s self- ratings ofproject performance and IEG’s final ratings ofdevelopment outcomes in fiscal 2007. In thisregard, it noted that a deeper analysis ofcontributing factors, especially of differencesamong regions and sectors, would have beenuseful. Management commented on the need toaddress the different outcomes of middle- income country (MIC) and low- income country(LIC) programs, and the issue of lower outcomeratings for country programs than for projects. Ithad some different views about the obstacles tobetter M&E. Management also offered its

Chairperson’s Summary: Committee on Development

Effectiveness (CODE)

On July 23, 2008, the Committee on Development Effectiveness metto consider the Annual Review of Development Effectiveness 2008:Shared Global Challenges (ARDE), prepared by the Independent

Evaluation Group (IEG), and Draft Management Comments.

perspectives on efforts to integrate GPGs at thesector, country, and regional levels; its role inmobilizing and providing innovative financing;support for different types of global programsand partnerships; the proposal to set asideadministrative funds to support high- priorityGPGs at the country level; and the Bank’s role inadvocating the climate change agenda.

Overall Conclusions The Committee heard that management foundthe structure and analytical context of the reportto be of high quality and was broadly comfort-able with the findings, with one exceptionregarding the recommendation to set asideearmarked country program budget for GPGs.This issue and, more broadly, the matter of theappropriate business model and incentives forintegrating global and more traditional, localdevelopment interests elicited comments frommost speakers, who expressed a diversity ofviews and also made suggestions. Another areathat resonated with most participants was the“disconnect” between staff self- ratings and IEG’sfinal ratings after project completion, which washigher in fiscal 2007 than in recent years.

The Committee appreciated management’sfocus on: (i) the differentials between countrygroupings or sectors with the aim of strengthen-ing Bank business in dealing with the challengesof low- capacity countries and softer sectors; (ii)the findings related to low- income countryprograms, especially since IEG did not rate theoverall development outcome of a singleprogram beyond moderately satisfactory duringthe evaluation period; (iii) the distinctionbetween project complexity and developmentrisks related to challenging country circum-stances; (iv) the emphasis on M&E not only toensure data availability for assessing results (andavoiding complacency) but also to obtain aclearer perspective on the complementaritiesand tradeoffs between public goods and moretraditional economic growth and developmentconcerns; and (v) the new structure of the ARDE.

With respect to Part II on GPGs, in addition toways of striking the appropriate balances in

country programs and operations, the discussionalso drew attention to several findings ofcorporate relevance, namely the advocacy rolewhere the Bank must push harder to strengthenthe developing countries’ perspectives andvoices, which are presently underrepresented.To this end, it was noted that GPGs are relativelynew in the development agenda, and the Bank isat the beginning of its learning curve in assistingcountries to address GPG issues. Therefore, theCommittee should keep an open mind about thechallenges of GPGs and avoid reachingpremature conclusions.

Next Steps The Board is scheduled to consider the reporton September 9, 2008. IEG will provide asummary of actions to which managementcommitted, in response to IEG recommenda-tions and where IEG has seen limited progressafter three years. For future ARDE reports, IEGwas asked to consider a closer link between PartsI and II. Management noted that M&E and resultsmanagement in country programs will bediscussed under the upcoming CountryAssistance Strategy (CAS) Retrospective Report.

Main issues raised at the meeting were the following:

Project Performance. Members agreed thatvigilance is needed to identify problem projectsand risks in real time, an issue previously raisedby the Quality Assurance Group (QAG). Theytook note of management’s recognition that thecurrent supervision for large infrastructureprojects does not differentiate betweencountries (low- income vs. middle- incomecountries and fragile situations) and betweenhard (infrastructure) and softer sectors (humandevelopment or public sector management).Some members acknowledged the higher fiscal2007 “disconnect” between the Bank’s self- ratingof project performance and IEG’s final ratings ofdevelopment outcomes. They sought furthercomments on the cause of this “disconnect.”One member suggested that the Bank and IEGshould find a mutually agreed scheme; IEGnoted that the rating systems were comparable.

x x i v


This member also recommended that the CASProgress Report should address the issue ofquality of supervision.

Issues were raised relating to project complexityand development risks under challengingcountry circumstances. A point was made thatproject complexity should be commensuratewith implementation capacity. Speakerscautioned against messages or incentives thatmay induce staff to avoid risks necessary toachieve development impact. One membernoted that (i) for measuring complex projects,there was a question about how to integrate arisk premium into the rating system, to ensurequality but without undermining the risk- takingincentives to staff; (ii) innovative approacheswere needed to address development issues indifficult countries, and (iii) the challenge was indesigning interventions adapted to differentcategories of countries (for example, LICs, MICs,or fragile states).

Country Programs. One member noted, withconcern, IEG’s finding that not a single programconcentrated in smaller countries or countrieswith extensive poverty has been rated “satisfac-tory” in meeting their stated developmentobjectives. Given the long engagement of theBank with LICs, the question arose aboutwhether there are some systemic issues andwhether these have been suitably identified.

M&E and Results Management. Members suggestedthat the use of impact evaluation, though still atan early stage, could improve understanding ofthe outcomes of projects, programs, and policies.The importance of disseminating the methodolo-gies of impact evaluation was noted. Onemember observed that the quality of M&E wasassociated with the quality of outcomes, andexpressed concern that while measuring interme-diate and final outputs require appropriate datagathering, these activities were not built intoproject design. While acknowledging that resultson the ground and success may be hard to attrib-ute to interventions by the Bank, he questionedhow it can be a knowledge bank if it cannot learnfrom its successes and failures. This member

further suggested that Implementation Comple-tion Reports should include a formal require-ment to rate a project’s M&E. Questions wereraised more generally on the quality of M&E,simplification of country results framework, andknowledge base of corporate results.

Challenge of GPGs. Members generally agreed thatGPGs were a relatively new topic in the develop-ment agenda, and the Bank needed to gainexperience about what may work well, includingenhancement of country ownership and the demand- driven approach, as well as strength-ened partnerships. They also stressed the needto identify the Bank’s comparative advantage inthe GPG work, given that the Bank was only oneof many players in this field. One member wasdisappointed about the perception that“networks are not working.” Some membersencouraged the identification of sectoralcomparative advantages beyond health andclimate change, such as market stabilization ofkey commodities like oil and grains, and thedevelopment of innovative financing. If GPGs arein conflict with a country’s interest, there is aquestion of ownership, incentives, and percep-tion of tradeoffs between GPGs and develop-ment. In this sense, one member felt the Bankwas focusing on “rounding a square.” Thismember also felt that it was difficult tooperationalize IEG’s recommendations toimprove the Bank’s support for GPGs.

GPG Country- Based Model. Members felt that the country- based model was a clear comparativeadvantage of the Bank’s work on GPGs.However, they regretted that systems forintegrating GPGs into the country- based modelwere underdeveloped when country and globalinterests diverge significantly. In this context,some members underscored the importance ofaddressing the limitations of the currentbusiness model to promote GPGs, whileavoiding imposing conditionalities and supply- driven approaches. There was a sentiment thatthe country- based model was not enough toaddress GPGs and should be complementedwith a global- level framework, which will provideincentives to countries, including financial

C H A I R P E R S O N ’ S S U M M A RY: C O M M I T T E E O N D E V E L O P M E N T E F E C T I V E N E S S ( C O D E )

x x v

mechanisms for incremental costs. In addition, aquestion was raised about whether the issue wasthe effectiveness of Bank instruments ratherthan the country model.

Resource Allocation for GPGs. There werecomments on the need for a new approach inallocating and tracking spending on GPGs (bothbudget and trust funds), and in recognizingperformance, but there were a variety of views asto how this should be done. One member sharedmanagement’s reservation about IEG’s suggestedoption of setting aside significant administrativefunding for allocation of high- priority GPG workat the country level.

ARDE Format. The new structure of the reportgenerally found favor with the Committee.

Speakers welcomed the new format but wouldhave liked greater substantive integration of thetwo parts. A more specific suggestion was todeepen the analysis of several key points in Part I,to make the report a more robust instrument ofmanagement’s accountability. One memberfound that the aggregate view of the Bank’sdevelopment effectiveness did not give an indica-tion of how various areas of the Bank perform,what lessons could be drawn from the report, andthus how resources should be allocated. A fewmembers sought clarification on why the reportwas not accompanied by the Management ActionRecord for monitoring progress of the Bank’sagreed actions. IEG provided additional informa-tion regarding the implementation report thatsets out which older IEG recommendations willbe retired from formal consideration.

x x v i


Jiayi Zou, Chairperson

x x v i i

Evaluation Snapshot in Selected Languages

The key findings and recommendations of the Annual Review of Development Effectiveness 2008: Shared Global Challenges are pre-sented in snapshot below. Translations of the report’s full summary into

each of the languages shown here are available at www.worldbank.org/ieg, andhard copies are available from IEG and World Bank Public Information Centers.

Arabic Chinese

x x v i i i


• Reducing poverty in any individual country isincreasingly intertwined with making progresson shared global challenges— that is, fosteringglobal public goods (GPGs) such as climateprotection and communicable disease con-trol. This year’s ARDE tracks World Bank per-formance in Part I and examines the Bank’swork in fostering GPGs in Part II.

• Development outcomes from Bank lendinghave improved over the medium term. But infiscal 2007 overoptimism in the Bank’s ongo-ing assessment of project performance rosesharply, while the share of projects rated mod-erately satisfactory or better dropped to 76percent from 83 percent a year earlier.

• Vigilance is needed to identify problem proj-ects in real time and ensure that the fiscal2007 drop in performance does not fore-shadow a persistent decline. Practical stepscan be taken to better use M&E in projects andprograms, including proper baseline infor-mation and clearer links between outputs and outcomes.

• The Bank’s country- based model has workedrelatively well in fostering global public goodswhen national and global interests dovetailand grants support country investments. Butthe greatest challenges, such as climate change,arise where local, national, and global benefits— actual or perceived— diverge sig-nificantly. Here the country model comesunder considerable strain.

• To more effectively bridge the gap betweenglobal needs and country concerns, the Bankshould consider: creating dedicated budgetsand better incentives for country teams towork on GPGs; deploying its global knowl-edge networks better; and using its standingmore powerfully to give greater voice to de-veloping countries in the governance of global programs.

English

Dans chaque pays, la réduction de la pauvreté estde plus en plus étroitement liée aux progrès réa-lisés face aux grands problèmes communs à l’en-semble de la planète – autrement dit, à lapromotion des biens publics mondiaux, tels quela protection du climat et la lutte contre les ma-ladies communicables. Cette année, l’Examen an-nuel de l’efficacité du développement consacresa Première partie à suivre les résultats obtenuspar la Banque mondiale dans ce domaine et saDeuxième partie à l’action menée par la Banquepour promouvoir les biens publics mondiaux.Un bilan des résultats obtenus par les prêts dela Banque laisse apparaître une amélioration surle moyen terme. Toutefois, au cours de l’exer-cice 2007, l’excès d’optimisme affiché par laBanque dans son évaluation de la performancede ses projets a grimpé fortement alors mêmeque le pourcentage de ses projets classés commemodérément satisfaisants ou mieux est tombé à76% contre 83% l’année précédente.Il faut faire preuve de vigilance si l’on veut iden-tifier les projets à problème en temps réel etveiller à ce que la baisse de résultats de 2007 nesoit pas annonciatrice d’un déclin persistant. Onpeut prendre des mesures pratiques afin demieux utiliser le Suivi et l’Évaluation dans lesprojets et programmes, notamment une bonneinformation de base et des liens clairement éta-blis entre les produits et les résultats.Le modèle par pays de la Banque contribue assezefficacement à promouvoir les biens publicsmondiaux lorsque les intérêts nationaux concor-dent avec les intérêts mondiaux et que des donssoutiennent les investissements du pays. En re-vanche, les problèmes les plus graves, tels quele changement climatique, se posent lorsque lesintérêts nationaux et mondiaux – réels ou per-çus – divergent sensiblement. En pareil cas, lemodèle par pays est mis à rude épreuve.our mieux combler le fossé entre les besoins

mondiaux et les préoccupations nationales, laBanque devrait envisager : d’établir des bud-gets spécifiques et de meilleures incitations pourles équipes-pays à travailler à la réalisation desbiens publics mondiaux ; de mieux utiliser sesréseaux mondiaux de savoir ; et d’utiliser plusénergiquement sa position afin de donner plusde voix aux pays en développement dans la ges-tion des programmes mondiaux.

French Français

E VA L U AT I O N S N A P S H OT I N S E L E C T E D L A N G U AG E S

x x i x

redução da pobreza em qualquer país estácada vez mais vinculada ao progresso em en-frentar desafios globais compartilhados – ouseja, promover os bens públicos globais (GPGs),tais como proteção climática e controle de do-enças transmissíveis. Revisão da Eficá-cia do Desenvolvimento deste anoanalisa o desempenho do Banco Mundial naParte I e o trabalho do mesmo na promoção dosGPGs na Parte II.Os resultados do desenvolvimento oriundos daconcessão de empréstimos pelo Banco Mundialmelhoraram no médio prazo. Mas no exercíciofinanceiro de 2007 o superotimismo da ava-liação contínua do Banco Mundial no tocanteao desempenho dos projetos aumentou con-sideravelmente, ao passo que a parcela de pro-jetos classificados como moderadamentesatisfatórios ou melhores caiu para 76% com re-lação a 83% do ano anterior.

preciso vigilância para identificar projetosproblemáticos em tempo real e assegurar quea queda de desempenho ocorrida no exercíciofinanceiro de 2007 não seja um prenúncio deum declínio persistente. Podem-se tomar me-didas para melhorar o uso de Monitoramentoe valiação em projetos e programas, in-clusive informação básica adequada e vínculosmais claros entre produtos e resultados.

modelo baseado no país, utilizado pelo BancoMundial, tem funcionado relativamente bem napromoção dos bens públicos globais em umaépoca em que os interesses nacionais e globaisse concatenam e subvenções apóiam investi-mentos dos países. No entanto, os maiores de-safios, tais como a mudança climática, surgemquando os benefícios locais, nacionais e globais– reais ou percebidos – divergem de forma sig-nificativa. Neste aspecto o modelo de país sofrepressão considerável.ara cobrir com mais eficácia o hiato entre as

necessidades globais e as preocupações dos pa-íses, o Banco Mundial deve considerar mu-danças tais como: Criar orçamentos dedicadose maiores incentivos para as equipes de país tra-balharem na promoção dos GPGs; melhorar aimplantação de suas redes globais de conhe-cimentos; e utilizar sua posição de forma maisincisiva para dar maior expressão aos países emdesenvolvimento na governabilidade de pro-gramas globais.

Portuguese PortuguêsLa reducción de la pobreza en un país deter-minado está cada vez más relacionada con losprogresos que se hagan con respecto a los de-safíos mundiales comunes, es decir, la promo-ción de los bienes públicos mundiales (BPM),como la protección del clima y la lucha contralas enfermedades transmisibles. En la Parte I dela versión del ARDE de este año se hace un se-guimiento del desempeño del Banco Mundialy en la Parte II se pasa revista a su labor de pro-moción de los BPM. os resultados en términos de desarrollo de las

operaciones de financiamiento del Banco hanmejorado en el mediano plazo. Sin embargo, enel ejercicio de 2007, el exceso de optimismo delas autoevaluaciones en curso con respecto al de-sempeño de los proyectos aumentó marcada-mente, mientras que la proporción de proyectosque recibieron una calificación de moderada-mente satisfactorios o superior bajó al 76%, encomparación con el 83% en el ejercicio anterior.Se debe prestar atención para detectar de in-mediato los proyectos que presentan problemasy velar por que la baja registrada en el ejerciciode 2007 no sea un presagio de un deterioro per-sistente. Se pueden tomar medidas prácticaspara utilizar de mejor manera los sistemas de se-guimiento y evaluación en los proyectos y pro-gramas, como recopilar información básicaadecuada y definir más claramente los vínculosentre los productos y los efectos directos. El modelo del Banco centrado en los países hafuncionado relativamente bien para promoverlos BPM cuando se han conjugado los interesesnacionales y de alcance mundial y las inversio-nes en los países se han respaldado con dona-ciones. Sin embargo, los mayores desafíos seplantean cuando los beneficios locales, nacio-nales y mundiales —-ya sean reales o percibi-dos— difieren considerablemente. En esos casos,el modelo se debilita de manera significativa.ara salvar de manera más eficaz la brecha entre

las necesidades de alcance mundial y los temasque interesan a los países, el Banco debería con-templar la posibilidad de establecer presupuestosespeciales y ofrecer mejores incentivos para quelos equipos a cargo de las operaciones del Bancoen los países se dediquen a los PBM; utilizar susredes de conocimientos mundiales de maneramás eficaz, y aprovechar de manera más enér-gica su posición para dar mayor voz a los paísesen desarrollo en la gestión de los programasmundiales.

Spanish Español

x x x


Russian

Chapter 1

Evaluation Highlights• Development success in any indi-

vidual country is increasingly inter-twined with making progress onshared global challenges.

• The Annual Review of DevelopmentEffectiveness is in a new, two- part format.

• Part I tracks World Bank perform-ance by analyzing projects, countryprograms, and results from moni-toring and evaluation.

• Part II examines a special topic: theBank’s work in fostering global pub-lic goods.

Chinese lanterns at night; photo ©Frank Krahmer/zefa/Corbis, reproduced by permission.

3

Introduction

For the World Bank and its partners, the ever- present test is to deliver results— to lift people out of poverty and promote socially and environ-mentally sustainable development. Achieving such success in any individual

country is increasingly intertwined with making progress on shared global chal-lenges. A fair and efficient international trade regime, for example, is a globalpublic good (GPG) that allows developing countries to trade more and growfaster. The growing global threat of climate change— a “public bad” by contrast— particularly imperils the poor who bear the brunt of more frequent natural dis-asters and hazards to health and agriculture (IEG 2006b; Stern 2006).

This year’s Annual Review of DevelopmentEffectiveness (ARDE) is in a new format andpresents evidence on the Bank’s efforts in twoimportant and connected areas. Part I, which is astandard section of the new format, analyzes thetrends in outcomes of Bank projects and countryprograms in recent years. Equally important— and building on past practice1—the report goesbeyond performance indicators and digs deeperinto the framework employed by the Bank tomonitor and evaluate its work. This ARDEassembles the available evidence on howmonitoring and evaluation (M&E) systems areprogressing, both in terms of quality andcoverage. It illustrates how new techniques arebeginning to be employed, including impactevaluations and new results management systemsfor the International Development Association(IDA). The review also refers to evidence tobenchmark the Independent Evaluation Group’s(IEG’s) own contribution to influencing theBank’s performance.

All of this is important because a successful M&Esystem that focuses on results can have threemajor benefits. It can provide an early warningwhen an intervention is not working well, and abasis for improving a project during implementa-tion. It can provide a broad measure of results foraccountability and demonstrate to stakeholdersthe degree to which the Bank has contributed todevelopment results. Finally, it can contribute toglobal knowledge for development, providinguseful lessons for other projects and programswithin the same country, or within the samesector in other countries.

Part II of the report examines a special topic.There has been enormous growth in interna-tional attention to the great shared globalchallenges of our time— a set of supranationalissues collectively known as global public goods.This year’s ARDE looks at the Bank’s work infostering GPGs, such as protecting the earth’sclimate and preventing the spread of dangerous

4


communicable diseases. GPGs are clearlyimportant in their own right and are becomingever more closely associated with the Bank’sdevelopment impact (as described in Part I). Tofight the many dimensions of poverty— including vulnerability— the world must be ablenot only to help antipoverty programs at theindividual country level, but also to create theframework for collective action on transnationalissues that can support or undermine the fate ofthe poor.

As the Bank has emphasized inclusive and sustain-able globalization, the importance of coordinatedglobal solutions to cross- border problems hasrisen in tandem. Indeed, the Bank has identified

fostering GPGs as one of its six strategic themes.This report focuses on two roles the Bank hasidentified as its comparative advantage: (i) identi-fying how the Bank’s country- based model hashelped or hindered its support for GPGs, and (ii)drawing lessons from the Bank’s experience as anadvocate for action on GPGs.

The thematic focus of ARDE 2008, taken togetherwith the components of Part I of the report,helps inform important choices for the Bank’sstrategy and its implementation. It also sets aframework for work by IEG in other evaluations— including next year’s ARDE— tofocus attention on development impact in the future.

PART I

Tracking Bank Performance

Chapter 2

Evaluation Highlights• Project outcome ratings improved

over the medium term.• Attention is needed to address the

growing lack of realism in assess-ments of ongoing project perform-ance.

• Unsatisfactory projects in fiscal 2007were hampered by overly ambitiousobjectives, complexity, and inade-quate design.

• Many country programs producedsatisfactory outcomes, especiallythose in large countries where manyof the world’s poor live.

• Overall, however, some 40 percent ofcountry programs have been mod-erately unsatisfactory or worse inoutcomes.

Tuareg child turns water pump wheel; photo ©Yann Arthus-Bertrand/Corbis, reproduced by permission.

9

Development Outcomes: Indicators of Performance

Over the past five years, the Bank’s lending has risen in real terms, andIDA made record commitments of $12 billion in fiscal 2007, alongsidesome $13 billion of International Bank for Reconstruction and De-

velopment (IBRD) finance provided to middle-income countries (MICs). In-deed, with the successful 15th replenishment round for IDA resources ($42billion), the Bank plans to maintain total annual lending in the $22 to $26 bil-lion range during fiscal 2008–10. Lending volumes by themselves, of course, arenot an indicator of development impact, andthere have been many cautions for the Bank (andother development agencies) to take care toavoid a “lending culture” which places undueemphasis on volumes delivered rather thanoutcomes secured.1 There are indicators ofperformance that can tell us about the Bank’sdevelopment impact—notably IEG evaluationsof Bank-supported projects and countryprograms. Overall trends and recent develop-ments revealed in both of these metrics arepresented below.

Measuring Project Performance: Trendsfrom IEG MonitoringHow are Bank-supported projects performing interms of achieving their development objectives?The performance of Bank projects in deliveringdevelopment results has unquestionably im -proved over the medium term.2 In the three yearsto end-fiscal 2007, 80 percent of projects weremoderately satisfactory or better in deliveringtheir targeted results, up from around 70 percent

at the start of the decade, as shown infigure 2.1.

Lending outcomes haveimproved over themedium term.

Figure 2.1: Project Performance Has Improved overthe Medium Term

Source: World Bank database.

60

70

80

90

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

Exit fiscal year (last in a 3-year rolling average)

Shar

e of

pro

ject

s w

ith o

utco

mes

mod

erat

ely

satis

fact

ory

or b

ette

r (%

)

By number of projects Weighted by value of project disbursements

1 0


A comparison of five-year subperiodsshows 78 percent of projects fromfiscal 2003 to 2007 rated as moderately

satisfactory or better in achieving their develop-ment results. This is a significant improvementfrom fiscal 1998–2002, in which 73 percent ofprojects had outcomes rated moderately satisfac-tory or better. Illustrations of developmentimpact from some of these projects are in box2.1. Moreover, a growing share of projects hasoutcomes that are sustainable.3 Almost 80percent of projects evaluated in fiscal 2003–07were rated as likely to be sustainable, up from 64percent in the previous five years.4 A moredetailed assessment of project performance isavailable in appendix A.

Against the backdrop of this overall improvementin outcomes, there are several features thatillustrate changes in the performance of Bank-supported projects, as shown in table 2.1. Therehas been a steep decline in the share of projectsrated unsatisfactory while, simultaneously, theshare reaching the satisfactory standard heldsteady. In addition, the Bank and its clients have

improved a number of projects, which inthe past may have been rated moderatelyunsatisfactory, to reach the level ofmoderately satisfactory. As a result, agreater proportion of projects is beingrated in the middle of the scale.

The Bank’s lending is spread across a spectrum ofsectors. It is encouraging that, as shown in figure2.2, project performance in about two-thirds ofsectors has improved over the medium term.5

The turnaround in the water supply and sanita-tion sector projects is especially dramatic: in fiscal1998–2002 only a little more than 60 percent ofprojects (by value of disbursements) weremoderately satisfactory or better, and this sectornow comes close to leading the way in fiscal2003–07, with over 90 percent of projects havingmoderately satisfactory or better outcomes.While there has been a creditable and significantuplift in economic policy project performance,the share of moderately satisfactory or betterproject outcomes in fiscal 2003–07 still trails othersectors by quite some margin. The largestdeclines in performance were in health, nutrition,and population (which, along with economicpolicy and the environment, were significantlybelow the Bank-wide average for the fiscal2003–07 period) and in public sector governance.

Looking at the Regional aspects of projectperformance, figure 2.3 shows that the East Asiaand Pacific and the Europe and Central AsiaRegions led the way for the fiscal 2003–07 cohort.Their share of projects with moderately satisfac-tory or better outcome ratings significantlyexceeded the Bank average of 83 percent(weighted by disbursement). A good illustration

There is no single metric that can be used across projects toassess satisfactory outcomes. Rather, development results spana range of different social and economic indicators dependingon the sector and type of project. A project is rated satisfactorywhen the operation’s objectives have been achieved with onlyminor shortcomings. This is illustrated with one type of devel-opment impact in Mozambique, where Bank-supported water andsanitation projects built 130 water source points, bringing drink-ing water to 62,000 people. And in Cambodia, a Bank-financed proj-ect brought clean water to 750,000 people in Phnom Penh.

In the energy sector, a Bank electrification project in the Lao

People’s Democratic Republic provided main-power-grid elec-tricity to 51,000 households in 721 rural villages, while another6,000 households gained access to off-grid electricity throughsolar and hydro systems. Having electricity greatly increased pro-ductivity of new small-scale, home-based businesses, and en-abled children to study at night.

Finally, another type of development impact is demonstrated inMali, where the Bank-supported Grassroots Initiatives to FightPoverty and Hunger Project helped more than 6,000 children to at-tend school, as well as created small-scale health centers in 19villages.

Box 2.1: What Does a Satisfactory Project Look Like? Illustrations of Development Impact

Outcomes of projects intwo-thirds of sectors

improved, while healthand public sector

governance worsened.

Project ratings moved upthe scale in recent years.

of such successful performance can be seen inan agricultural development project in China’sAnning Valley. The Bank’s $120 million of supportsupplied more reliable irrigation, introduced

new plant varieties and cultivation techniques,and added value to production through process-ing. Local farmers in the project’s area enjoyed atripling in their per capita income; the propor-

D E V E L O P M E N T O UT C O M E S : I N D I C ATO R S O F P E R F O R M A N C E

1 1

IEG rating of projects meeting Percentage of Projects in Rating Categorydevelopment objectives FY98–02 FY03–07 Percentage point change

Highly satisfactory 6% 4% –2

Satisfactory 45% 45% 0

Moderately satisfactory 21% 29% +8

Moderately unsatisfactory 9% 10% +1

Unsatisfactory 16% 10% –6

Highly unsatisfactory 2% 1% –1Source: World Bank database.

Table 2.1: Distribution of Project Ratings Moved Up the Scale in FY03–07

40% 50% 60% 70% 80% 90% 100%

IMPROVERS

40% 50% 60% 70% 80% 90% 100%

DECLINERS

Share of projects with outcomes moderately satisfactory or higher(weighted by disbursement)

Sect

or b

oard

FY98–02 FY03–07

Environment

Economic policy

Energy and mining

Bank-wide

Agriculture and rural development

Water supply and sanitation

Financial and private sector development

Transport

Social protection

Education

Urban development

Public sector governance

Health, nutrition, and population

Figure 2.2: Trends in Sectoral Performance

Source: World Bank database.Note: The Sector Board classification applies to the whole project and enables outcomes to be matched to it.

1 2


tion of the area’s families living in poverty fellfrom one-third to one-tenth; and the projectgenerated a 27 percent rate of return on itsinvestment.

In these Regions, MICs make up a large share of theclient base, and their relatively strong institutionalcapacity is conducive to stronger projectimplementation. Nonetheless, the Bank could wellbe faced with increasing demands from theseclients to take on projects that are complex orlocated within a country’s lagging or remotesubregions, under difficult operating conditions.These are challenges for the Bank to embrace, butexpectations for the success of projects must beadjusted to recognize concomitant risks.

In the Africa Region, the share ofprojects with moderately satisfactoryor better outcomes improved themost of any Region over fiscal2003–07. Again, it should be acknowl-edged that difficult operating

conditions can make it difficult to securesuccess—and that may at least partly explain whyproject performance, overall, in Africa continuesto lag behind all other Regions. But the Bank hasto find ways to deliver outcomes in adversity, as acentral part of its mission. Indeed, this can bedone, as illustrated in the Bank’s $220 millioninvestment to assist war-affected households in

northern Ethiopia. By removing landmines,building new homes, and equipping householdswith basic goods and agricultural inputs—includ-ing seeds and fertilizer—some 67,000 familiesreturned home, rebuilt their lives, and generatednew economic activity, which delivered a 50percent return on the project’s investment.

Project Outcomes in Fiscal 2007 DataProject development outcome ratings fluctuate,sometimes significantly, from year to year. Differ-ences are sometimes a result of the vicissitudesof a particular cohort of projects rather than asubstantive change in underlying institutional orpartner performance. So while a single year’sdata should not be taken in isolation, the mostrecent data are worth examining to see whetherany patterns or signals emerge. There are twomovements in fiscal 2007 that warrant theattention of Bank management.6

First, increasingly the Bank is too optimistic in itsown assessment of ongoing project performance.Of the 45 projects that exited in fiscal 2007 andwere rated as moderately unsatisfactory or worseby IEG, over two-thirds—32 projects in total—hadbeen reported by the Bank as moderately satisfac-tory or better just before they were closed. Thisfailure to identify problem projects early impairsreal-time managing for results, since overlyoptimistic ongoing ratings mean management is

Figure 2.3: Africa’s Projects Have Improved Substantially but Still Lag Behind Other Regions


40 50 60 70 80 90 100

Regi

onEast Asia and Pacific

Bank-wide

South Asia

Latin America and the Caribbean

Middle East and North Africa

Sub-Saharan Africa

Europe and Central Asia

Share of projects with outcomes moderately satisfactory or better (weighted by disbursements, %)

FY98–02 FY03–07

Africa’s performanceimproved the most of

any Region but still lags behind other parts

of the Bank.

less likely to take remedial action. Most Regionswere too optimistic in their ongoing reporting ofproject ratings for fiscal 2007, with almost a fifth ofall projects—and in Africa close to a third—downgraded during the process of self-evaluationand independent evaluation.7 Comparatively,these “disconnect” rates in previous years weremuch lower—less than 1 in 10 were downgradedin fiscal 2005 and 2006, as shown in table 2.2.Reducing this disconnect—by more carefully andaccurately identifying underperforming projectsduring implementation—is needed to focusattention on project supervision and to improveperformance. Of course, to subsequently improvedevelopment outcomes will then require project-specific actions on the ground.

Second, in fiscal 2007 there was a significant fallto 76 percent in the share of projects ratedmoderately satisfactory or better, from 83percent in fiscal 2006.8 This absolute level ofperformance still meets the 75 percent target setby the Bank a decade ago, although it is lowerthan the 80 percent benchmark noted by theBank’s own Quality Assurance Group (QAG) inrecent annual reviews. If a 75 percent thresholdfor project portfolio performance is consideredreasonable, given the risks of the “developmentbusiness,” then this one-year drop, by itself, isnot too worrisome. But certainly vigilance isneeded to ensure that this drop does notforeshadow a persistent decline.

What has caused the drop in ratings in fiscal 2007and produced 45 projects with outcomesmoderately unsatisfactory or worse? Sometimesa difference in the performance of the Bank’s

portfolio from one year to the nextcan be influenced by a change in thecomposition of projects beingevaluated—for example, if there is alarger share of projects in challengingsectors or countries. But the change in portfoliocomposition in fiscal 2007—related to Region,sector, instrument, lending arm, and otherfactors—does not explain the fall in ratings. Evenif the fiscal 2007 cohort had maintained the samecomposition of lending to conflict-affected andpostconflict countries as the fiscal 2004–06cohorts, the result would not be materiallyaffected. Appendix A shows the impact of variouschanges in the fiscal 2007 composition in moredetail.

Another possible explanation is that the drop inmeasured project performance is due to method-ological changes in the way projectswere evaluated in fiscal 2007. Somechanges in methods were introducedby IEG and the Bank together in fiscal2007 to strengthen the robustness ofproject ratings and to cover new elements ofproject design. In the near term they may haveintroduced some element of discontinuity in thedata series between fiscal 2007 and earlier years.9

It is estimated that the influence of methodologychanges has been small—accounting for around1 percentage point of the fall.

The final possibility is that in the fiscal 2007cohort, there was simply a greater occurrence offive key factors influencing weak outcomes. First,poor or overly complex project design has been aproblem in more than half of these underper-


1 3

Ongoing self-assessmentsof project performanceare increasingly overlyoptimistic.

Project outcomesdeteriorated in fiscal2007.

Share of projects rated moderately satisfactory or betterSource of rating FY05 FY06 FY07

Bank’s final Implementation Status and Results report (ISR) 87.9% 91.1% 93.0%

Bank’s Implementation Completion and Results report (ICR) 85.7% 89.9% 83.3%

IEG’s ICR review 80.8% 82.6% 75.8%

Difference between ISR and ICR review ratings—“disconnect” –7.1% –8.5% –17.2%Source: World Bank database.

Table 2.2: Disconnect between the Bank’s Self-Ratings and IEG Ratings Increased Dramatically in FY07

1 4


forming loans, a finding also made bythe World Bank’s QAG (World Bank2008a). For instance, two projectsfailed to recognize the importance of

an appropriate legal and regulatory framework asa precondition to a privatization process. Severalhealth projects failed to ensure a heightenedfocus on those interventions that would yield thegreatest impact, leading, for instance, toinadequate targeting of the poor, the absence of acost effective package of health services, orinadequate funding for behavior change interven-tions to prevent HIV/AIDS transmission amonghigh-risk groups.

Second, overambition was a weakness. Whileproject objectives were almost always relevant, amajority were too far-reaching. Sometimes thiswas in terms of assessing political commitmentand the feasibility of certain reforms. Other timesit was in assessing government effectiveness andcapacity, or in requiring coordination acrossseveral ministries or cumbersome financialmanagement procedures that were not manage-able by the parties involved. IEG’s evaluation ofpublic sector reform (2008c) shows severalexamples where these factors led to unsatisfac-tory outcomes.

Third, delays in implementation caused difficul-ties, because circumstances changed and projectdesign or implementation could not respond.About one-fifth of underperforming projectssuffered from this problem.

Fourth, a majority of the unsatisfactory projectshad a weak results framework with poor or nobaseline data, making it difficult to assess theoutcomes of the project, and outcomes wereoften not well linked to inputs and outputs.

Finally, various gaps in the Bank’s own perform-ance contributed to a lack of success. For example,despite being flagged by the QAG for poor qualityat the outset, three projects were not reassessedor redesigned. The quality of the Bank’s supervi-sion was rated as moderately unsatisfactory orworse in two-thirds of all underperforming

projects (and many such projects were not identi-fied as problems in ongoing status reports). AndBank overall performance—as distinct fromborrower performance or the effects of uncontrol-lable events—was ranked moderately unsatisfac-tory or worse in two-thirds of these problemprojects, compared with only about one-fifth ofthe full sample. All of this points to a challenge inre-emphasizing a proactive quality control inmanagement’s attention to ongoing projectperformance.

How It Adds Up: Outcomes of BankCountry ProgramsWhile Bank-supported projects have largelyyielded positive development outcomes, countryprogram outcomes—measured against their ownobjectives, which typically include growth,poverty reduction, and the environment—are farless satisfactory. Some previous assessmentssuggest that synergies between and among theBank’s lending, knowledge services, and dialogueare not fully exploited or that the projects are notalways directly relevant to the country’s coredevelopment challenges (World Bank 2008a; IEG2006c).

The lower country-program ratings could also beinfluenced by other factors. The number andscale of the factors that affect broader country-level objectives is typically much greater than atthe smaller-scale project level. Hence, outcomesmeasured at this level are more likely to beaffected by pressures outside the control ofdevelopment partners. It is also possible thatobjectives may be relatively more ambitious at theprogram rather than project level (and thereforeless frequently attained) or indeed that it is moredifficult to establish the connection betweenBank inputs and outcomes at the program level(as opposed to at the project level).

Over the past 10 years, evaluations of 81 Bankcountry programs—incorporating projects,policy and technical advice, and other types ofassistance—show that three-fifths of them weremoderately satisfactory or better in meeting theirdevelopment objectives. Looking at specific

Five factors haveinfluenced weak project

outcomes.

grades on IEG’s ratings scale, as shown in figure2.4 and table 2.3, the Bank succeeded in support-ing satisfactory outcomes in 30 percent ofevaluated programs; a further 30 percent of

country programs were rated moderatelysatisfactory. But the remaining 40 percent ofprograms—concentrated in countries that aresmaller or have extensive poverty, such as


1 5

Figure 2.4: CAEs Show Three-Fifths with Outcomes Moderately Satisfactory or Better

Source: IEG Country Assistance Evaluations.Note: CAEs assess country program performance over a long period. Years in parentheses indicate the period that Bank country programs are assessed.

(23)(22)

(21)Albania (98–04)Bhutan (93–03) Argentina (91–00)

Bulgaria (91–97) Bolivia (85–96) Armenia (93–02)Cameroon (82–94) Burkina Faso (89–99) Bosnia–Herz.(96–03)Costa Rica (90–00) Cambodia (92–99) Brazil (90–02)

Croatia (94–01) Cameroon (95–00) Bulgaria (98–01)Ecuador (94–98) Dom. Republic (85–02) China (93–02)

Guatemala (85–89) Egypt, Arab Rep. (91–00) Croatia (02–03)Haiti (86–01) Eritrea (92–00) Ethiopia (90–00)

Jamaica (80–98)(12)

India (90–00) Ghana (95–99)Malawi (96–05) Indonesia (90–98) Guatemala (90–01)

Mauritania (92–03) Angola (91–06) Indonesia (99–06) Lithuania (91–02)Moldova (93–03)

Ethiopia (01–06)Jordan (90–00) Maldives (80–98)

Nepal (90–99)Bolivia (98–04)

Kazakhstan (90–99) Mexico (89–91)Pap. New Guinea (90–99) Honduras ( 95–05) Kyrgyz Rep. (93–00) Mexico (97–00)

Paraguay (90–00) Lesotho (90–99) Mali (95–05) Peru (90–96)Peru (97–00) Madagascar (95–05) Mexico (95–96) Romania (00–04)

Romania (91–99) Mexico (92–94) Mongolia (91–01) Russian Fed. (99–01)Russian Fed. (92–98) Morocco (97–00) Rwanda (95–01) Tunisia (90–03)

(3)Rwanda (90–94) Pac. Islands (92–02) Senegal (94–04) Uganda (87–99)Ukraine (93–98) Pakistan (94–03) Sri Lanka (89–98) Vietnam (88–01) Chile (85–00)

(0)Zambia (96–01) Yemen, Rep. (90–95) Turkey (93–04) W. Bank & Gaza (93–00) El Salvador (89–00)

Zimbabwe (90–02) Yemen, Rep. (99–05) Ukraine(99–06) Yemen, Rep. (96–98) Uruguay (87–99)

Highlyunsatisfactory

Unsatisfactory Moderatelyunsatisfactory

Moderatelysatisfactory

Satisfactory Highlysatisfactory

Outcome rating of county programs in CAEs

Num

ber o

f cou

ntry

pro

gram

ratin

gs in

CA

Es

5 –

10 –

15 –

20 –

Distribution of CAE ratings, FY98–08

MICs LICs All countries

Highly satisfactory 6% 0% 4%

Satisfactory 33% 18% 27%

Moderately satisfactory 25% 33% 28%

Moderately unsatisfactory 13% 18% 15%

Unsatisfactory 23% 30% 26%

Highly unsatisfactory 0% 0% 0%

Satisfactory and highly satisfactory 39% 18% 31%

Moderately satisfactory or better 64% 51% 59%Source: IEG Country Assistance Evaluations.

Table 2.3: Summary of CAE Ratings, FY98–08

1 6


Malawi—were moderately unsatisfac-tory or worse in meeting their stateddevelopment objectives.

Very few country programs areproducing best-practice results—indeed, of 36programs rated since fiscal 2002, not one hasbeen highly satisfactory. And among the 14 low-income countries (LICs) rated since fiscal 2002,not a single program has been rated satisfactoryor better. At the same time, no program has everbeen rated highly unsatisfactory.

Many programs in the largest countries that housethe majority of the world’s poor—including Brazil,China, India, Indonesia, and Mexico—were ratedmoderately satisfactory or better in their develop-ment outcomes. Box 2.2 shows such outcomesfrom the Indonesia country program. Whencountry programs are weighted by the number ofpeople in poverty (living on less than $2 per day),86 percent of country programs are given a

satisfactory rating. Even when exclud-ing China and India, both above theline, the poverty-weighted satisfactoryshare is nearly two-thirds.

There are also significant differences amongsubgroups of countries, as shown in table 2.3. Bankprograms in MICs far outperform those in LICs.While IEG rated 39 percent of programs in MICs

satisfactory or better, this was true of only 18percent in LICs. Using a lower threshold, the differ-ence is smaller but still substantial: 64 percent ofratings in MIC programs were moderately satisfac-tory or better, compared with 51 percent of ratingsfor LIC programs.

A less in-depth but more up-to-date snapshot ofcountry program out comes is provided by IEGreviews of Country Assistance Strategy Comple-tion Reports (CASCRs). CASCR reviews differ fromCountry Assistance Evaluations (CAEs) in that theyare a review of the Bank’s own self-evaluation ofits country assistance program and cover a shortertime period, typically three to five years, ascompared with a decade or more in CAEs. CASCRreview data, available since fiscal 2004, present aslightly more favorable assessment. Two-thirds ofall programs reviewed were rated moderatelysatisfactory or better, as shown in figure 2.5 andtable 2.4.10 Here too, none of the countryprograms were rated highly unsatisfactory.

Consistent with the CAEs, higher ratings are almostentirely due to better outcomes in MICs than inLICs. More than three-fourths of the MICs hadmoderately satisfactory or better country out -comes, compared with half of LICs reviewed. Andin only one MIC was Bank assistance rated unsatis-factory, as compared with four LICs, even thoughthere were many more MICs in the sample.

No single template exists for a satisfactory Bank-supportedcountry program, since such programs vary greatly dependingon the country’s institutional capacity, stage of development, andparticular development needs. The characteristics of the Indone-sia country program illustrate some features of good outcomesfound in several cases.

During fiscal 1999–2006, the Indonesia country program waslargely successful. Bank support provided help to national au-thorities in securing economic recovery from the late 1990s eco-nomic crisis. By 2004, per capita income achieved precrisis levels,inflation fell to under 7 percent, and public debt was cut from

nearly 100 percent of gross domestic product in 1999, to under 50percent by 2005. These gains largely managed to turn around theeffects of the crisis—the percentage of Indonesians living on lessthan one dollar per day was 16 percent in 2005, compared with 23percent in 1999. The fall in income poverty and improvement inhuman development indicators were assisted by the Bank’s workin community-driven development through the Kecamatan Devel-opment Program and by reconstruction following the 2004 tsunami.These achievements outweighed some areas in which outcomesfrom the Bank’s program were less satisfactory—including in de-centralization and fighting against corruption.

Box 2.2: What Does a Satisfactory Country Program Look Like?

Outcomes of countryprograms are less

satisfactory than projectoutcomes.

Few country programshave produced best-

practice results.


1 7

MICs LICs All countries

Highly satisfactory 0% 0% 0%

Satisfactory 22% 13% 18%

Moderately satisfactory 54% 39% 48%

Moderately unsatisfactory 22% 30% 26%

Unsatisfactory 3% 17% 8%

Highly unsatisfactory 0% 0% 0%

Satisfactory and highly satisfactory 22% 13% 18%

Moderately satisfactory or better 76% 52% 66%Source: IEG reviews of Country Assistance Strategy Completion Reports.

Table 2.4: Summary of CASCR Review Ratings, FY03–08

Figure 2.5: CASCR Reviews Indicate That Bank Programs in MICs Outperform Those in LICs

Source: IEG reviews of Country Assistance Strategy Completion reports.Note: CASCR Reviews cover a shorter period than CAEs. Years in parentheses indicate the period that Bank country programs are assessed.

(26)

Albania (FY02–05)Argentina (4/04–12/05)Azerbaijan (FY03–05)Bangladesh (FY01–04)

Belarus (FY02–06)Bhutan (FY00–05)

Cameroon (FY96–02)Egypt, Arab Rep. (FY02–04)

Ghana (FY04–07)

(15)Honduras (FY03–06)

Jordan (FY03–05)Bosnia–Herz. (FY05–07) Macedonia FYR (FY04–06)

Gabon (FY99–04) Madagascar (FY04–06)Gambia, The (FY03–07) Maldives (FY00–07)

(11)Georgia (FY98–05) Mali (FY04–07)Guatemala (FY99–03/4) Mauritius (FY02–04) Armenia (FY02–04)

Kyrgyz Republic (FY03–06) Mozambique (FY04–07) Brazil (FY00–03)Malawi (FY04–06) Nicaragua (FY03–07) Bulgaria (FY03–05)

Mauritania (FY03–07) Pakistan (FY02–05) Burkina Faso (FY01–05)

(5)Morocco (FY01–04) Peru (FY03–06) Chile (FY02–06)

Org. E. Carib. States (FY02–05) Philippines (FY03–05) China (FY03–05)Cambodia (FY00–03/05) Russian Federation (FY03–06) Romania (FY02–04) Montenegro (FY05–07)

Lesotho (FY98–05) São Tomé & Princ. (FY01–05) Senegal (FY02–06) Tanzania (FY03–06)Nigeria (FY99–04) South Africa (FY00–06) Serbia (FY05–07) Turkey (FY04–07)

(0)Uzbekistan (FY02–05) Uganda (FY01–03) Tajikistan (FY03–05) Uruguay (FY01–05)

(0)Zambia (FY00–03) Yemen, Rep. (FY03–05) Ukraine (FY04–07) Vietnam (FY03–06)

Num

ber o

f cou

ntry

pro

gram

ratin

gs in

CA

SCR

revi

ews

Outcome rating of county programs in CASCR reviews

5 –

10 –

15 –

20 –

25 –

Highlyunsatisfactory



Satisfactory Highlysatisfactory

Distribution of CASCR Review ratings, FY04–08

Chapter 3

Evaluation Highlights• The Bank’s overall approach to M&E

has many strengths. • Progress has been made in updating

policies and frameworks, but thereis considerable room to improve howM&E is put into practice.

• At the project level, the overall qual-ity of M&E is low.

• At the country level, results frame-works are increasingly produced butare often poorly formulated.

• The independence of external eval-uations of global programs is im-proving, but their M&E systems areoften weak.

• Impact evaluations are a useful ad-dition, but topics need to be chosenstrategically.

Child studying in public school in the Amazon region of Brazil; photo by Julio Pantoja, courtesy of the World Bank Photo Library.

2 1

Underpinning Impact—M&E and Results Management

Monitoring and Evaluation Systems at the Project Level

The Bank’s overall approach to M&E has many strengths (see appen-dix B for an overview). Indeed, its policies for monitoring and evalu-ating projects have been revised, to place greater emphasis on project

outcomes. Specifically, the Bank:

• Required (a) investment projects to include re-sults frameworks, outlining (final) project andintermediate outcomes as well as the processfor carrying out M&E;1 and (b) developmentpolicy operations to specify expected out-comes and measurable indicators for M&E in2004 (World Bank 2004a).

• Replaced the Project Supervision Report withthe Implementation Status and Results report(ISR) in 2005. The ISR now gives more promi-nence to project outcome and intermediateoutcome indicators by including them in themain report, whereas in the past they were inan optional annex (infrequently updated).

• Harmonized the Bank and IEG’s evaluationcriteria for Implementation Completion and Re-sults reports (ICRs) and IEG’s ICR reviews in2006. The procedures for programmatic de-velopment policy loan ICRs were also simpli-fied to improve the effectiveness of the ICRs.

As part of the revisions, ICRs are now required toinclude an assessment of project M&E qualityalong three dimensions—design, implementation,and utilization. The review of M&E designexamines the extent to which adequate indicators

were identified to monitor progress toward projectdevelopment objectives. The assessment of M&Eimplementation is the extent to which appropri-ate data were actually collected, and utilizationreviews the extent to which appropriate data wereevaluated and used to inform decision making andresource allocation. IEG, as part of its ICR reviews,has started rating overall M&E quality using a four-point scale: high, substantial, modest, and negligi-ble.2 This quantitative assessment does not reflecta methodology that has been agreed to by Bankmanagement; ICRs assess, but are not required torate, M&E quality.

A review of the quality of project M&Eshowed a positive associationbetween good project M&E andbetter project outcomes. It found thatprojects with highly satisfactoryproject outcome ratings had, onaverage, higher M&E quality rating(3.0 or equivalent to a rating of “substantial”)compared with projects with unsatisfactoryoutcome ratings that had lower M&E quality (1.7or equivalent to a rating of “negligible”) as shownin (figure 3.1).

There is a positiveassociation between goodproject M&E and betterproject outcomes, but theoverall quality of projectM&E is low.

2 2


But the overall quality of project M&E has beenlow. Of the 242 ICR reviews for which M&Equality ratings are available,3 IEG rated thatquality as modest or negligible in two-thirds ofcases, as shown in figure 3.2. Poor design was akey factor in low M&E quality ratings. Someprojects had weak results frameworks where theproject outcomes were poorly defined and/orthe link among the project outcomes, that is, theproject development objectives, intermediateoutcomes, and the project outputs was not clear.Another factor was poorly designed M&E

systems, where the monitoring indicators lackedbaselines and targets and not enough attentionwas given to implementation.

Projects with high M&E quality ratings includedM&E systems that were well designed,implemented, and used. The Second RuralRoads Project in Peru included performancemonitoring indicators for outputs, intermediateoutcomes, and outcomes that were conceptuallyclear, realistic, and measurable and were linkedclosely to the project’s four main objectives.Impact evaluations were designed into theproject, and the evaluation of the previousproject provided the baseline for this project.The M&E framework also constituted an activelearning process for the project’s executingagency. The project was rated satisfactory foroutcome. The Partnership for Polio EradicationProject in Pakistan was rated highly satisfactoryfor project outcome and high for M&E quality.The project was able to use existing systems,combined with substantial external technical andfinancial support, to develop a system and createdemand for data, which are essential for diseaseeradication.

The impact of the changes in Bank M&Eprocedures may not yet be fully reflected in thisanalysis because most projects approved by theBoard since fiscal 2005 are still under implemen-

Figure 3.1: Projects with Higher Outcome Ratings Have Better M&E Ratings

Source: IEG.

1.71.8

2.0

2.6

3.0



Satisfactory Highly satisfactory

IEG project outcome rating

Aver

age

IEG

ratin

g of

M&

E sy

stem

Negligible (1)

Modest (2)

Substantial (3)

High (4)

High2%

(6 projects)

Negligible15%

(37 projects)

Substantial32%

(78 projects)

Modest50%

(121 projects)

Figure 3.2: M&E Is Rated Modest orLower in Two-Thirds of ICR Reviews

Source: IEG.

U N D E R P I N N I N G I M PAC T — M & E A N D R E S U LT S M A N AG E M E N T

2 3

tation. Of the 242 projects with M&E qualityratings, 23 have been approved since fiscal 2005,of which 21 are development policy operations.Close to half (48 percent) of these projects wererated substantial or high for M&E quality.However, it is not clear whether this is due tobetter M&E quality ratings for developmentpolicy operations or because of improvements inM&E policies. A clearer picture will emerge insubsequent years as more projects from fiscal2005 and afterward exit the portfolio and haveICRs and ICR reviews prepared.

It may not necessarily be policies and procedures,but rather a lack of incentives and priority that isconstraining the design and use of M&E systems.As far back as 2000, a Bank working group onimproving M&E found that M&E tended to beperceived as the least important of thedimensions of operational quality (World Bank2000). Focus group participants for IEG consulta-tions stated that they sometimes set ambitiousdevelopment objectives and then faced difficultylinking them to projects, that there was excessiveemphasis on quantitative data, and that monitor-

ing got less priority during implemen-tation. Difficulties in M&E have alsobeen highlighted in several IEG sectorand thematic evaluations, as noted inbox 3.1.

Many of these IEG findings are echoed in reportsfrom the Bank’s Quality Assurance Group. TheQAG’s most recent Quality of Supervision(QSA7) assessment noted satisfactory focus onmanagement of inputs and outputs, but less soon outcomes (World Bank 2007a). The QAG’slatest quality at entry assessment (World Bank2008b) found improvements in results frame -work design (where more than 95 percent of theproject were rated marginally satisfactory orbetter), but also observed that project develop-ment objective clarity and realism, consistency ofproject design with outcomes, and impact andoutcome measurement could be improved. TheBank is proposing to strengthen monitoringunder IDA15 by tracking the quality of theproject development objectives at design andthe adequacy of baselines during implementa-tion (World Bank 2007b).

IEG’s evaluation of the World Bank’s project-based and WorldBank Institute training (IEG 2008d) found that Bank projects andthe World Bank Institute both report on outputs (the number ofpeople trained, the number of training days, participant satisfactionwith training), but seldom on outcomes (changes in workplacebehavior, institutional capacity development). It recommended thatthe Bank: (a) develop guidance and quality criteria for the designand implementation of training, to enable quality assurance andmonitoring and evaluation of all its training support; and (b) im-prove the quality and impact of training by making available tech-nical expertise on the design, implementation, and monitoring andevaluation of training to its staff and borrowers.

IEG’s evaluation of World Bank assistance to agriculture in Sub-Saharan Africa (IEG 2007d) found that data systems and supportfor M&E have been insufficient to adequately inform the Bank’s ef-forts to develop agriculture in Africa. M&E at the project level isoften of limited value for answering fundamental outcome, im-pact, and efficiency questions, such as who benefited, which

crops received support and how, what has been the comparativecost effectiveness, and to what can one attribute gains. It recom-mended that the Bank improve data systems to better track activitiesit supports and strengthen M&E to report on project activities invarious agro-ecological zones for different crops and farmer cat-egories, including women.

IEG’s recent evaluation of the Bank’s Group’s work in sup-porting environmental sustainability (IEG 2008a) recommended thatthe Bank Group, as a whole, improve its ability to monitor andevaluate the impact of its environment-related interventions. Thisincluded using current International Finance Corporation and Mul-tilateral Investment Guarantee Agency evaluation systems as astarting point for betting monitoring for the whole Bank Group. Itcould further mobilize efforts for: (a) better baseline environmen-tal assessment studies; (b) more holistic evaluation methods; (c)further development of environmental performance indicators;and (d) improved monitoring and reporting on project implemen-tation and results.

Box 3.1: M&E Findings and Recommendations in Recent IEG Evaluations

Poor M&E impairs theability to manageongoing projects andsubsequently to evaluatethem robustly.

2 4


Country-Level M&E: Early Evidence fromResults-Based CASsThe Bank adopted the results-based CountryAssistance Strategy (CAS) approach in 2005,4

following a review that found that the evaluationframework remained the weakest area of the CAS,with the key issue being the “missing middle”between the CAS objectives and Bank operations(World Bank 2003a). A results-based CAS wouldbe “a country assistance strategy that contains a

strong orientation toward achievingrealistic outcomes and results-orientedmonitoring and evaluation system”(World Bank 2005a).

The two main innovations of the results-basedCAS approach were the CASCR as a self-evalua-tion tool and the results framework as a monitor-ing, management, and evaluation tool for theBank country program. The results framework isexpected to link higher-level country outcomeswith Bank CAS outcomes, intermediateoutcomes/milestones, and operations, andwould include measurable indicators of progressthat can be tracked through CAS implementa-tion, encouraging active management andallowing both self- and independent evaluation(World Bank 2005a).

Experience with this new approach suggests that,so far, the CAS results framework typically has beenused as a reference for evaluation, but not formonitoring or program management. Thisemerged from a review of results-based or results-oriented CASs and interviews with the Task TeamLeaders of the reports to examine the extent theresults framework was used for monitoring,management, and evaluation and to enhance theBank’s development effectiveness. CAS resultsframeworks served as an inventory of commit-ments in the CAS Progress Reports and were usedto report on the status of CAS outcomes and/or

intermediate outcomes without furtheranalysis. The results framework wasalso used as a reference in evaluatingthe country program in the CASCRs,which included an annex with thecurrent status of the CAS outcomes.One exception is Armenia, which in -

corporated the CAS outcomes into their countryportfolio performance review and is jointlymanaging the Bank program with the borrower tofocus on CAS outcomes (box 3.2).

Poorly designed results frameworks limited theirusefulness for country program monitoring andmanagement. It is difficult to interpret theinformation on a project’s status if the expectedoutcome is vaguely defined (“banking systemfinancial capacity strengthened”) and/or lacksbaselines and targets to measure progress.Another issue is the volume of information to beprovided. Assembling the data has provenchallenging: the Task Team Leaders for theProgress and Completion Reports mentioned thatthey had to devote a significant amount of timeand effort to collecting the data and updating theresults framework. In some cases, the amount ofinformation provided was overwhelming and yetfailed to answer the basic question: “Is the Bankmaking sufficient progress toward achieving itsCAS objectives.” The Ghana Progress and Comple-tion Reports addressed this issue by reportingactuals only for the 35 CAS monitoring indicators,which (except for those under the GovernancePillar) had baselines and targets.

IEG has also raised concerns over the poordesign of CAS results frameworks. Many CAS

The Bank strengthenedthe results focus of

its CASs.

CAS results frameworksare used as a basis for

evaluation but less so forcountry programmonitoring and

management.

As preparation for the 2007 CAS Progress Report, theBank conducted a joint portfolio review with thegovernment of Armenia, which covered not only theoverall status of the portfolio, implementation is-sues, follow-up on previous commitments, and indi-vidual problem projects, but also the progress towardCAS objectives and the contribution of Bank instru-ments. Information was prepared on the status of 29CAS outcomes and discussed with the government.Recommendations included specific steps, typicallybroader and policy-oriented, which are needed to im-prove project performance and to ensure achieve-ment of CAS outcomes.

Box 3.2: Armenia Joint Country Portfolio Performance Review

results frameworks had poorly articulatedresults chains—including in several IDAcountries where the distinction between CASoutcomes and intermediate outcomes/mile -stones was unclear and the links between Bankproducts and CAS outcomes were poor. Manyresults frameworks had too many out comemeasures—more than 40 CAS outcomes indica-tors and over 60 intermediate indicators—and/or lacked indicators with baselines andtargets, making them less effective as a manage-ment and evaluation tool.

The lack of outcome data makes it more difficultto assess the achievement of Bank CAS objectivesin the CASCR reviews. While most of the CASsreviewed to date were prepared before theresults-based CASs were introduced, some didinclude expected outcomes and/or resultsframeworks that were retrofitted in the CASCRs.IEG reviewers noted that the CASCRs discussedwhat the Bank did (process, inputs and outputs),but did not necessarily provide convincingevidence of the success of the Bank countryprogram in achieving its objectives.

Whether staff have the right incentives topromote effective results-based frameworks forCASs remains open to question. In interviews forthis report, some Task Team Leaders talked of thedifficulty of getting support from their sectorcolleagues when updating the information in theCAS results frameworks. Some sector staff didnot see value in the monitoring indicatorsbecause they were not involved in the formula-tion of the original CAS results framework. Giventhe demands on their time, monitoring of theCAS results framework and analyzing Bankcountry performance gets low priority, reflectingsome skepticism about the value added by theexercise. This echoes findings from previous IEGevaluations (IEG 2004a, 2006d) which concludedthat “getting results” was not yet part of thereward system for individual staff and that theBank’s incentives rewarded the work that thestaff do at the early stages of the project cycle—that is, preparing new operations—more highlythan work at the later stages of supervision, suchas evaluation and learning lessons.

One of the lessons learned from theProgress Report and CASCRs was theneed to better align Bank activities withCAS outcomes. The Armenia CASProgress Report noted that it is betterto build the results framework around PovertyReduction Strategy Paper outcomes and indica-tors. Baselines are important and other indicatorsshould be sought if baseline data are not available,and data sources should be verified to make surethat information can be made available in a timelymanner. The Ghana CASCR observed that one ofthe weaknesses was that some indicators wereonly indirectly related to Bank interventions. TheMozambique CASCR also commented on theimportance of the Poverty Reduction StrategyPaper, but noted that the CAS shouldset more realistic targets, distinguish-ing them from Poverty ReductionStrategy Paper targets and choosingsuch targets only if they are related tothe Poverty Reduction Support Credit(PRSC) or where the Bank is the main source ofsupport to achieve the Poverty Reduction StrategyPaper target.

Another lesson was the importance of strength-ening M&E capacity at the sector or project level.The Mozambique CASCR noted that projects alsoneed to strengthen their monitoring of results,and focus on helping sectors to develop monitor-ing and evaluation systems, which would helpproject data collection and reporting. TheBosnia-Herzegovina CASCR made a similar point,noting that the focus on results should gobeyond Bank-financed operations and that theBank teams should help their counterparts todevelop appropriate results frameworks for alltheir operations.

Linking the CAS and project resultsframeworks and strengthening thecountry’s capacity to conduct M&Ewould not only reduce the cost of datacollection, but also increase its rele -vance and use.

While the discussion focused on countries withCASs, M&E is also important in Low-Income


2 5

Poor design limited theusefulness of the resultsframeworks for Bank’scountry programs.

Incentives favoringproject preparation anddelivery over M&Econtinue to be a concern.

It is important to linkCAS and project resultsframeworks, andstrengthen countrycapacity to conduct M&E.

2 6


Countries Under Stress (LICUS),which are covered by Interim StrategyNotes or Transitional Support Strate-gies. An IEG evaluation (2006e) con -

cluded that M&E is just as critical, maybe moreso, in LICUS (box 3.3).

Going forward, country teams should examineresults chains and explore synergies amongvarious interventions in country programs(projects, analytic and advisory activities, trust-funded activities, and so on) to identify sharedhigher-level outcomes. This could lead to areduction in the number of country outcomes tobe measured, making the frameworks morestrategically focused and usable. Teams could also

identify areas where the project resultsframework could be linked into theCAS results framework. Establishingsynergies between interventions andidentifying possible gaps in the Bankprogram could go toward reducing the

disconnect between country and project portfoliooutcomes.

The basics need to be ensured. CAS resultsframeworks need to have monitoring indicatorswith baselines and targets to measure progress.

Systems need to be in place to ensure that theinformation is collected. This may requiretrading off data availability with desirability.

Finally, integrating the CAS results framework intoBank operational procedures would establish theuse of the performance data and could provide anincentive to update and utilize the CAS resultsframework. There are some pilots under way. Atthe country level, the Moldova Country Manage-ment Unit is piloting a scorecard to link CASoutcomes, operational performance, and Bankbudget allocation (box 3.4). At the regional level,the East Asia and Pacific Region piloted and isexpanding an approach to link the achievement ofCAS objectives with budget allocation (box 3.4).Linking Bank resource allocation with progresstoward CAS outcomes would also provide anincentive to monitor and analyze Bank countryperformance indicators. The Bank could learnfrom and build on these pilots.

Managing Global Programs andPartnerships: An Emerging AgendaThe Bank has put in place new business processesfor global programs and partnerships (GPPs),which recognize these as a separate product lineof the Bank, and which aim to integrate GPPbusiness processes and information with theBank’s regular operational business systems (IEG2004b). Throughout their life cycle, GPPs nowfollow procedures similar to a simplified lendingprocess—using familiar concepts and systems andallowing new GPPs to be tracked from their startthrough to evaluation and results assessment.

Enhanced processes have also been put in placeto encourage greater selectivity although theireffect remains to be assessed fully. Proposals fornew GPPs and requests for new DevelopmentGrant Facility (DGF) funding are now presentedin a standard Partnership Review Note and consid-ered, based on input from relevant parties in areview meeting chaired by a director. The GPPGroup and the Legal Department are providingearly advice to task teams with regard togovernance, which focuses on the quality of thegovernance structure (how the partners will formthe partnership and interact with each other),

M&E is just as critical inLow-Income Countries

Under Stress.

The Bank could take stepsto strengthen the resultsframeworks and better

manage for results.

Monitoring and evaluation are at least as important in LICUS as theyare in any other country. Monitoring and evaluation are crucial inLICUS for a number of reasons. First, the Bank, like other donors, is stilllearning what approaches work in LICUS contexts. Closely monitoringexperiences in order to draw lessons is critical, and learning and shar-ing needs to become a more prominent feature of LICUS work. Second,given that progress is often slow in these countries, it is important toreassess continually whether the program is on course to achieve thedesired outcomes. Third, a constantly changing and volatile LICUS en-vironment where progress is often nonlinear means that program adap-tation is essential—closely tracking performance will help determinewhen and what kind of adaptation is necessary. Effective learning-by-doing to improve the Bank’s future effectiveness in LICUS can only hap-pen with strong monitoring and evaluation.

Box 3.3: Results Measurement and Monitoring inLICUS

Source: IEG 2006e.

legitimacy, the voice of developing countries, andthe Bank’s roles (particularly whether the Bank’saccountability and responsibility are aligned withthe Bank’s formal authority and actual control). Anew results framework and performance indica-tors were added to the Partnership Review Noteand the DGF Progress Report on a voluntary basisin fiscal 2007, and made compulsory in fiscal 2008.

Quality-at-entry reviews of DGF-supported pro -grams by QAG also provide feedback that con -tributes to enhanced selectivity. QAG noted severalimprovements in fiscal 2007, as compared withfiscal 2006, including stronger quality of programdesign and improved readiness for implementa-tion, as well as a number of areas needing improve-ment, such as unrealistic objectives, weak resultsframeworks, and poor documentation.

Taken together, these represent a substantialimprovement over the previous partnershipapproval and tracking system for GPPs, which wasnot integrated into the Bank’s regular operationalbusiness systems. However, to date the improve-ments have mainly occurred with respect to DGF-supported programs, since these programs haveto comply with the new business processes inorder to receive grants. Except for DGF grantsreceived, the financial information in the Bank’sdatabases (on the sources and uses of funds foreach program) is not complete or reliable. None of

the Bank’s existing information systemsinclude financial resources that are notchanneled through the Bank, and theBank’s Task Team Leaders are notrequired to provide or update thisinformation on an annual basis.

The requirement for all programs receiving DGFfunding of $300,000 or more, over the life of theprogram, to undertake an independent evalua-tion every three to five years has led to an increas-ing number of program-level evaluations.However, the Bank has not yet extended thisrequirement to the many other programs notreceiving DGF support, as previouslyrecommended by IEG (2002), thoughit is currently revising the relevantOperational Policy to achieve this.Implementing such a requirementshould be fairly straightforward forprograms supported by Bank-adminis-tered trust funds. For programs that are notsupported either by the DGF or by Bank-adminis-tered trust funds, the Bank should still use itsinfluence, as a member of the governing body, toencourage periodic independent evaluations.

The independence of external evaluations isimproving. Out of the first seven evaluations thatIEG reviewed, five were executed independent ofthe management of the program, and without


2 7

Moldova Results ScorecardThe Moldova Country Management Unit is developinga Moldova Results Scorecard which will integrate, in oneplace, the different elements of country program man-agement: progress toward country strategic outcomes,lending and delivery costs of analytic and advisory ac-tivities, and quality (QAG and IEG assessments). Thescorecard will be used to link country program man-agement and resource allocation and improve programand project results.

Strategically Managing Country Programs in East AsiaThe East Asia and Pacific Region piloted the use of

CAS results frameworks to increase management ac-countability for results. The new approach is groundedin CASs and, at the same time, focused across core Re-gional priorities and actionable milestones. It is in-tended to be consistent across Regional and countrylevels and enables Regional aggregation to allow dis-cussion with senior management. Country teams tooktheir respective CAS results frameworks, agreed on thecritical CAS outcomes where they would focus their ef-forts, and narrowed the monitoring indicators to a man-ageable number in a one-page table. This informationwas used in the annual and midyear program/strategydiscussions.

Box 3.4: Use of CAS Results Frameworks in Country Program Management

Bank management hasput in place new businessprocesses that aim tointegrate GPPs with theBank’s regular business.

Feedback from QAGquality-at-entry reviewsof DGF-supportedprograms is contributingto enhanced selectivity.

2 8


apparent conflicts of interest. Thegoverning bodies generally commis-sioned the evaluation, approved theterms of reference, managed the selec -

tion process, and reviewed the draft reportindependent of program management. In fourout of these five cases, the selection of theexternal evaluators was competitive.

However, the quality of three of the seven wascompromised by a weak M&E system for theprogram. Either the objectives and strategies ofthe program were not well defined (too diffuse,process-oriented, difficult to measure, or open todifferent interpretations by different stakehold-ers), the M&E system was not well designed(focusing only on inputs and outputs, and not

outcomes), or the data on the progressof activities and on the achievement ofoutcomes were not systematicallycollected. And in three of the sevenprograms an inadequate budget didnot permit any fieldwork to verify factson the ground or elicit the perceptions

of implementers and beneficiaries in developingcountries. As a result, neither the external evalua-tions nor IEG found much systematic evidencerelating to the achievement of the sevenprograms’ objectives at the outcome level, and itis difficult to say whether the $100 million spenton the programs annually ultimately had asubstantial effect on the ground.

Notwithstanding these shortcomings,the external evaluations have hadsignificant impacts on the programsreviewed. The evaluation led to arevision in the strategic direction or

design of the program in six out of seven cases.The evaluation led to a change in the governanceof the program in four cases, and led directly tocontinued or increased funding in four cases.

Improving Our Understanding ofCausality: The Use of Impact EvaluationsIn response to the pressures for accountability,the development com munity in recent years hasaccorded increasing attention to impact evalua-tion as another tool for generating knowledge

about the effectiveness of development interven-tions. The challenge now is how best to use thisapproach for increasing knowledge regardingdevelopment issues, advancing the broaderresults agenda, and stimulating client engage-ment in, and capacity for, monitoring and evalua-tion.

The results agenda can potentially benefit fromimpact evaluations, although they are clearly nota panacea. Impact evaluations explicitly identify acounterfactual through experimental or quasi-experimental designs to measure the true impactof development interventions. Accurate measure-ment of results can help for efficient allocation ofresources and effective project/policy selectionand design. The potential benefits of an impactevaluation extend beyond any particular projector program because the evaluation can provideinformation and lessons for the broader develop-ment community. It is possible, therefore, thatimpact evaluations may be underfunded unlessthere is explicit support from management andclients. On the other hand, it must be recognizedthat impact evaluations are highly specific to thecontext and often quite costly to launch, and theirbenefits are uncertain until they are completed.

The number of impact evaluations at the Bankhas grown rapidly recently, with a rise from 60under way in 2006, to 158 in mid-2008.5 Yet, theevaluations are concentrated in a few sectors andtopics. Almost 70 percent of the ongoing evalua-tions are clustered in five areas, as shown infigure 3.3, and three of these—education, health,and conditional cash transfers (many of whichalso focus on education outcomes)—alsocomprise the top categories of previouslycompleted evaluations. There is also a concen-tration by Region, with nearly two-thirds ofongoing evaluations located in Sub-SaharanAfrica and South Asia (see figure 3.4).

Why evaluations are concentrated in certain areasand whether the observed distribution is efficientare questions for Bank management. Somesubstantive areas are more amenable to impactevaluations, and Bank clients differ in theirreceptivity to evaluations. But staff incentives may

The independence ofexternal evaluations is

improving.

External evaluations areinfluencing the strategic

directions and designs of GPPs.

The quality of someevaluations was

compromised by a weakM&E system for the

program.

also play an important role, and care must betaken not to overexamine some topics whileothers are underresearched. In an ideal setting,the decision to fund impact evaluations in a givenarea would take into account one or more of thefollowing five criteria: the value of answering thequestion in terms of benefits and costs of a specificproject, the value of answering the question forother current or future projects, the cost of theevaluation, the innovative nature of the project,and the likely feasibility of designing a convincingimpact evaluation.

The Bank has the capacity and re -sources to be a global leader in usingimpact evaluations to drive effectivedevelopment assistance. Required stepsinclude identifying important gaps inour knowledge; closing these gaps with effectiveand replicated evaluations; leveraging knowledgefrom impact evaluations through effective discus-sion, dissemination, and application across theinstitution; and shifting expenditures fromunsuccessful to successful interventions. Clientsshould be closely engaged and ideally, in many


2 9

Impact evaluations arenot a panacea, but theyare an important part ofthe results agenda.

Figure 3.3: About 70 Percent of Ongoing Evaluations Are Clustered in Five Areas

Source: Development Impact Evaluation database.

23

14

10

7

29

17

0 5 10 15 20 25 30 35

Education

Health

Community-drivendevelopment

Conditional cash transfer

Microfinance

All other areas

Eval

uatio

n to

pics

Share of ongoing impact evaluations (%)

Figure 3.4: Nearly Two-Thirds of Ongoing Evaluations Are Located in Two Regions

Source: Development Impact Evaluation database.

0

10

20

30

40

50

39

2420

13

2 2

Sub-SaharanAfrica

South Asia Latin America andthe Caribbean

East Asiaand Pacific

Europe andCentral Asia

Middle East andNorth Africa

Region

Shar

e of

ong

oing

impa

ctev

alua

tions

(%)

3 0


cases, leading the process ofimplementing evaluations and learningfrom their findings. A major challengefacing the Bank is to assess and comparethe relative effectiveness of alternativeinterventions rather than largely assess-

ing the effectiveness of different ways ofimplementing a given intervention. Existing initia-tives, such as the Development Impact EvaluationInitiative, have made some progress toward thesegoals.

IEG itself is aiming to increase its own use ofimpact evaluations and capacity to review impactevaluations within the Bank. Through meta-evalua-tions and mixed methods approaches, impactevaluations provide important opportunities toenhance IEG’s evaluation role.

Monitoring Institutional Effectiveness The development community is focusing on resultson the ground, with the Millennium DevelopmentGoals serving as a common framework. In 2002,the heads of the multilateral development banksissued a joint statement on their commitment to

scale up their work on measuring,monitoring, and managing for results,following the Monterrey Conference onFinancing for Development.

The IDA15 replenishment negotiationsdistilled this focus on results in a veryreal way. During the negotiations, the

Bank faced competition for donor funding fromdifferent organizations and programs, many ofthem focused on specific issues. Donor govern-ments had to answer to their citizens on theeffective use of their resources and thereforerequested that IDA demonstrate that it was achiev-ing results on the ground and what those resultswere. IDA’s competition often focuses on singleissues—such as children, health, and the environ-

ment—and are better able to presentaggregated output data (such asnumbers of beneficiaries assisted, goodsprovided, and so on) and to describetheir results. The Bank, on the otherhand, carries out not only projects butalso additional activities, such as policy

and institutional development or donor coordina-tion; impacts are harder to measure and attributeto the Bank. In response, the report “Additions toIDA Resources: Fifteenth Replenishment” (WorldBank 2008c) served to illustrate accomplishmentsin individual projects, sectors, and countries, thuscomplementing the information provided in theIDA Results Measurement System (RMS).

The IDA RMS assesses Bank-wide performance,but only for IDA countries. The RMS has two tiers.Tier 1 focuses on monitoring country outcomesand includes 14 country outcome indicators thatare consistent with the Millennium DevelopmentGoals, are priorities in poverty reduction strate-gies, and reflect IDA’s activities. These are high-level country outcomes that are influenced by theBank, but for which the Bank is not solely respon-sible. Tier 2 monitors IDA’s effectiveness. Itconsists of indicators: (a) at the country level,where the cumulative introduction of results-based CASs in IDA countries is monitored; (b) atthe project level, where four project qualityindicators are monitored; and (c) outputs fromcompleted projects in fours sectors (health,education, water supply, and rural transport).

IDA prepared a progress report on the RMS in2007 and made suggestions for improvedmonitoring under IDA15. It reported, amongother things, that IDA had built 7,500 km of road,trained 81,400 health professionals and 282,500teachers, and made 87,000 new water connec-tions based on IDA project ICRs in fiscal 2006–07.The report concluded that the introduction ofthe RMS “has generated a sharper focus onresults at the country level and a strongerinternal results culture.” It also acknowledgedthat internal process weaknesses remain in thequality of Tier 2 outcome indicators and theavailability of baselines for all outcome indicatorsat entry in IDA projects. IDA recognized thatbecause “many ICRs still lack figures on outputs,or report them in a variety of forms (differentunits, no standard categories), aggregationremains very challenging.” For IDA15, the reportproposed that IDA track the quality of the projectdevelopment objectives at design, the adequacyof baselines during implementation, and the

The developmentcommunity is focused on

results on the ground andis interested in the impact

of Bank operations.

The IDA ResultsMonitoring System and

AfricaRMS provide abroader view of results,

but not for all countries.

Impact evaluation topicsshould be chosen

strategically and theresults should influence

future funding decisions.

reporting quality of outputs and outcomes inICRs, based on the original results framework aspart of Tier 2. The report also suggested includ-ing ratings on CAS implementation from IEG’sCASCR reviews to measure the quality of Bankcountry programs.

The Africa Region developed the Africa ResultsMonitoring System (AfricaRMS) as a comprehen-sive online system in the Bank for monitoring on-the-ground results of Bank activities. It consists ofa set of harmonized project and country indicatorsto facilitate counting and aggregating results at thecountry, sector, Region, and Bank-wide levels.AfricaRMS partnered with the education, malaria,HIV, water, private sector development, gov -ernance, agriculture, energy, and roads teams tobuild monitoring frameworks that are fully alignedwith existing global initiatives. It also includeswrite-ups to tell the story behind the numbers,describing how good results are changing people’slives, and descriptions of impact evaluationscarried out in Africa. The AfricaRMS is aligned withthe Africa Action Program.

The goal of the AfricaRMS is to use selectinformation to show the results of Bankoperations and development progress for both

the Region and individual countries. It not onlyhelps teams focus on achieving results but alsoprovides internal and external clients access tolive information about stories from projects andbeneficiaries, impact evaluation results, and datafor priority sectors.

The system is still being refined and faces severalchallenges. The Region is working to improvethe availability of country data and harmonizedindicator measurement across Bank operations.It is also making efforts to better integratefinancial data, and output and outcome datafrom operations at the sector and country level.Measurement systems across countries andamong development partners will also need tobe harmonized.

IEG’s mandate is to evaluate the outcome ofBank-supported projects and programs, asmeasured against their objectives, and its work—including ARDE 2008—obviously represents apart of the Bank Group’s efforts to monitorinstitutional effectiveness. An assessment ofIEG’s effectiveness, including the perspectives ofits clients and the extent to which its recentrecommendations have been implemented, canbe found in appendix C.


3 1

Chapter 4

Man sifting seeds along the Red River in northern Vietnam; photo by Quy-Toan Do, courtesy of the World Bank Photo Library.

3 5

Lessons and Opportunities

Practical Lessons for the Near Term

Some practical lessons emerge from this analysis. The increasing dis-connect between Bank and IEG project ratings needs to be reversed byrenewing attention to project reporting and self- assessments consistent

with standards set by IEG. The drop in 2007 project ratings also must be watched carefully.

To improve the focus on results, steps areneeded: (a) at the project level and in global andregional programs, to enhance the quality of theM&E systems, especially by working to put inplace good baseline information, to elucidateclearly the link between project outputs andtargeted outcomes, and to extend evaluationrequirements to global programs beyond thosesupported by DGF; (b) at the country level, tosimplify results frameworks and thus make themmore useful in guiding and evaluating programs;and (c) at the institutional level for the Bank andin partner countries, to manage and learn from agrowing number of impact evaluations, includ-ing by better integrating them into countryprograms and exploiting cross- country synergiesin conducting and sharing studies.

Directions for the Overall Bank AgendaThe development community’s focus on resultson the ground has also generated interest in theBank’s overall development impact. The IDA15replenishment exercise reinforced the importanceof articulating the results (both outputs andoutcomes) of Bank operations to donors andother external stakeholders.

Despite increased international interest, the Bankis not yet well positioned to articulate the results ofits interventions. The IDA RMS and the AfricaRMSaim to report on the Bank’s achievements in asystematic manner, but for a specific set ofcountries. The Bank is considering a corporate- level results report, which is expected to synthe-size the Bank’s contribution to results, in place ofthe Sector Strategy Implementation Update.

The experience of developing M&E systems atthe project and country level, and for the IDARMS and AfricaRMS, provides lessons andchallenges for developing a Bank- wide resultsreport, including:

• Performance measures should be limited to amanageable number that capture the critical el-ements reflecting development effectivenessand can be measured and monitored rigorously.

• A monitoring system should be designed not onlyto report on results but also to manage for re-sults, that is, to collect performance informationto assess whether the Bank is on track to achieveits corporate strategic objectives, make changeswhere needed, and allocate resources. Opera-

3 6


tional use of the performance measures wouldprovide an incentive for managers and staff to or-ganize, collect, and analyze the data.

• The Bank would need to ensure that its infor-mation systems are aligned, to supply the in-

formation and minimize the data assemblycosts, for example, by standardizing projectand country performance indicators as muchas possible (a move already under way in theAfrica Region and in several networks).

PART II

Shared Global ChallengesLessons from the Bank’s Experience

Chapter 5

Evaluation Highlights• Development issues that involve in-

ternational collective action, suchas climate protection, present someof the greatest challenges of our time.

• Fostering global public goods is nowone of the Bank’s six strategic prior -ities.

• The differing nature of the variousGPGs influences the way in whichthe Bank and its development part-ners address them.

Satellite image of hurricane; photo ©Adastra/ Getty Images, reproduced by permission.

4 1

The Challenge of Global Public Goods

In an era of rapid globalization, the importance of coordinated global so-lutions to cross- border problems has risen in tandem. In recent yearsthere has been enormous growth in international attention to climate

change, avian flu, international trade agreements, and other supranational is-sues collectively known as global public goods.

Many important GPGs— whose key characteris-tics are highlighted in box 5.1—need collectiveaction across countries to be deliveredeffectively. This applies, for example, to keepingthe air clean or concerted campaigns againstcommunicable diseases. Individual countriesmay not have the incentive or wherewithal totake action, and thus GPGs can be chronicallyundersupplied even when there is widespreadrecognition of the urgency of worldwide action.

The Bank, for its part, is taking steps to addresssome of these issues. In previous decades, whilemost of the Bank’s financial and advisory supportwas about providing public goods, it wastraditionally focused at the country level, onnational public goods such as public health,regulatory frameworks, and the provision ofinfrastructure. More recently, the Bank hasincreasingly been called upon to help in thesupply of public goods that transcend borders.

Economists describe pure public goods as “nonrival” and“nonexcludable.” Nonrival means the supply of the good, suchas clean air, to one person (or country) does not lead to therebeing less of it for another. Nonexcludable means that once thegood is provided for one person, it is available for all to bene-fit from it. Typically, at the margin, the net benefits accruing toprivate individuals from such goods are less than the net ben-efits for society as a whole, and hence the public good is un-dersupplied in private markets. Public goods require collectiveaction to be properly provided and, at the national level, this can

often be coordinated by using government powers (including tax-ation, spending, and regulation).

Importantly, public goods also have a spatial dimension. Theirgeographic reach runs across a continuum from local communityboundaries, to national borders, to regions of several countries, tothe global sphere. The usual problems in supplying public goodsare exacerbated for truly global public goods. That is because thereis a divergence between the costs and benefits captured at the na-tional and global levels, and it is particularly difficult to secure col-lective action across countries.

Box 5.1: Key Characteristics of Global Public Goods

4 2


To this end, the World Bank Group president haspromoted the fostering of GPGs as one of theBank’s six strategic pillars (World Bank 2007c).And the Development Committee endorsed anew framework for the role of the Bank in thisarena in October 2007 (World Bank 2007d). TheWorld Bank Global Public Goods Working Grouphas been appointed to take forward thepresident’s strategic pillar. Further steps areneeded for the Bank to achieve its objectivesoutlined in the framework.

It is important to recognize that the variety ofGPGs have very different features, as discussedin more detail in appendix D. From a demandside, when viewed from an individual countryperspective, some have benefits that are closelyaligned to national interest (for example, thecontrol of HIV/AIDS). In others, such as protect-ing the earth’s climate by reducing emissions ofgreenhouse gases, there is a great divergencebetween national and global benefits. Part of theBank’s role— particularly in advocacy work andin providing grant finance— can be viewed as away of “internalizing the externality” andbringing national and global demand for GPGscloser together.

From a supply side, GPGs also havedifferent characteristics. Some requireonly one or a few countries to beinvolved in their supply, whereasothers need all countries to be active(for example, ensuring the participa-

tion of a country that would, otherwise, be theweakest link in a strategy to stop the spread of acommunicable disease). Some GPGs requiremany donors to be involved— opportunities forthe Bank to use its convening power— while thesupply of others may rest with other major publicand private institutions.

The differing nature of GPGs therefore has implica-tions for the way in which the Bank (and otherbodies) deals with them. The Bank, as a global

institution with considerable reach, is potentiallywell positioned to help the international com -munity work at all levels in fostering GPGs. TheBank claims (World Bank 2007d) its comparativeadvantage in fostering GPGs is that it can:

• Work with countries to help them integrateGPGs into their national policies and programsas well as country assistance strategies, ana-lytical and advisory activities, and lending.

• Display constructive advocacy by using its re-search and analytical capacity to communicatethe perspectives and interests of developingcountries in international arenas.

• Be an active partner in responding to globalchallenges and a source of innovative financ-ing mechanisms.

Putting such comparative advantage into actionhas posed challenges for the Bank and itspartners. The Bank has focused on five identifiedGPG clusters: preserving the environmentalcommons, controlling communicable diseases,enhancing the participation of developingcountries in the global trading system, strength-ening the international financial architecture,and creating and sharing knowledge relevant fordevelopment (World Bank 2007d).

This report examines the ways in which theBank’s country- based model helps or hinders itssupport for GPGs. Given the breadth of the topic— spanning numerous programs andvirtually all sectors— and the lack of comprehen-sive evaluative evidence, this report is necessar-ily a partial stocktaking of the situation. It paysspecial attention to evidence on experienceswith two of the most prominent GPG themes— the environmental commons and communicable diseases— and refers to other examples (includ-ing trade) where the context and availableinformation warrants. The closing chaptersreview the Bank’s experience as an advocate foraction on GPGs and draw overall lessons for theBank as it develops its approach in this area.

Global public goods havedifferent features, which

has implications for the way the Bank deals

with them.

Chapter 6

Evaluation Highlights• Systems for integrating GPGs into

country strategies are under -developed.

• Attention to GPGs gets diluted as itmoves from the Bank’s corporatestrategy to lower levels.

• Bank administrative spending onGPGs is among the smallest for its sixpriorities, and the growing role oftrust funds presents challenges.

• Concessional finance is importantto foster many GPGs.

• A mismatch between country needsand global ambitions for GPGs lim-its the use of the country model, especially without an attractive financial instrument.

Wind turbine farm; photo ©Frank Whitney/ Getty Images, reproduced by permission.

4 5

Using the Bank’s Country- Based Model to Foster GlobalPublic Goods: Does It Work?

How It Works in Theory—the Bank’s Strategic Setting for Fostering Global Public Goods

Relying on the country-based model as the platform for the Bank’s workon GPGs is a double-edged sword. Why? Because the Bank’s systemslargely mirror international structure, giving primacy—for obvious rea-

sons—to national decision making on policies and programs supported by theBank. GPGs that permit significant benefits to be captured at the country level—and so global and national interests coincide—are best suited for the Bank’scountry-based model.

Country directors and their teams are able towork in the traditional way with their counter-parts to develop approaches supported by theBank, be it through knowledge, conveningpower, or finance.

When a GPG has benefits, however, which arenot easily or meaningfully appropriated at thenational level (for example, cleaner air providedas a result of more energy-efficient productiontechnologies), or where the costs of providingthe GPG fall disproportionately on an individualcountry (for example, income foregone by notusing forest resources), then the country modelcomes under strain. Staff interviews confirm thatcountry directors and their teams do not havethe incentive to advance an agenda that does not

directly appeal to their counterparts. Nor do theytypically have the internal budget or instrumentsto be able to do so effectively.1 In short, it isdifficult to bridge the gap between global needsand country preferences.

Experience with Bank Country AssistanceStrategiesCountry Assistance Strategies2 for partnernations define proposed country programs andform the bedrock of the Bank’s overall engage-ment with developing countries. Moreover,because a great deal of the Bank’s work, particu-larly its financing and knowledge services, isexplicitly incorporated into individual CASs, suchstrategies are of pivotal importance as a startingpoint for the Bank’s work in connecting global

4 6


and country issues. What does experi-ence tell us about how GPGs areplaying out in country strategies?

The templates for compiling a CAS donot make explicit reference to global programsor GPGs as issues that must be considered.Likewise, templates for CAS Completion Reportsand IEG’s own Country Assistance Evaluationsdo not give explicit guidance on GPG issues. Thismay well be a gap worthy of further review. TheBank’s internal CAS review process does allowfor draft material to be circulated for comment tovarious parts of the Bank beyond the specificcountry team, including relevant groupsinvolved with GPGs, such as the SustainableDevelopment Network and the Bank’s GlobalPrograms and Partnerships (GPP) unit. Thisprocess gives some opportunity for that perspec-tive on GPGs to be added. It is not common,however, for experts from the centrally basedanchor units (in the Bank networks) to be activemembers of the CAS preparation team.

Which GPGs feature strongly in Bank countrystrategies? A review of major recent CASs foundthat the environmental commons is the GPGmost frequently noted explicitly in countrystrategies. Climate change, air pollution, andcarbon emissions are reported (in the vast

majority of CASs) as being addressedthrough the Bank’s own lending oranalytic and advisory activities, theGlobal Environment Facility (GEF) orthe carbon funds. This observationresonates with IEG’s earlier evaluationof MICs (IEG 2007a), which found that

mention and integration of global programs inCASs was more likely for those close to the Banksuch as the GEF and the carbon funds.

Attention to communicable diseases is thesecond most frequently emphasized GPG inCASs. But in examining CASs it is often difficultto determine whether work on communicablediseases is primarily focused on nationalinterests (and national goods) or whether theinterests of the wider global community are alsodriving or influencing the Bank’s stance. In about

70 percent of CASs reviewed, the documentsmention work on either HIV/AIDS, tuberculosis,malaria, or avian flu—which might be taken asthe upper bound of attention on these GPGissues. About one-third of the CASs reviewedmention Bank support for national diseasesurveillance systems.

This pattern has been confirmed in interviewswith Bank country directors and their staff.3 Theyreport that they are open to considering andincluding important GPG issues in their dialoguewith counterparts and, ultimately, in corecountry strategies. When the Bank is able to drawa practical connection between a GPG and thepoverty focus of the Bank, it has a base uponwhich to build. The key then is to exploit thecomparative advantage of combining the Bank’sglobal and local knowledge, because it is the mixof the two that allows the greatest added value tobe delivered to the client. But country directorsreport two significant drawbacks in deliveringthese potential benefits.

First, many respondents argue that, in a nutshell,“the networks are not working.” From theirperspective, the Bank’s centralized anchor unitshave not been providing the valuable cross-country research, expertise, and support neededto help foster GPGs at the individual countrylevel. In part, this may reflect gaps in the Bank’sown global knowledge but respondents alsosuggested it was a problem of ineffective dissem-ination (internally, and to external audiences) ofglobal expertise and research. This last theme isa recurring one—several evaluations from IEGand others have drawn attention to continuingflaws in the way the Bank transmits its analyticaland research work (IEG 2007a).

Second, country directors remain concernedabout what, at times, appears to be a bewilderingarray of global programs with which the Bank isassociated. In some instances, the informationmade available on global programs is so volumi-nous and broad that it is difficult for country teamsto digest and use. In other cases, some globalprograms (and their internal sponsors) circum-vent the normal country dialogue approach, and

Systems for integratingGPGs into CASs are

underdeveloped andworth upgrading.

Environmental commonsis the GPG noted mostfrequently in country

strategies.

U S I N G T H E B A N K ’ S C O U N T RY- B A S E D M O D E L TO F O S T E R G L O B A L P U B L I C G O O D S : D O E S I T WO R K ?

4 7

foist their agenda onto the national dialogue.Many country directors appeal for a rationalizedapproach that tailors information from globalprograms much more carefully to country circum-stances, which would facilitate a stronger engage-ment over the long term.

With the Bank’s increased attention at thecorporate level to GPGs, has there been a similarincrease in coverage of GPGs in CASs? Thisreport reviewed a relevant sample of countries inwhich the Bank has significant engagement andfor which recent and historic CASs are available.In sum, there does not seem to be enoughevidence to say that the treatment of any of themain GPGs is significantly different in the mostrecent CASs from their earlier equivalents. Infact, the picture is rather mixed. For example, thelatest Vietnam CAS and Brazil Country Partner-ship Strategy (CPS) seem to address GPGs in amore extended fashion than earlier strategies. InPakistan, the treatment is similar between thetwo CAS periods, as is also the case for the ChinaCAS (which has had reasonably good coverageover an extended period).

One important issue highlighted by staff incountry and network positions is the differingtime horizons that apply in different strategies.Country strategies typically cover two to fouryears, which has to be aligned with national

pressures (often including electoralcycles). Yet, global strategies to dealwith GPGs have time horizonssometimes measured in decades, andimplementation measures typicallyfor five years or more. This “mismatchof time horizons” poses an overarching problemfor bringing public policy to bear on medium-and long-term global challenges. It is also asource of tension reflected in the country-levelclient dialogue and priorities across differentgroups (for example, country teams and anchorspecialists) within the Bank. How to make amore effective link between available short-terminstruments and medium-term challengesdeserves further consideration (IEG 2008a).

The latest Brazil CPS4 provides a very good exampleof clear and prominent integration of GPG themesin country-level strategies (see box 6.1).

The Experience with Bank Network (Sector)and Regional StrategiesGiven the sporadic attention to GPGs in countrystrategies, are strategies above the country levelproviding more attention to GPGs? That is thecase with regard to corporate-level strategy,where the Bank has paid significant attention toGPGs, most notably in the GPG Framework(World Bank 2007d) and Long-Term StrategicExercise (World Bank 2007e). But these strate-

The latest Brazil CPS is unique among the CASs reviewed in thisARDE for its treatment of GPG issues. The partnership strategyexplicitly references GPGs and particularly highlights the in-herent tension in promoting GPGs at the country level:

The Bank will engage with Brazil with climate change, trade, andother global public goods issues. In so doing, the Bank will helpto ensure that Brazil’s perspectives and interests are repre-sented, in ensuring that the line between global goods and na-tional ownership is not blurred, in ensuring that developmentneeds are given equal billing. In short, the Bank’s perspectiveswill be to ‘level the playing field’ on global public goods.

Brazil is special for its central role in dealing with many global publicgoods (including AIDS, climate change, biodiversity, and clean en-ergy and for the fact that it is a demanding borrower that ‘pushes theBank to the next level,’ by insisting that the Bank enter areas where theBank has been reluctant to participate . . . that the Bank engage withglobal public goods from the perspective of the developing world.

The CPS is also notable because it is a joint strategy betweenIBRD and the International Finance Corporation, which includesboth private and public sector responses to GPGs, particularlyon environmental commons issues. The CPS is so recent that theeffectiveness of the new approach remains to be evaluated.

Box 6.1: Brazil: A Best Practice in Integrating GPG Themes in Country Strategies

There is no evidence thatthe treatment of GPGs in CASs has expandedacross-the-board over time.

4 8


gies could have been more specific onhow to translate corporate prioritiesinto country-level action. Indeed, theLong-Term Strategic Exercise acknowl-edges the limitations of the country-based model as a tool for fostering

GPGs, particularly regarding country incentivesto diffuse country knowledge (World Bank2007e).

What about the strategies “in the middle”—thoseof the networks5 and Regions? A review of suchdocuments found that explicit attention to GPGsbecomes diluted as it passes from network anchorstrategy to Regional strategy. And typically, neithernetwork nor Regional strategies fully elucidatehow these strategies will play out at the countrylevel.6 Attention to GPGs is more prominent inboth network and Regional strategies dealing withthe environment, as compared with those dealingwith the health sector.7 The likely reason isbecause of the type of intervention needed forcontaining communicable diseases, whichrequires a strong national focus that might not beexplicitly connected to global action.

A good example is the Bank’s environmentstrategy, Making Sustainable Commitments(World Bank 2001), and the 2007 SectorImplementation Strategy Update (World Bank2007f), which provides a strong focus on institu-tions, governance, and outcomes. It is one of thefew network strategies that elucidate in detail howeach Bank Region should undertake actions tofoster global/regional environmental commons.Translating strategy into practice, however, provedto be difficult.

The Sustainable Development Network reportedin 2003 that mainstreaming environmental consid-

erations in sectoral projects, programs,and policies had been slower thanexpected. IEG (2008a) found thatalthough main streaming has advancedsince 2001, it has remained incompletebecause of limited incentives and toofew independent Bank resources to in -tegrate environmental componentsinto other projects. The Environment

Sector Board has identified and catalyzed newpartnerships for emerging climate change issues,but in other areas has not fully followed throughon its stated commitments to realign globalpartnerships with strategic objectives and toimprove the governance and management of suchpartnerships (IEG 2008b). Similarly, the DGFCouncil stated during the fiscal 2007 screeningprocess that the environment sector “seemed tolack a unifying strategy and that proposals werenot linked with country-based and other globalprograms” (IEG 2008b).

The Bank’s Regional strategies (which vary signif-icantly in format and vintage across Bank Regions,with Africa being particularly comprehensive)typically cover GPGs somewhat superficially.Climate change and HIV/AIDS are mentioned inall Regions, but scant attention is paid to lookingat the promotion of GPGs in a more comprehen-sive or strategic manner. Nonetheless, some goodpractice is emerging in Regional and sector-specific strategies, as highlighted in box 6.2.

The Bank’s attention tofostering GPGs getsdiluted as it moves

from corporate–levelstrategy down.

Though mainstreamingenvironmental

considerations haveadvanced since 2001, it

has remained incompletedue to incentives andresource limitations.

The current Latin America and Caribbean Regionalstrategy includes concrete references to proposed ac-tion on important GPGs. Global issues is one of the fourpillars of Bank action for the Region and includes top-ics such as climate change, trade negotiations,HIV/AIDS, and avian flu. The Regional vice presidentelevated the importance of climate change and theBank’s involvement in the external media by noting inan op-ed piece that a fair global system should makelower carbon-intense emissions possible by levelingthe playing field between developing and developedcountries.

The 2005 Health Strategy developed by the AfricaRegion includes a substantive discussion of how to in-tegrate global priorities with country priorities (WorldBank 2005c). Acknowledging the growing number ofglobal initiatives aimed at controlling communicablediseases, the document outlines the importance of theBank in helping client countries manage and benefitfrom the complex array of global initiatives.

Box 6.2: Emerging Good Practice fromthe Regions


4 9

Several factors could contribute to the dilutionof GPG focus in Bank strategic documents fromthe corporate level to the networks and Regionsand finally to the country. It may be that account-ability for GPG issues is lessened, or control overresources diminishes, as one moves down themanagement chain. Both could compromise acountry team’s ability and desire to advocate forand to deliver GPGs. Some interview respon-dents also postulate that task managers andcountry teams feel they have their hands full withthe Bank’s traditional mandate and are naturallyreluctant to take on another. Without accompa-nying resources or accountability, a corporatemandate from above becomes more difficult totrace down to the country implementation level.

Country Programs in Practice—from Strategy to ActionIn moving from strategy to action at the countrylevel, at least three factors come into play. First,the broad attention paid by the Bank to GPGs ispartly reflected in the overall allocation ofcorporate resources—the Bank’s own budgetand trust funds—to this priority. Second, theparticular financial instruments at the Bank’sdisposal—including grants, concessional IDA,IBRD loans—influence country activities. Third,global programs have become an increasingly

popular tool for the Bank to deploy. Each ofthese is discussed below.

Allocating the Right Level of CorporateResourcesThe Bank reports that its expenditures andlending for the purposes of fostering GPGs haverisen rapidly in recent years (using a widely drawndefinition of GPGs, owing to the lack of specificadministrative coding).8 These estimates shouldbe treated with some caution becausethey may be subject to significantvariations depending on the defini-tions and classifications used in theirconstruction. Indeed, going forward,more precise definition and tracking ofspending would be important asmanagement tools. Expenditures, including trustfunds, have nearly doubled in five years, from$56.2 million in fiscal 2002 to $108.0 million infiscal 2007, reaching approximately 4 percent oftotal Bank expenditures. The bulk of this $50million rise is concentrated on increased spendingin the GPG areas of environmental commons (anincrease of $22 million) and trade (an increase of$16 million), as shown in figure 6.1.9

These funds come from three sources, includingthe Bank’s core budget, Bank-executed trust

Figure 6.1: Bank Expenditures on Main GPG Themes

Source: World Bank, Global Public Goods Working Group.

0

20

40

60

80

100

120

2002 2003 2004 2005 2006 2007

FIscal year

Ban

k ad

min

istr

ativ

e bu

dget

and

trus

t

fund

exp

endi

ture

s on

GPG

s ($

mill

ion)

Communicable diseases

Environmental commons

Trade

International financial architecture

Bank expenditures arerising rapidly, with thebiggest increase for workon environmentalcommons.

5 0


funds, and other funding sources suchas GEF. The core budget expenditureshave nearly doubled over the period,from $32.3 million in fiscal 2002 to$59.7 million in fiscal 2007. Inaddition, trust fund expenditures haverisen from $13.9 million to $26.2

million, and the other sources have increasedfrom $10.1 million to $22.1 million.

IBRD and IDA lending has grown even faster,from $968 million (and 5 percent of all lending)to $2.5 billion (10 percent) over the same period.The bulk of this increase was in the area of trade,where lending volumes increased from $76million in fiscal 2002 to $988 million in fiscal 2007as shown in figure 6.2. A large increase alsooccurred in the area of environmental commons,from $467 million to $879 million. Lending forcommunicable diseases has fluctuated around$400 million over the period, and that forinternational financial architecture has been very

small.

Other sources of funds have beenprominent in financing work in environ-mental commons. The carbon funds,GEF, Montreal Protocol, and otherspecial funds, together, accounted for

over $1 billion of finance in fiscal 2006 alone. Whilethis amount was a peak, these funds accounted foran average of $270 million during fiscal 2002–07.

Are these numbers—particularly the Bank’sadministrative budget—adequate for the task? Itis impossible to say because there is nobenchmark against which to calibrate. It ispossible to highlight the difficulties of budgetsand incentives staff face, as reported ininterviews with key respondents.10 The budgetsystems of the Bank have largely been designedto fit its model of business, which has been inplace for several decades—with lending as thecenterpiece—and have not yet been smoothlyadapted to reflect the requirements of newbusiness. Country directors and their sectordirector colleagues can struggle to meetcorporate pressure to conduct GPG work at thecountry level when there is modest or littlecounterpart demand and no obvious instrumentfor follow-up. Some innovative responses havebeen developed, such as the Latin America andCaribbean Region’s “beam” funding, whichallocates a modest share of Bank budget adminis-trative funding on a competitive basis to Sustain-able Development Network sector teams tostimulate their work on GPGs in specificcountries.

Bank lending for GPGshas more than doubled

over the past five years—to about 10 percent

of total lending in fiscal 2007.

The Bank’sadministrative

expenditure on GPGs isone of the smaller

allocations for its sixstrategic priorities.

Figure 6.2: IBRD and IDA Lending for Main GPG Themes

Source: World Bank, Global Public Goods Working Group.

0

500

1,000

1,500

2,000

2,500

3,000

2002 2003 2004 2005 2006 2007

IBRD

and

IDA

lend

ing

($ m

illio

n)

Communicable diseases

Environmental commons

Trade

International financial architecture

FIscal year

At the network level, staff also report that budgetand incentive pressures are inhibiting, but it isdifficult to assess this from an external viewpoint.The Bank’s need or desire to be responsive to itsshareholders and stakeholders leads it to be anactive partner in a growing number of global initia-tives. Network staff often do not have adequatebudgets to fund their work—such as attendingplanning and governance meetings of globalprograms, or producing tailored research inresponse to partner requests. And against such asqueeze, the staff are placed in difficult positionsof trying to fulfill other services, including provid-ing operational and other specific support tocountry teams.

Overall, while the increase in resources of theBank to GPGs has been substantial, renewedattention to those GPGs—both globally andwithin the Bank—implies there will be pressurefor the Bank to devote substantially moreemphasis and, therefore, resources to GPGs.Since the Bank’s administrative budget isprojected to remain flat, and across-the-boardincreases in lending volumes will be modest atmost, the Bank would then have to rely increas-ingly on other sources. Grant resources andmarket-based mechanisms may be used to financecountry-based projects, and trust funds to financeBank expenditures. If not, the Bank would have tomake a major reallocation of its own Bank budgetaway from its traditional work and toward GPGs.

This corporate decision to use external fundingstreams, particularly trust funds, to address GPGscould lead to fragmentation of efforts, rigidityover allocations, greater administrative costs,subjection to particular donor requirements andpreferences, and a disconnect from the basicresource allocation mechanism that governs thecountry-based model. The Bank has sought tomitigate such risks through several measures,including standardizing its trust fund policies andengaging with donors to harmonize approaches.

Using Different Financing Instruments

Bank Concessional Finance—IDA For LICs, the Bank can support country action

through the concessional—and hencemore attractive—financing of IDA.Indeed, in recent years, the Bank hascommitted substantial IDA funding tohelp countries in programs with clearGPG dimensions—such as HIV/AIDS,avian influenza, and environmentalcommons. For example, in Vietnam,the Bank has been able to use its multisectoralexpertise, combined with concessional IDAfinance, to help the authorities cope with thethreat of avian flu, in part, because there was astrong national interest in averting economicfallout in the domestic food industry.

Often, implementation capacity on the ground isstretched, however, and in those circumstances,Bank staff report that national demands,understandably, may take precedence over someGPG considerations. Furthermore, for nationalcounterparts who are dealing with wretchedpoverty or postconflict reconstruction on a day-to-day basis, the goals of GPGs can seem ratherdistant and lofty. Again, staff report that there isgreat reluctance among national partners andcountry teams to allow IDA allocations targetedfor poverty reduction to be diverted to fosteringGPGs whose benefits may not be felt by thepoorest.

A related and pressing point that has beenobserved in several high-profile cases in Africa isthe growing presence of vertical global programsand funding mechanisms. Most vertical funds arein the health sector and they focus much neededattention and resources on specific problems,such as HIV/AIDS, malaria, and other communi-cable diseases. While these are vital issues thatpose national, regional, and sometimes globalchallenges, the influence of vertical funds risksdiverting domestic resources from other healthsector and national priorities. Such funds mayplace more pressure on national administrationsand can exacerbate aid fragmentation.

For example, in Ethiopia, more than half of thecountry’s health budget recently came fromHIV/AIDS global programs (IEG 2005). In caseswith large resources from global programs, the


5 1

Trust funds are a growingforce in the delivery ofGPGs—this poseschallenges with regard tofragmentation, donorrequirements, andpreferences.

5 2


Bank must work with client countries to carefullyweigh its role and comparative advantage. Indoing this, there may be instances in which a trackrecord of successful Bank support for, say,HIV/AIDS projects in a specific country warrantscontinuing active engagement. In other circum-stances it may be better to reallocate resourcesaway from “inundated” areas to where the Bankcan be more effective, for example, in integratedhealth care systems. The Bank should alsocontinue its aim to support government-led aidcoordination mechanisms, and at the global level,through the Paris Declaration and follow-upmeasures. Progress needs to be accelerated.

A recent innovation in IDA has beenthe creation of a separate fundingenvelope to support Regional projects.Established on a pilot basis under

IDA13, the approach was continued under IDA14,and the allocation for it increased by over 70percent under the IDA15 framework (World Bank2008c). These resources can be used to “top-up”IDA resources provided to countries through theregular country performance-based allocationsystem, as applied to pertinent Regional projects.The rationale is to encourage and facilitatecountries’ participation in Regional projects, helpdefray the extra costs associated with such cross-border cooperation, and meet what is perceivedto be a significant demand and opportunity formore (and more ambitious) Regional projects.The Bank’s Africa Region has helped push Banksupport for Regional cooperation through thecreation of a director position and a dedicated unitto promote Regional projects and programs.Although it is still early, there are signs that thisIDA initiative is bearing fruit. It should bemonitored for lessons as to whether this approachshould be replicated for some GPGs, though greatcare would be needed to avoid fragmenting IDA’soverall framework.

Other Concessional Finance—GrantsWhen the Bank has had a clear andviable instrument to support itscountry partners in taking action onsome environmental concerns, thereare signs of success. A good example of

this is the GEF. Formed in 1991, the GEF had avery clear mandate, with the full backing of theinternational community. Its basic operationalstructure and financial management originated inthe Bank itself, and the professional experts whohelp promote GEF projects are Bank staff. Its grantfinance has proved equally appealing to MICs andLICs alike (IEG 2007a). Indeed, in China—whichhas benefited from about $510 million of GEFfinancing in 45 projects since 1991—the presenceof the GEF has been a key ingredient in helpingthe Bank and national authorities form a strongand practical partnership to tackle issues thatwould not otherwise have been addressed. Clientrepresentatives report that Bank staff have keptthem adequately informed on a range of globalprograms and initiatives. Furthermore, Bankinvolvement has been valued where it helpsdemonstrate new approaches that can besubsequently scaled up—an important feature ofChina’s 11th five-year plan (IEG 2007a).

In cases where the Bank has not had an obviouslyattractive financial instrument—and/or wherethere has been a lack of demand from countrypartners—it is less easy to see progress. Forexample, Bank effectiveness in promoting globalenvironmental sustainability, including tacklingclimate change, has been mixed. Considerableattention has been given to biodiversity conser-vation, but, with the significant exception ofChina, less has been given to greenhouse gasmitigation, and, until recently, almost none tohelp countries adapt to the likely future impactsof climate change. In Senegal and Uganda, forexample, the link between natural resourcemanagement and poverty has been largelyoverlooked in Bank lending (IEG 2008a). Muchof this could be the result of a wider concernabout how developing countries are compen-sated for their investments in GPGs. The Bank’sapproach is beginning to change, however, andmuch greater attention is envisaged by both theBank and International Finance Corporation toclimate-related challenges, including with thenewly emerging Climate Investment Funds.

Several developing countries have expressedapprehension about having to design national

IDA has innovated withspecial support forRegional projects.

When the Bank has anattractive instrument to

support its countrypartners in taking action,there are signs of success.


5 3

projects that include global interests withoutreceiving additional funding for the benefitsaccruing to the international community (ITF2006). Although the inclusion of GPG concerns innational strategies is socially desirable, the issuestill remains contentious about how developingcountries will be compensated for having to incurhigher costs (while benefits are distributedglobally) when implementing developmentprojects. For example, the creation of renewableenergy electrification projects, without compen-sation for higher costs (as compared withtraditional electrification projects), will mean areduction in the number of beneficiaries.

Nonconcessional and Market-Based FinanceA key point reported by some operational staffand client representatives is that there is often amismatch between country needs (andresources) and global ambitions on many GPGs.11

In MICs, for example, the Bank tends not to be asignificant player in financial terms and, indeed,its main instrument—IBRD lending—is noncon-cessional (IEG 2007a). Hence the Bank’s ability toinfluence (or persuade) a country to takeconcrete action on some GPGs is inherentlylimited, but the effective provision of those goodsrequires deep participation by these MICs, asdiscussed in box 6.3.

The limits of nonconcessional finance areillustrated in the Bank’s work on avian influenza.

Only some of the benefits of control-ling avian influenza can be capturedby individual countries, and so it tendsto remain a low national priority. Sofar, the Bank has supported 50 avianinfluenza and pandemic preparednessprojects under its Global Program for AvianInfluenza.12 But only seven of these projectsinclude IBRD finance, for a total of $94 million,and among these, only two are for sizable sums.Moreover, to date, only $12 million of IBRD loanshave been disbursed.

In the two large IBRD projects—in Romania andTurkey—the positive externality of controllingavian influenza was largely captured withinnational borders by factors having nothing to dowith the Bank. In both cases, poultry exporterswere prevented from getting theirproducts to the important EuropeanUnion market because of fears that thedisease could spread to people, provid-ing strong national incentive to takequick action. This is a good example ofhow external pressure (economic or legal) canmake the most of the Bank’s country-based modelfor fostering certain GPGs.

In Indonesia, an evaluation of the Bank’s countryassistance program from 1999 to 2006 showedthat it covered forestry issues with large-scaleanalytical work but little lending. Over that

The provision of many GPGs depends critically on the actionsof middle-income countries. Indeed, MICs are home to many ofthe world’s most important environmental assets. They are thesource of more than 40 percent of the world’s carbon emis-sions; the strength of their financial systems directly affects thefortunes of other countries globally, and the prevention of com-municable diseases by MICs could be critical to avoiding wide-spread contagion. Actions taken by MICs are therefore essentialto the GPG agenda. Moreover, the Bank has indicated that GPGswill be a critical focus of its engagement with MICs, providingone of the primary justifications for a continuing relationship(World Bank 2006a).

In addition to these potential synergies, however, there are alsotensions between these two agendas. MIC clients are increasinglycalling on the Bank to become more client-focused and respon-sive to their needs, given the vast expansion of choice they haveenjoyed in financial and technical support for development (IEG2007a). At the same time, almost by definition, many GPGs—par-ticularly those with the weakest link and aggregate effort char-acteristics—require action that countries would not otherwisechoose to take. The Bank must carefully navigate these inherenttensions and trade-offs. Doing this well would lead to success andeffectively deliver on both the MIC and GPG agendas, but failingon one would likely weaken the other.

Box 6.3: Bank GPG and MIC Strategies: Fates Entwined

Although the inclusion ofGPGs in nationalstrategies is desirable,doing so remainscontentious.

The Bank’s work on avianinfluenza illustrates thelimits of the Bank’snonconcessional finance.

5 4


period, the traction achieved by the Bank wasvery limited, and deforestation continued at arapid clip.

In contrast, when national and global interests(and benefits) are closely aligned, nonconces-sional finance can prove a workable instrument.Over the last two decades, the Bank has been ableto secure very substantial country-based action onthe promotion of trade—recognized as animportant GPG. To complement its global-levelwork on trade regimes, the Bank has supportedprojects and programs in some 117 countries,with a total of $38 billion of finance (8 percent oftotal Bank commitments) since the late 1980s(IEG 2006a).13 About 70 percent of those projects

have delivered satisfactory develop-ment results in the countriesconcerned, although the poverty-reducing aspects of trade reform havenot often been fully delivered (IEG2006a). In recent years, the Bank hasalso creatively brought together the

nexus of global, regional, and country interven-tions on trade—for example through successful,regionally structured, trade and transport facilita-tion projects (IEG 2007c).

The use of market-based finance, alongside othermore-established instruments, is demonstratedwell in the Bank’s work on China, which is one ofthe world’s largest emitters of carbon dioxide andwhere 70 percent of energy is coal-based. TheChinese authorities have actively engaged withthe Bank, and priority has been given in the Bank’scountry strategies to financing clean andrenewable energy as well as projects for the cleanstorage of carbon dioxide emissions, afforestation,and recycling. Indeed, to date the Bank’s programhas helped the Chinese to adopt a RenewableEnergy Law, while other analytic and advisoryactivities and technical assistance work hasfocused on enhancing biodiversity. Furthermore,the Bank has been able to leverage funds fromglobal programs like the GEF and the MultilateralFund for the Implementation of the Montreal

Protocol for renewable energy andozone-depleting substance projects.Finally, the Bank has helped China

utilize the Clean Development Mechanism of theKyoto Protocol through carbon funds, focusingparticularly on the capture of the greenhouse gas,trifluromethane.

Deploying Global Programs at the CountryLevelGlobal and regional programs and partnerships14

are another vehicle the Bank can use to bringnational and global interests closer together. Thenumber of these global programs supported bythe Bank has grown rapidly in recent years,reaching about 160 in fiscal 2008. Global andregional programs are very diverse, in both theirobjectives and structure. A program can actprimarily as a cross-country network, with littleor no financial resources available; it can financetechnical assistance; it can finance specific invest-ments; or it can act in a combination of theseways. It can support GPGs through investmentsat the country, regional, or global level; or it canhave nothing to do with GPGs at all.

Indeed, when one looks at the financing forthese programs, three characteristics emerge.First, only 56 global programs (35 percent of thetotal) focus primarily on providing GPGs, asshown in figure 6.3a.15 Second, a majorityshare—57 percent—of Bank-executed resourcesthat the Bank devotes to global programs areallocated to the GPG-focused programs. Third,when one looks at all funds that the Bankmanages, including recipient-executed trustfunds, the share devoted to GPGs grows to alarge majority, as shown in figure 6.3b—92percent. These figures raise interesting issuesregarding the prioritization of GPGs in theBank’s support for global programs.

Many global programs—particularly those thatprimarily deliver GPGs through national invest-ments and technical assistance—require actionat the country level. But despite the Bank’s directrole as a partner in these global programs,systematic linkages to country programs haveproved challenging. Many of the global programshave garnered only modest participation by MICs(IEG 2007a). And Bank performance in globalprograms overall has been better at the global

In China, the Bank hassuccessfully used market-based finance alongside

its other, moreestablished instruments.

Global programs havegrown rapidly in number.

than at the country level, in part because, in theabsence of a requirement to do so, taskmanagers for global programs rarelydemonstrate how the program will help specificcountries (IEG 2004b).

One reason for limited participation by develop-ing country partners is that the Bank’s decisionto support a global program can be the result ofpressures beyond those coming from its clientcountries, including strong donor interests,advocacy by international nongovernmentalorganizations (NGOs), and other geopoliticalconsiderations. Sometimes these programsattract limited interest from developingcountries.

How can global programs be better integratedinto country operations? Simply basing a globalprogram within the Bank—there are 57 suchprograms—does not guarantee effective linkages.For example, these linkages were weak in thePopulation Reproductive Health CapacityBuilding Program, in spite of the potentialsynergies with Bank investment operationsoccurring in the same countries. Even whencountry teams and global program teamscommunicate, the natural tendency is for global

program staff and Bank staff to focuson their own activities, for which theyare held accountable, while collaborat-ing primarily with established counter-parts. For example, Bank country teamstaff often liaise with ministry officialsoverseeing country programs, while globalprogram staff often liaise with a different set ofofficials, donors, specific interest groups, andspecialists dealing with GPGs.

Indeed, Bank Task Team Leaders of globalprograms have reported that there are sharpincentives against close linkages between countryand global programs. On occasion, the strongpresence of a global program in a countrysometimes leads a country team to take a division-of-labor approach, scaling back its program in thesame sector. Importantly, country teams have fewresources to work on global programs unless thereis a demonstrated interest by the country client.This is even more acute when a global program ishoused outside the Bank because there is little orno core Bank budget earmarked forprogram oversight. Moreover, whilecountry teams are interested in budgetand concessional finance that maycome with participation, their interest


5 5

Figure 6.3: Most Global Programs Do Not Focus on GPGs, but Most of the Bank’s GPP ResourcesAre Devoted to Those That Do

Source: World Bank database. Note: Bank-executed resources cover Bank budget and Bank-executed trust funds. Bank-managed resources also include recipient-executed trust funds.

-

50

100

150

200

250

Global public goods National public goods andother issues

US$

mill

ions

US$

mill

ions

a. Bank-executed resouces b. Bank-managed resouces

(20 programs)

(36 programs)

(98 programs)

(4 programs)-

500

1,000

1,500

2,000

2,500

Global public goods National public goods andother issues

(20 programs)

(36 programs)(98 programs)

(4 programs)

Financing investments

Supporting technical assistance or network activities

It has proven challengingto make systematiclinkages between globalprograms and countryprograms.

Merely basing a globalprogram in the Bank doesnot guarantee effectivelinkages.

5 6


de pends on the flexibility of the fundingto support country priorities that may ormay not overlap with global priorities.16

IEG has found that stronger legitimacy of aprogram appears to foster stronger linkageswith country operations (IEG 2004b, 2008b).This legitimacy is enhanced by substantialdeveloping-country representation in thegoverning body (discussed in the chapter 7).Stronger linkages emerge in the programs whendeveloping countries have greater voice, in partbecause these programs are more likely toreflect developing-country interests and to berelevant to developing-country needs andcircumstances.

The approach toward M&E for global and regionalprograms is substantially less developed than thatwhich has been established over time for theBank’s traditional IBRD and IDA instruments.Global programs are relatively new and rarely self-

sustaining financially, and hence it isvery important to establish a robustM&E framework at the outset of aprogram. But IEG’s recent review ofseven global program evaluationsfound the quality of all seven wascompromised by weak M&E systemsfor the program. There fore, neither the

external evaluations nor IEG found much system-atic evidence relating to the achievement of theseven programs’ objectives at the outcome level.It is impossible to say whether the global programinterventions—together accounting for about

$100 million of spending in 2007—ultimately hadany effect (IEG 2008b).

The multicountry aspect of GPGs also applies totopics and responses that are best handled at theregional level, that is, by groups of neighboringcountries. The Bank faces a similar challenge indemonstrating how it can link regional andcountry concerns and opportunities. One majorinstrument for doing this is through regionalprograms, which have increased in importancein recent years, though they still account for amodest share of Bank lending. In reviewingCASs, it is clear that mentioning regionalprograms is also much more the exception thanthe rule.

As the cross-border dimensions of health,environment, and trade facilitation are expand-ing worldwide, the contribution of regionalprograms to address regional public goods andGPGs is likely to grow in significance. Consensusamong participating countries regarding thedistribution of program benefits and costs, aswell as strong country voice in governancearrangements, are key characteristics which IEGidentified in successful regional programs (IEG2007c). An example of an effective regionalprogram is the Regional Hydropower Develop-ment Project, which was designed to manage theSenegal River Basin serving Mali, Mauritania, andSenegal. This project built a hydroelectric plantthat successfully responded to the needs of thethree countries by providing a reliable, low-costpower supply and increased electricity access.

There are, in fact, strongincentives against

making such linkages.

The approach to M&E forglobal programs is

substantially lessdeveloped than

approaches used in Bank lending.

Chapter 7

Evaluation Highlights• Bank advocacy goes beyond the

country level and involves producingcollective global responses and pro-moting the development interests ofthe poor.

• Promoting global trade reform rep-resents the Bank’s advocacy at its best— in its analytical and practicaldepth and its proactive dissemination.

• The Bank’s advocacy work on avianinfluenza built on robust economicanalysis, convening power, fiduci-ary reputation, and multisectoral expertise.

• The Bank has had some advocacysuccesses related to environmen-tal commons, though its influenceon advancing work on climatechange is more debatable.

• The voice of developing countriesis still underrepresented in globalprograms, particularly in their gov-ernance.

Indonesian elementary schoolchildren wear surgical masks in response to 2003 outbreak of SARS in China; photo ©Reuters/Corbis, reproduced by permission.

5 9

The Bank’s Advocacy onGlobal Public Goods:

What Has Worked and What Has Not?

The Dimensions of Advocacy

The Bank can advocate for action on GPGs in at least three ways. First,it can encourage specific actions supported by the global communityat the country level. This lies at the heart of the Bank’s country- based

model and was discussed in chapter 6.

Second, the Bank can help inform collectiveglobal responses to providing GPGs. For example,independent and robust research on the costs ofcommunicable disease threats (such as avian flu)can help inform international partners in develop-ing an appropriate response. Practical analysis,and sometimes the Bank’s convening power, canalso be used to help develop markets— such asthat for carbon finance— and so deliver new toolsto provide GPGs.

Third, the Bank’s focus on poverty and develop-ment can be used to promote the interests ofdeveloping countries in international dialogueon mechanisms to tackle GPG issues. This isparticularly applicable where the community ofnations considers developing and implement-ing (binding) international agreements to set aframework for collective action. In such cases,the Bank has a potentially powerful ability toensure that development interests are properly considered— not least, how costs and benefits

of new arrangements will affect the world’s poor.

This chapter explains aspects of advocacy usingthree high- profile examples— trade, environ-mental commons, and avian influenza. It drawson the importance of voice for developingcountries, especially related to the governance ofglobal programs. Finally, it presents a perspectiveon very recent innovations of the Bank insupporting advocacy through its financial capabilities.

Advocacy at Its Best: The Bank’sExperience with TradeThe role of the Bank as a constructive advocatefor developing countries on GPG issues has beenshown in its work on trade (IEG 2006a). Arguably,the Bank was somewhat slow to emphasize thisstrand of its work, but it was given prominencefrom 2001 onward, significantly later than othercountry-based components of the Bank’s work

6 0


which had been growing since the late 1980s. Butthe Bank emerged as a leader in global tradework with high-profile reports ahead of majorinternational trade reform meetings in Doha(2002) and Cancun (2004).1

Furthermore, the Bank’s advocacy role wasuseful in positioning the Bank among clientcountries, especially those with which it mightotherwise be less engaged on trade issues. Forinstance, even though some Latin Americacountries had been skeptical with regard to theBank’s policy advice because of the perceivedfailure of the Washington Consensus (Rodrik2006), the Bank has had an active country-level

engagement on trade. This illustratesan important connection—that theBank may indeed have to be seen as aconstructive advocate in the globalarena if it is to open doors at thecountry level, to persuade and workwith national partners to deliver on-the-ground action. It may well be thaton other GPG priorities—for example,

climate change—the Bank is not yet seen as sucha high-profile advocate for developing countries,hence constraining its delivery capacity at thecountry level.

What were the ingredients that contributed to theBank’s advocacy role in trade? Certainly a longgestation period of working on the ground anddirectly with partner countries (es peciallythrough the late 1980s and into the 1990s) ontrade issues, gave the Bank a good understandingof, and motivation to promote, the impact oftrade on development. This was a case in whichthe global-country link worked in both directions.

Beyond that, the assembly in-house of first-rateintellectual and analytical research capacity ontrade was essential (Deaton and others 2006).2

This gave the platform for the Bank to producedetailed, innovative, and well-regarded reports that informed theongoing debate in trade, most notablythe benefits to be gained from a morefree-market approach to the marketsfor agricultural products and other

items produced in significant quantities in thedeveloping world. The research itself wasaccompanied by a proactive and highly visibledissemination effort. One aspect was a willing-ness to engage in public debate—including inmainstream media channels—and take whatsome might regard as fairly strong positions onissues of some contentiousness.3 All of this tookplace in a very specific setting: the live negotia-tions for the Doha round of the World TradeOrganization rules-based regime. This last pointis also critical, because it gave the Bank’s work adirect opportunity to gain traction.

The advocacy role was not an unqualifiedsuccess, and at least two drawbacks haveappeared. Recently, the analytical basis andfindings of some of the Bank’s earlier work hasbeen challenged, and indeed some recalibrationshave been made. This, in itself, hardly undoesthe impact of the Bank’s advocacy but itillustrates the difficulty of putting in place analyt-ical foundations that are deep and can stand thetest of time. The second drawback relates to howadvocacy is then followed through with practicalaction to build developing-country capacity. Themost high-profile initiative in trade capacity-building work was the Bank’s involvement in theIntegrated Framework for Trade-Related Techni-cal Assistance—a multidonor, multiagencycollaboration set up in 1997 to provide trade-related technical assistance to 49 less-developedcountries. This global program instrumentexhibited some weaknesses, however, includinga slow pace of implementation, insufficient focuson improved trade outcomes, and a shortage offunds. The absence of a good results-basedmanagement framework also typifies a problemcommon to a good number of global programs.

The Complex Challenge of EnvironmentalCommons and Climate ChangeThe Bank has long paid attention to the globaldimension of the environmental commons—andmore recently to climate change itself. Indeed,there are very specific examples in which theBank has played a practical advocacy role todevelop new approaches to what is a verycomplex challenge. When the GEF was being put

Key ingredients includeda long period of working

directly with partnercountries, first-rate

research capacity, and awillingness to engage in

public debate.

The Bank’s work alsogained traction in the

context of “live”negotiations for the Doha

round of trade talks.

together, there was considerable debate amongpartners as to whether resources should be newor drawn from existing allocations to the Bank(and others). For its part, the Bank did play anadvocacy role and is reported as holding theposition that “it would participate only ifadditional funding was made available.” In theend, an agreement was reached in London in1990 with “the World Bank, and specifically thepresident of the bank, clearly designated as theadministrator and manager of the centralfunction of the fund: financing projects andprograms to meet the incremental costs of article5 [that is, developing country] parties”(Benedick 1998).

The Prototype Carbon Fund (PCF) emerged fromadvocacy work by the Bank and others, asexplained in box 7.1. The main, universallyrecognized contribution of the PCF is that it gavecredibility to the burgeoning field of carbonfinance. The PCF is also respected by several

experts in the field because they assessthe methodologies it has developedfor ascertaining additionality to berelatively sound, especially as com -pared with other projects approved bythe United Nations Clean Develop-ment Mechanism. Some 22 methodologies formeasuring additionality developed in the PCFand Carbon Finance Unit of the Bank have beenapproved by the Clean Development Mechanism.But there is not unanimity on these issues, asshown in box 7.1.

Advocacy has also been very noticeable in theBank’s contribution to setting up new ClimateInvestment Funds. Emerging from preliminaryhigh-level discussions at the Gleneagles Summitin 2005, the creation of these funds has beenassisted by the Bank and other partners over thelast three years. Now it is reported that a multibil-lion dollar pledge has been made by the govern-ments of Japan, the United Kingdom, and the

T H E B A N K ’ S A DVO C AC Y O N G L O B A L P U B L I C G O O D S : W H AT H A S WO R K E D A N D W H AT H A S N OT ?

6 1

On environmentalcommons, the Bank hasplayed a positiveadvocacy role in somevery practical settings.

Since the 1990s, the Bank has supported proposals for carbonfinance—an instrument to contain carbon dioxide and other harm-ful emissions by allowing a developed country government or com-pany to meet some of its own environmental obligations by investingin projects in developing countries, to help reduce emissions there.The Bank’s president promulgated the concept at the United NationsGeneral Assembly in 1997 (World Bank 1997), and the Bank’s advo-cacy bore fruit with the launch of the Prototype Carbon Fund in 2000,the world’s first carbon fund. Since then, the PCF’s catalytic effecthas been an example of advocacy through demonstration, and hasbeen one factor in the global carbon market’s exponential growth,from $38 million in 2002, to $64,000 million in 2007.

Carbon finance is not without controversy, and critics have ques-tioned the extent to which additionality occurs and if investedprojects are truly sustainable. For example, two studies by outsideobservers conclude that “additionality is unlikely or questionablefor up to 40 percent of registered projects” and that “left to mar-ket forces, the Clean Development Mechanism does not signifi-cantly contribute to sustainable development.” A forthcoming

detailed IEG examination of additionality in Bank-supported car-bon projects should give a more accurate assessment.

The PCF’s advocacy position was initially hampered around theissue of governance. The host developing countries had no voicein PCF governance. In response to this situation, the Bank, afterthe first year of PCF operation, created a Host Country Committee,which now covers all Bank-managed carbon funds. But the HostCountry Committee remains strictly advisory and some membersstill complain that they have no contact with the funds’ participantscommittees.a Other members and Bank managers argue that someaspects of the operational structure cannot be fully inclusive be-cause of the need to discuss confidential pricing information.Some Host Country Committee members would like to see in-country capacity-building efforts on a greater scale and impact thanthe $21 million allocated to the Carbon Finance Assist program,housed in the World Bank Institute.

The new Carbon Partnership Facility, scheduled to becomeoperational later in 2008, will need to carefully address these andother tensions.

Box 7.1: Advocacy for Carbon Finance: The Bank’s Role in the Growth of a World Market

a. The two committees meet at the same time but in different rooms. The current Chair of the PCF Participants Committee has never met the Chair of the Host Country

Committee. Under the new Bank-supported Carbon Partnership Facility, expected to become operational in late 2008, host-country members are to have equal repre-

sentation and voting rights with company members.

6 2


United States to kick-start this significant initia-tive. But as illustrated in box 7.2, there is substan-tial pressure on the Bank to mainstreamclimate-friendly approaches into all its develop-ment work.

The example of the Montreal Protocol (andindeed of the carbon finance business, whichemerged from the incentives created by KyotoProtocol commitments) illustrates an importantpoint: the World Bank can play a strong role inthe supply of GPGs, but primarily whensupported by an appropriate treaty instrument.The Montreal Protocol is one of the mostsuccessful treaties in history. The Bank hascontributed to its success, but the real creditbelongs to the treaty itself and to the nature ofthis particular global challenge (Barrett, 2008).Similarly, the Bank’s ability to contribute to themuch harder problem of climate change willdepend on the effectiveness of the treatyarrangement for this challenge.

The extent to which the Bank hasbeen a leading advocate to date on thebroad issue of climate change is moredebatable, particularly now that globalclimate change is considered by manyas one of the greatest challenges of

our time. The Bank’s analytical and researchwork has certainly been less prominent in thissphere than in the trade arena. While majorpublications such as the Stern report, and theadvocacy of Nobel Prize winners, have beenrecognized worldwide, the Bank’s materials haveneither been as hard-hitting nor attracted thesame level of attention. This may be because theBank’s need to balance different viewpointsamong its shareholders has constrained its roomto maneuver. It may reflect a less-than-strong skillset to produce the necessary leading research, assuggested in IEG’s environment evaluation(2008a). Or perhaps at this stage of the cycle, theBank is only now (appropriately) beginning togear up on the advocacy front.

Whatever the mix of factors in the explanation, itis becoming ever more pressing for the Bank toapproach the advocacy element of climatechange (and its important dimension ondevelopment) more visibly and forcefully. TheBank president’s public statements,4 and theopportunity opened by the new StrategicFramework for Climate Change, provide aplatform upon which to build.

One important area in which the Bank’sadvocacy can be ramped up relates to the adapta-

Some external commentators criticize the Bank Group, argu-ing that while it is doing the right thing in expanding its climate-friendly initiatives—through GEF projects, the creation of thePrototype Carbon Fund, and some analytical research—it is doingthe wrong thing by continuing to finance large energy and infra-structure projects that emit carbon dioxide on a massive scale(Wheeler 2008).

Withdrawing from or reducing investments in traditional proj-ects would be a major challenge. As recently as fiscal 2006, theseenergy investments accounted for 92 percent of the Bank’s total en-ergy portfolio, or nearly $2 billion (World Bank 2007g). And there con-tinues to be equally substantial pipeline demand. It is argued thatthese investments are meeting a real development need, and onecannot expect such a transformational “sea change” to occur

overnight (House of Commons 2008). Moreover, some critics arguethe Bank has little incentive to scale down a line of business that,particularly in IBRD countries, provides a steady flow of profitableprojects in the context of declining lending volumes (Redman 2008).

Can the seeming contradiction be reconciled? There are stepsthe Bank Group can take. One would be to include a shadow pricefor carbon emissions in project appraisals, and to set a timetablefor its introduction as a key factor in decision making by, say, 2010.Another would be to start costing out and regularly reporting onthe subsidy that would be required in any given year to help theseenergy and infrastructure projects become climate-friendly. Thiswould provide the international community with better ongoingknowledge of the price tab for collective action, and how far it isbeing met.

Box 7.2: Clean and Dirty Energy—Can the Bank Do Both?

The extent to which theBank has been a leadinginfluential advocate onclimate change is more

debatable.

tion to climate change. Adaptation is neededbecause of the failure (to date primarily byindustrialized countries) to supply the GPG ofclimate change mitigation. IEG’s evaluationshave argued that the Bank could be strategic andeffective with regard to reducing the vulnerabil-ity of countries and the poor to natural disastersand climate change (IEG 2008a). They alsoconcluded that the Bank Group has not beenable to provide financial resources to assistcountries to address environmental concerns ashigh priorities. The emphasis in the IDA15replenishment on the Bank’s role in assistingeligible countries to adapt to climate change is astep forward in this regard.

The industrialized countries have acknowledged aresponsibility to assist poor countries in adaptingto climate change. What they have not determinedis, first, a basis for calculating this assistance and,second, an arrangement for determining burden-sharing among the industrialized countries. TheWorld Bank can help in calculating the level offinancial support needed and then in implement-ing an adaptation assistance program. Indeed, theconnection between adaptation and developmentis so intimate that the Bank is arguably the onlyglobal institution capable of playing this role.

Another angle on which advocacy could beenhanced relates to research and technologycapability to produce climate-friendly solutionsin many areas—including transport and energyproduction—suited for developing countries.Most attention has focused on the need forresearch and development (R&D) to beundertaken by the industrialized countries, withthe technologies embodying this newknowledge being transferred to developingcountries.5 However, technologies appropriatefor industrialized countries may be less so fordeveloping countries, given the differentcontexts. New technologies may be needed butcurrently this kind of R&D is seen to besomewhat neglected.

Developing countries need to be able todetermine which technologies are necessary,whether the technologies available have to be

adapted to suit local circumstances,and how these technologies shouldbe deployed and used. All of thisrequires a robust technical andscientific capability. This links toanother of the Bank’s strategic priori-ties—engaging with MICs, includingBrazil, China, India, and Russia. This strategy isessential if the Bank is to play a meaningful rolein helping the development transformationneeded to mitigate climate change. TechnologyR&D is also needed for the world’s poorestcountries, but for reasons of scale, a policy thatfocuses exclusively on this latter group ofcountries would be unlikely to properly addressthe mitigation challenge.

Such an advocacy position is consistent with thatput forward by the International Task Force onGlobal Public Goods (ITF 2006), an exercise led byrespected developing country leaders,practitioners, and opinion makers. Itrecommended establishing an Interna-tional Consultative Group on CleanEnergy Research, “which includes bothdeveloped and developing countries, tocollaborate and exchange informationon research and development of more efficient andcleaner technologies.” The recent report by theGlobal Leadership for Climate Action (GLCA 2007)endorses this idea. These proposals draw inspira-tion from the Consultative Group on InternationalAgricultural Research (CGIAR)—a network towhich the World Bank is tightly connected.

Climate change adaptation will also requireinnovation in other areas. A key area for futureinnovation might be called climate-resilientagriculture, a need which CGIAR has recognized.World Bank research has also begun to show theeffects of climate change on agriculture.6

Creating a Unified Response: Learning from Avian FluThe World Bank is seen by some partners to beplaying an important role in the globalresponse to threats emerging fromavian flu and a potential humanpandemic influenza. In terms of global


6 3

It is becoming ever morepressing for the Bank toadvocate for developmentinterests in climatechange more visibly andforcefully.

Other opportunities foradvocacy includepromoting research anddevelopment tailored forthe developing world.

The Bank has advocatedfor global action onavian flu.

6 4


action, the Bank identifies its comparativeadvantages as coordinating global responses andmobilizing finance, and thus far it has been quitean effective advocate along those dimensions.

The Bank’s leadership role in coordinating aglobal response to avian and human influenza—beyond what is typically expected of a financialinstitution—has been seen as critical.7 Its earlyanalysis of the global costs to the spread of avianinfluenza and the threat of a global pandemic(Brahmbhatt 2005) provided an umbrella underwhich the international technical agencies and theUnited Nations System Influenza Coordinator anddonors could operate. This work identified thecontrol of avian and human influenza as a GPG,noted its urgency, and helped bring globalattention to the issue.

Given the urgency of this threat, the Bank helpedquickly to convene major global partners arounda strategy for action. Along with the World HealthOrganization, Food and Agricultural Organization,and World Organization for Animal Health, theBank sponsored the first large-scale internationalmeeting on the topic in Geneva in November2005. The meeting was attended by more than 600experts from over 100 countries, with substantialhigh-level government participation.

According to high-level participants, the Bankmade several contributions at these meetings.First, it convinced international partners of theimportance of integrated country plans that takeinto account animal health, human health, andpandemic preparedness. This was importantbecause the Bank was the only sponsoringorganization with a multisector perspective,without prejudice toward intervention in onesector over another. Second, the Bank argued

that a global program should benefitdeveloping countries, given that theresponse would have to be deliveredon a country level and country-ownedinitiatives have proven to be mosteffective. Third, it helped to keep focuson developing the implementationarrangements for a global program

that could operate effectively, and an agreementof this type was reached during the run-up to andincluding the Geneva conference.

Close collaboration with other major internationalorganizations, with each working to its compara-tive advantage, was essential in the response toavian flu. There have not been any major conflictsbetween the Bank and the relevant UN agencieson avian flu. Some counterparts have expressedconcern in IEG interviews, however, that the Bankmay have gone a bit beyond its comparativeadvantage (to arrange finance for the technicalstrategies of the World Health Organization, Foodand Agricultural Organization, and World Organi-zation for Animal Health) and taken a more wide-ranging role that considered technical agencies asinputs. Others have been critical that the interna-tional community’s collaboration has a weak spotin underestimating the severe global economiccosts—both direct and indirect—that are likely inthe event of a pandemic (Osterholm 2007).Overall, this experience illustrates a wider pointabout the importance of collaboration amonginternational agencies, also pertinent to theBank’s advocacy position on HIV/AIDS, asdescribed in box 7.3.

The Bank also supported efforts to mobilizefinance to address avian and human influenza,particularly with regard to the “nuts and bolts”aspects of the challenge. The Bank’s researchinto the possible global economic impact of apandemic influenza that might be the result ofuncontrolled avian influenza, noted above,found that the ultimate costs to the worldeconomy could be several percentage points ofgross domestic product (Brahmbhatt 2005).More recent research has put the price tag at $2trillion, and economic (but not human) costswould be concentrated among the wealthycountries. This revealed to developed nationsthe potential costs of inaction. The Bank alsoestimated country-by-country financing needs tocontrol avian influenza and delay or reduce theprobability of a pandemic (World Bank 2006b).This created a baseline against which to measurepledges, commitments, and disbursements.

Important ingredientsincluded the Bank’s

robust economic analysis,convening power,

fiduciary reputation, andmultisectoral expertise

and orientation.

Working with the European Commission andother partners, the Bank helped prompt donorsto pledge $1.8 billion (well above the $1.2 billiontarget) at an early 2006 conference in Beijing.Since that time, there has not been an outbreakof a pandemic, media attention has receded, andthe topic may be seen as less urgent. Althoughan additional $900 million was pledged at laterconferences in Bamako and New Delhi, theamount was below the financing needs identifiedon those occasions.

The Bank and United Nations System InfluenzaCoordinator have since worked closely tomonitor commitments and disbursements offunds pledged at the three conferences, and toencourage donors to follow through on theircommitments. Thus far, resources actuallycommitted have not kept up with existing financ-ing gaps (UNSIC and World Bank 2007). Andwhile the international partners agreed thatintegrated country plans were the key platformfrom which to tackle avian flu—and wouldinvolve activities for which very significantresources would be needed—only 40 percent ofcommitted funds have been allocated to theseplans. Furthermore, as shown in figure 7.1, ofthose $700 million of funds committed tocountry programs so far, less than half have beendisbursed, a much lower share than for fundscommitted to international or regional banks.8

This has left a gap in the finances available inmany poorer countries, particularly in Africa, to

tackle this global challenge. Perhaps thisdifficulty could have been ameliorated eitherwith better coordination among donor partnersor with a centralized funding mechanismdedicated to actions in developing countries.

Effective Advocacy Benefits from Voiceand RepresentationTo be an effective advocate, the programs theBank supports must be seen to be legitimate. Butthe voice of developing countries in GPGarenas—not least in the governance of manyglobal programs—is still underrepresented, andthe question remains as to whether the Bankcould have pushed harder to rectify thisimbalance. While the organizational and strategicsettings of such global programs vary, they sharethe common feature that they need to promotethe cooperation and collective action of theirown members to be effective. This puts theemphasis squarely on how topromote the participation of develop-ing countries, which are among theintended beneficiaries of globalprograms and without whose engage-ment the effectiveness of many suchprograms may not be assured (Woods,forthcoming).

There is not, however, a single best-practicegovernance framework for global programsbecause they differ markedly in size and scopeand employ a diverse array of governance


6 5

The Bank’s work on HIV/AIDS stepped up consider-ably in the late 1990s after internal pressure from thehealth sector staff in Bank Regions and the Research De-partment, and externally from newly created interna-tional agencies such as UNAIDS. The Bank thenproactively raised awareness and demand for HIV/AIDSsupport among its staff and client countries.

The collaboration with civil society also ultimately in-fluenced HIV/AIDS advocacy within the Bank on otherimportant global health issues. International NGOs worked

with developing country partners to spearhead a globalcampaign to make existing drugs for HIV/AIDS and otherdiseases affordable to the populations of developing coun-tries. They lobbied for preferential pricing for drugs and drugdonations to developing countries. They supported de-veloping countries in the potential to exercise their rightsunder international trade and intellectual property rightsagreements. While the Bank and World Health Organiza-tion were slow to take a position on these issues, they havecome to support wider access to drugs (IEG 2004b).

Box 7.3: Importance of Collaboration on Advocacy: HIV/AIDS

Giving proper voice andrepresentation in globalprograms improves theirresponsiveness and long-term sustainability.

6 6


models associated with the history and culture ofeach program (see box 7.4 for an example). IEGhas emphasized the need for a global program“to establish legitimacy on a basis other thanshareholder rights,” pointing to the value ofadopting an inclusive and participatorygovernance framework (IEG 2007b). It can takeconsiderable time and effort by partners and theBank to develop good governance arrangementsat the start-up of a global or regional program—but this is time well spent.

The evidence from important programs bearsthis out. In the case of the regionalprograms for Central Asia biodiversityand for the West Africa HIV/AIDSproject, it took about two years toreach agreements among the partici-pants on the appropriate institutional

and governance structure. But that was time thestakeholders considered a good investment, inlight of the inclusiveness of the governancearrangements on which they were able to agree(IEG 2007c). It paid off because the structureswere able to help resolve ongoing differencesand sustain the buy-in of participants. And a farbigger global program—the GEF—took aconsiderable degree of institutional experimen-tation to achieve a governance framework that iscurrently regarded by many as participatory andinclusive, as noted in box 7.4.

Nonetheless, there is often a tension—actual orperceived—between organizational efficiency(which may be fostered by streamlinedgovernance arrangements) and voice. While inmany programs, developing countries arerepresented in the governing bodies, “stake -

Figure 7.1: Less Than Half of Funds Committed to Integrated Country Plans Have Been Disbursed

Source: UNSIC and World Bank 2007.Note: Approximately $650 million of pledged funds have yet to be committed to specific countries, projects, or programs.

707

332

433

206

324

220

317

15746%

66%

73%

76%

0

100

200

300

400

500

600

700

800

Regional body Internationalorganization

Other organization Country or territory

Channel for funding

Am

ount

ple

dged

, com

mitt

ed, o

r dib

urse

d ($

mill

ion)

30

40

50

60

70

80

90

Dis

burs

emen

t rat

e (%

)

Commitments Disbursements Disbursement rateLeft-hand scale: Right-hand scale:

Governancearrangements in several

programs—including theGEF and CGIAR—have

improved over time.


6 7

holder and shareholder influence is not alwaysbalanced,” and “donors and internationalagencies still largely govern the programs” (IEG2004b). In some cases, the increase in represen-tation in the main governing body has createdproblems of organizational efficiency. Butgovernance arrangements need not be static,and the changes in the CGIAR (highlighted inbox 7.5) illustrate how adjustments and compro-mises can be made. Likewise, the Association forthe Development of Education in Africa hasrecently decided to establish an executivecommittee and Cities Alliance has transformed

its former steering committee into an executivebody (IEG 2008b).

The emphasis in the current debate on fosteringpublic goods raises the issue of the extent towhich MICs are appropriately engaged in thegovernance of relevant global programs. TheBank itself has indicated that “a promising aspectof its relationship with MICs lies in its role inmaking connections between these partnercountries and the provision of global publicgoods” (World Bank 2006a). Yet insufficient voicein global program governance is still a concern

The GEF began as a pilot program in 1991, with 16 OECD mem-bers and 9 developing countries pledging resources and repre-senting the sole form of governance. Subsequently, the move froma pilot to a more established program resulted in a governancestructure that melds those of the United Nations and the WorldBank. Governance is based on a constituency system that allowsfor “a relatively small and effective Board,” instead of the “one-country, one-seat, and one-vote system in the United NationsGeneral Assembly,” while maintaining the potential for universalrepresentation (Woods and Lombardi 2006).

The more balanced representation of developing countries—through 18 of the 32 constituencies—is enhanced by the double-majority voting rule. When decisions are not supported by aconsensus, they must garner the formal votes of at least a 60 per-

cent majority of the total number of participants in the GEF, as wellas of a majority representing 60 percent of total contributions tothe facility. Another feature supporting a greater sense of owner-ship of developing countries over the GEF is that the councilcochairperson—in addition to the permanent cochairperson po-sition provided by the chief executive—rotates between developedand developing countries.

The GEF governance also provides an institutionalized settingfor engaging NGOs—a crucial factor in enhancing country-levelcoordination and country ownership. Participation by NGOs, bothlocal and international, does not only take place at the project levelbut also at the policy-making level. GEF provides observer statusfor NGOs at council meetings and holds consultations with themin conjunction with each council meeting.

Box 7.4: Governance as Institutional Experimentation: GEF

CGIAR, founded in 1971 and jointly sponsored by the World Bank,Food and Agriculture Organization, International Fund for Agri-cultural Development, andUnited Nations Development Programme,has grown from an informal small club of donors with a shared con-cern for agricultural research into a partnership that now includes25 developing and 22 industrialized countries, 4 private foundations,13 regional and international organizations, which provide fi-nancing, technical, and strategic support to a network of 15 re-search centers, mostly located in developing countries.

The expansion of the membership, the widening of the researchagenda, and the collective-action problems arising from the grow-

ing number of centers and donors, led CGIAR to establish an exec-utive council, with the aim of simplifying the network’s decision mak-ing. Following the recommendations of the Change Design andManagement Team, in 2001, the council was set up as a stakeholderbody appointed by CGIAR members to follow up on the group’s de-cisions, to ensure alignment and congruence of recommendations,and to act on decisions requiring a more urgent time frame than theschedule of the group’s Annual General Meeting would allow.

The establishment of the council, with formally elected membersaccountable to the particular groups they represent, was neededto increase the legitimacy as well as the efficiency of CGIAR.

Box 7.5: From Shareholder to Stakeholder Model: CGIAR

6 8


for MICs, inhibiting their enthusiasm for andengagement in such programs (IEG 2007a). Eventhe larger MICs are represented in the decision-making bodies of only a small fraction of globalprograms, while, in contrast, several high-income countries are included in the governanceof a much larger number.

The governance arrangements for anyspecific global program—and how itsadvocacy position then emerges overtime—are the results of manyinfluences. The Bank is but one of

those influences and hence responsibility forsuccess or failure cannot rest solely—or perhapseven primarily—at its door. Indeed, the Bank’sposition is complex, and in a program it can playup to 11 different roles including that ofconvener and trust-fund manager.9 But thatmultiplicity of roles creates tensions and, indeed,potential for conflicts of interest (such as thatbetween being a funder and a beneficiary), whichmust be carefully managed.

Moving forward, the growing involvement of theBank in the provision of GPGs is associated withits own governance framework. Goodgovernance is essential to help the Bank shapeits broader strategy for GPGs, define and beselective in its own role in global programs,manage potential or actual conflicts of interestsarising from the various global programs, andenhance its role as an advocate for developing-country needs. Alternatively, there is a risk that“strategies will be hard to define, legitimacy willsuffer, and implementation will lag” (ITF 2006).

The experience of some global programs is quiteencouraging in certain respects. Once all thestakeholders acknowledged the dy namic nature

of a program’s gov ernance, theyembarked on a reform processthrough which considerable changeshave been introduced to thegovernance framework. Far fromimpairing the overall effectiveness ofthe program itself, this has promptedwider engagement of developing

countries, resulting in greater effectiveness for theoverall program. This reinforces the BankDevelopment Committee’s own underscoring of“the im portance of enhancing voice and participa-tion [in the Bank] for all developing and transitioncountries” through an “inclusive and consultative”process among shareholders.10

New Dimensions of Advocacy:Innovation through Financial CapabilitiesThe creativity and innovation of the Bank hasbeen displayed in its work to leverage funds forinnovative approaches to development, especiallyapplied to GPG challenges. The InternationalFinancing Facility for Vaccines and Immunization(IFFIm) provides an example. IFFIm sought tofind a way for the international community toincrease resources for life-saving vaccines andhealth services in developing countries. Itssolution is to borrow from the internationalcapital markets, based on donors’ firm medium-term commitments, and then delivering thatfinance quickly and predictably through theGlobal Alliance for Vaccines and Immunization.

The Bank can hardly be said to have played anexplicit, traditional advocacy role in proposingand developing the concept of IFFIm. Rather, theoriginal idea was formulated in a proposal by theUnited Kingdom’s government, with the supportof other interested parties, including the Bill andMelinda Gates Foundation. But the Bank hasplayed an important role in using its financialcapabilities and reputation in moving thisadvocacy to action. Specifically, at the request ofIFFIm, the Bank has a range of treasury manage-ment and related services, which has gotten IFFImoff the ground. The Bank, in effect, acted as thestart-up treasurer for this global initiative, anddeployed creative approaches with prudentfiduciary standards in the international capitalmarkets. It raised nearly US$1 billion through abond issue in November 2006, almost all theproceeds of which Global Alliance for Vaccinesand Immunization scheduled to spend onimmunization and health programs in poorcountries through 2007. The initiative is still youngand it will be important to assess the extent to

Governancearrangements for global

programs can changeover time.

The Bank’s financialrobustness and

reputation can be put togood use through

innovative financialinstruments that help

move advocacy to action.

which IFFIm delivers additional financing over thelong run.

Another innovative approach is the advancedmarket commitment for vaccines, also aimed attackling communicable diseases on a globalscale. Again, it would not appear that the Bankwas a prime mover in the genesis of the proposal,and hence the potential for it to play a first-moveradvocacy role was not captured. But the Bank’s

technical expertise and reputation for financialsoundness has been called upon by partners tomove this advocacy to action. The Bank hascontributed to an international advisory groupand independent expert committee that hasworked on the technical and structural optionsfor a pilot advance market commitment, andproposed that such a commitment targetpneumococcal diseased as the first problem andconduct subsequent work on malaria.


6 9

Chapter 8

Bengal tiger; photo ©DLILLC/Corbis, reproduced by permission.

7 3

Improving the Bank’s Supportfor Global Public Goods:

Lessons from Experience

The Bank’s country- based model works well in fostering GPGs when thereis one (and preferably more) of the following: (a) a reasonable con-vergence at the country level of national and global interests and time

horizons for taking action; (b) a dedicated cadre of staff with credible tech-nical and managerial competence to advance specific GPG concerns; and/or(c) grant finance to provide the right incentives for action.

So the country model has its place, but thesefeatures are not always present. Where theycannot be established, a very significant shift inthe country- based model is needed for the Bankto find a way to bridge the gap even moreeffectively between global needs and country preferences.

Indeed, looking ahead, the great shared globalchallenges are increasingly those where nationaland global benefits— actual or perceived,immediate or for the next generation— divergesignificantly from each other. For example, theinvestments needed to protect the earth’sclimate and environmental commons varyconsiderably at local, national, and global levels,as do the costs and benefits of such actions.There are lessons emerging from this review thatsuggest effective measures in five areas that mayhelp the Bank upgrade its ability to foster GPGsand to bridge the gap between global needs andcountry concerns more effectively.

First, the Bank can create better incentives todeliver GPGs effectively at the country level. This

will include new approaches to setting budgetsand recognizing the performance of managersand staff. On budget setting, one option is to setaside significant administrative funding at thecorporate level to be allocated— transparently andperhaps competitively—to high- priority GPGwork at the country level. Care would be neededto make sure such funding was used as a genuineaddition by country teams and not simply todisplace other activity. To provide better incentivesto staff, managers at all levels should considerrecognizing country- and global- level work onGPGs in performance management systems.

Second, the Bank can consider clearer organiza-tional arrangements to best select, and indeedlink together, responses at country, regional, andglobal levels. Some Regions may want to have adedicated staff resource advancing work onregional programs (and regional public goods),as has been done in Africa, and perhaps expandtheir purview to cover GPGs as well. But this isnot a one- size- fits- all prescription, and otherRegions may have different arrangementssuitable to their circumstances.

7 4


Third, a more effective approach to the deliveryof the Bank’s global knowledge and capacity tocountry teams working on GPGs would bebeneficial. To this end, the way the Bank can bestdeploy its expertise, particularly that of itsspecialists located at the center of the institutionin the network anchors, should be reviewed.

Fourth, the Bank and its stakeholders couldrenew attention toward ensuring that theperspective of developing countries is connectedeffectively with global responses. The Bankmight be able to use its standing more powerfullyto give greater voice to developing countries inthe governance of significant global programs. Itshould take a more proactive stance in advocat-ing for development interests— and developingcountry partners—in international forums (andagreements) dealing with GPGs. That wouldinclude the Bank continuing to secure additionaldevelopment assistance and to promote thedesign and use of market- based instruments to

help developing countries provide GPGs. TheBank could also explore further ways to stimulate South- South exchange of knowledge—and thedevelopment and application of new technolo-gies designed with and for the South—oncontributing to GPGs, such as climate- friendlyenergy production and use.

Finally, a firmer and more precise justification isneeded for the costs and benefits of actionsbeing proposed for the Bank’s work on foster-ing GPGs, to ensure that such work is financiallyand institutionally sustainable over the longterm. Particularly for global programs, the Bankmust redouble its efforts to be more selective inits engagement and more forthright in exitingthose programs whose benefits and cost- effectiveness are questionable. The Bank shouldalso be insistent about putting in place, andusing, sound results frameworks, underpinnedby realistic and cost- effective monitoring andevaluation systems.

Appendixes

7 7

This appendix presents long-term trends inproject performance, based on IEG projectevaluations using the year 1990 as a starting pointand with data up until June 16, 2008. The analysisof the Bank’s lending effectiveness focuses onIEG’s key performance criteria: the developmentoutcome of projects.

Performance Trends

Outcome Seventy-six percent of projects (by number) inexit fiscal 2007 were rated moderately satisfac-tory or better, as shown in figure A.1, just aboutmeeting the Strategic Compact target of 75percent. But this was a sharp decline of 7percentage points in performance from 83

percent in fiscal 2006. Project performance alsodeclined when weighted by the value ofdisbursements, from 90 percent in fiscal 2006, to83 percent in fiscal 2007.

How much do the results of fiscal 2007 affect thethree-year rolling average of the Bank’s projectperformance? In the medium term, lendingoutcomes were satisfactory—in the three yearsup to end-fiscal 2007, 80 percent of projectssatisfactorily delivered their targeted results, upfrom around 70 percent at the start of thedecade, as shown in figure A.2. But the three-year average, which had increased every yearsince 1999, except for a small dip in fiscal 2003,was flat in 2007 because of the weak fiscal 2007cohort.

APPENDIX A: PROJECT PERFORMANCE RESULTS

Figure A.1: Project Performance Improves in FY06, But FY07’s Cohort Declines


40

50

60

70

80

90

100

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

Exit fiscal year

Shar

e of

pro

ject

s w

ith o

utco

mes

m

oder

atel

y sa

tisfa

ctor

y or

bet

ter (

%)



7 8

What drove the fiscal 2007 decline? Sometimes, adifference in the performance of the Bank’sportfolio from one year to the next can beinfluenced by a change in the composition ofprojects being evaluated—for example, if there isa larger share of projects in challenging sectorsor countries. But the change in portfoliocomposition in fiscal 2007 from the previousthree years—related to Region, sector, instru-ment, lending arm, proportion of loans toconflict-affected countries, and other factors—does not explain the fall in ratings. Indeed, thefiscal 2007 composition of projects was actuallyfavorable along several dimensions.

For instance, even if the fiscal 2007 cohort hadmaintained the same composition of adjustmentand investment loans as in fiscal 2004–06, onewould expect the cohort to have been 75.3percent satisfactory, practically the same resultthat occurred in the event. The Regionalcomposition in fiscal 2007 actually favored theoutcome—had the Regional compositionremained unchanged, the overall fiscal 2007outcome would have actually declined to 73.3percent. The change in proportion of lending toconflict-affected and postconflict countries, or to

IDA versus IBRD lending, did not materially affectthe outcomes. And the change in the sectoralcomposition and the share of projects utilizingspecial lending instruments was only marginallyunfavorable (affecting the total by less than 2percent).

Another possible explanation is that the drop inmeasured project performance is due tomethodological changes in the way in whichprojects were evaluated in fiscal 2007. Somechanges in methods were introduced by IEG andthe Bank, together in fiscal 2007, to strengthenthe robustness of project ratings and to covernew elements of project design. In the near termthey may have introduced some element ofdiscontinuity in the data series between fiscal2007 and earlier years. It is estimated that theinfluence of methodology changes has beensmall—accounting for about 1 percentage pointof the fall.

Besides these small changes in composition,what explains the weak performance of the fiscal2007 cohort? It was driven by projects in health,financial and private sector development, andpublic sector governance, which performed very

Figure A.2: Project Performance Has Improved over the Medium Term


60

70

80

90

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007



Shar

e of

pro

ject

s w

ith o

utco

mes

m

oder

atel

y sa

tisfa

ctor

y or

bet

ter (

%)

A P P E N D I X A : P R O J E C T P E R F O R M A N C E R E S U LT S

7 9

poorly relative to the Bank average, as well asprojects in South Asia and Africa. In fiscal 2007,health, financial and private sector development,education and urban development showedmajor declines, when measured by the numberof moderately satisfactory or better projects, asin South Asia, Africa, and Latin America, whencompared with the prior 3-year period (seefigure A.3). The portfolio of loans to conflict andpostconflict countries also showed signs ofworsening, with 76 percent moderately satisfac-tory or better in fiscal 2007, as compared with 82percent in the previous three years. However,they still perform better than loans to noncon-flict countries, of which only 69 percent weremoderately satisfactory or better.

In fiscal 2007, there was a greater occurrence offive key factors influencing weak outcomes. First,poor or overly complex project design was aproblem in more than half of underperformingloans, a finding also made by the World Bank’sQAG (World Bank 2008a). For instance, twoprojects failed to recognize the importance of anappropriate legal and regulatory framework as aprecondition to a privatization process. Severalhealth projects failed to ensure a heightened focuson those interventions that would yield thegreatest impact, leading for instance, to inadequatetargeting of the poor, absence of a cost-effectivepackage of health services, or inadequate fundingfor behavior change interventions to preventHIV/AIDS transmission among high-risk groups.

Second, overambition was a weakness. Whileproject objectives were almost always relevant, amajority was too far-reaching. Sometimes this wasin terms of assessing political commitment and thefeasibility of certain reforms. Other times it was inassessing government effectiveness and capacity,or in requiring coordination across severalministries or cumbersome financial managementprocedures that were not manageable by theparties involved. IEG’s 2008 evaluation of publicsector reform shows several examples where thesefactors led to unsatisfactory outcomes.

Third, delays in implementation caused difficul-ties, as circumstances changed and projectdesign or implementation could not respond.About one-fifth of underperforming projectssuffered from this problem.

Fourth, a majority of the unsatisfactory projectshad a weak results framework, with poor or nobaseline data, making it difficult to assess theoutcomes of the project; and outcomes wereoften not well linked to inputs and outputs.

Finally, various gaps in the Bank’s own perform-ance contributed to a lack of success. For example,in spite of being flagged by the Bank’s QAG ashaving poor quality at the outset, three projectswere not reassessed or redesigned. The quality ofthe Bank’s supervision was rated as moderatelyunsatisfactory or worse in two-thirds of allunderperforming projects (and many such

Percentage of Satisfactory ProjectsAdjusting FY07 composition to reflect that of Would yield the following FY04–06 along the following dimensions: hypothetical outcomes:

Region 73.3

Adjustment or investment 75.3

Proportion of conflict- or postconflict-affected countries 75.7

Actual FY07 Outcomes 75.8 percent

IDA, IBRD, GEF, or SPF 76.0

Sector 76.7

Lending instrument 77.5Source: World Bank database.Note: Conflict- or postconflict-affected countries are those defined as such by the World Bank for any of the years fiscal 2003–06.

Table A.1: Project Performance in FY07 Was Not Dependent on the Composition of the Cohort

FY07 composition of projects was slightly favorable

FY07 composition of projects had no effect

FY07 composition of projects was slightly unfavorable


8 0

projects were not identified as problems inongoing status reports). Bank overall perform-ance—as distinct from borrower performance orthe effects of uncontrollable events—was rankedmoderately unsatisfactory or worse in two-thirdsof these problem projects, compared with onlyabout one-fifth of the full sample. All of this points

to a challenge in re-emphasizing a proactivequality control in management’s attention toongoing project performance

Sustainability and InstitutionalDevelopment ImpactUnder the new harmonized evaluation criteria

Figure A.3: Projects in Financial and Private Sector Development, Health, and PublicSector Governance Decline in FY07; South Asia and Africa Lag Behind

Source: World Bank database. Note: Some changes in performance may be explained by a small sample size of projects that exited in fiscal 2007, including in the Environment (14 projects)and Poverty Reduction (4 projects) Sector Boards, and the South Asia Region (9 projects).

40

50

60

70

80

90

100

Finan

cial a

nd pr

ivate

secto

r dev

elopm

ent

Health

, nut

rition

,

and p

opula

tion

Ener

gy an

d mini

ng

Publi

c sec

tor g

over

nanc

e

Educ

ation

Urban

deve

lopmen

t

Bank-w

ide

Envir

onmen

t

Agricu

lture

and r

ural

deve

lopmen

t

Econ

omic

polic

y

Wat

er su

pply

and s

anita

tion

Trans

port

Pove

rty re

ducti

on

Social

deve

lopmen

t

Social

prot

ectio

n

Shar

e of

pro

ject

s w

ith o

utco

mes

mod

erat

ely

satis

fact

ory

or b

ette

r (w

eigh

ted

by d

isbu

rsem

ent,

%)

Shar

e of

pro

ject

s w

ith o

utco

mes

mod

erat

ely

satis

fact

ory

or b

ette

r (w

eigh

ted

by d

isbu

rsem

ent,

%)

FY04–06 FY07

FY04–06 FY07

Sector Board

a. Outcome by sector board

b. Outcome by Region

40

50

60

70

80

90

100

South

Asia

Sub-Saharan

Africa

Middle East

and North

Africa

Bank-wideLatin America

and the

Caribbean

Europe and

Central

Asia

East Asia

and

Pacific

Region


8 1

for project evaluations, approved in October2005, projects would no longer be rated for theirsustainability and institutional developmentimpact (World Bank 2005b). Thus, only 122 ofthe 249 exit fiscal 2006 projects that wereevaluated received a rating for their sustainabil-ity, and 132 for institutional development.

For this partial fiscal 2006 cohort, 83 percent ofprojects are rated likely or highly likely to beresilient to future risks, maintaining the fiscal 2005rating for sustainability. Fifty-five percent of thepartial fiscal 2006 cohort is rated substantial orhigh on institutional development impact, a smalldecline from fiscal 2005.1 However, sustainabilityratings represented a 10 percentage pointincrease, and institutional development impactratings a 3 percentage point increase, for the fiscal2006 (partial) cohort over the depressed fiscal2003 cohort ratings (figure A.4).

Regional PerformanceFigure A.5 presents the percentage of satisfac-tory project outcomes, weighted by disburse-ment, for the fiscal 2003–07 cohort, as comparedwith the fiscal 1998–2002 cohort. The East Asiaand Pacific and Europe and Central Asia Regions

are the top performers for the fiscal 2003–07cohort, exceeding the Bank average of 83percent. The Latin America and CaribbeanRegion is the only one that declined in perform-ance for the fiscal 2003–07 cohort. In spite ofimproving the most in performance for fiscal2003–07, Africa continues to lag behind all otherregions. Its performance declined in fiscal 2006to 69 percent of outcomes moderately satisfac-tory or better, weighted by disbursement, and to64 percent in fiscal 2007, an overall reduction of10 percentage points from fiscal 2005.

Sectoral PerformanceCompared with the fiscal 1998–2002 cohort, theoutcome performance weighted by disburse-ment for the fiscal 2003–07 cohort improved in 8of 12 sector boards.2 Figure A.6 presents thesector boards’ (and Bank-wide) outcomeperformance in order of their fiscal 2003–07performance. The biggest improvements inoutcome ratings were in water supply and sanita-tion, economic policy, energy and mining, andthe social protection sector. The largest declinesin performance were in health, nutrition, andpopulation (which, along with economic policyand the environment, were significantly below

Figure A.4: Long-term Trends in Sustainability and Institutional Development

Source: World Bank database.Note: Data for 2006 are partial (shown by dashed line).

20

40

60

80

100

1990 1993 1996 1999 2002 2005

Exit fiscal year

a. b.

Shar

e of

pro

ject

s w

ith s

usta

inab

ility

likel

y or

bet

ter (

%)

20

40

60

80

100

1990 1993 1996 1999 2002 2005

Exit fiscal year

Shar

e of

pro

ject

s w

ith in

stitu

tiona

l de

velo

pmen

t im

pact

sub

stan

tial o

r bet

ter (

%)



8 2

the Bank-wide average for the fiscal 2003–07period) and in public sector governance.

Lending Instrument PerformanceThe most recent data show a small dip in theshare of development policy lending projectsrated moderately satisfactory or better, both bynumber of projects and value of disbursements(see figure A.7) as measured on a three-yearrolling average basis. Investment lending, too,has seen a decline in outcome performance.

New Lending Instruments IEG has evaluated 241 operations employing theBank’s four new lending instruments—Adaptable Program Loans, Development PolicyLoans, Learning and Innovation Loans, andPoverty Reduction Support Credits. More than90 percent of these operations exited the Bank’sportfolio during the fiscal 2003–07 period,amounting to $10.5 billion in disbursements, and17 percent of all the projects that exited duringthat period. PRSCs have pulled up the perform-ance of the group, exceeding the Bank averagefor the past five years of 77 percent and 82percent, respectively, of moderately satisfactoryor better projects and disbursements (figureA.8). Their outcome ratings have increasedsteadily since fiscal 2003 with 100 percent of

PRSCs being ranked satisfactory in fiscal 2006.Some of the project ratings are still provisional,however, with full ratings available for only 25percent of PRSCs. IEG’s ongoing evaluation ofPRSCs will shed more light on the performanceof this instrument. However, Learning andInnovation Loans are performing below the Bankaverage, and Adaptable Program Loan ratingshave shown a decline in the two years since fiscal2005 in performance, but an increase as aproportion of the portfolio in fiscal 2007. Twenty-one Development Policy Loans, the newest Bankinstrument, have been rated so far, and haveperformed just above the Bank average,weighted by disbursement, over fiscal 2003–07.

Bank-Managed Special Programs IEG has evaluated 87 operations financed underfour Bank-managed special programs that haveexited the Bank’s portfolio since fiscal 2003(table A.2). Seventy-seven percent of the specialoperations had satisfactory outcomes, a declineof 8 percent as compared with operations exitingbetween fiscal 1998 and fiscal 2002. This declinein performance is mainly due to the lowersatisfactory outcomes of Special FinancingGrants or multidonor-sponsored trust funds thathave assisted five conflict-affected or postcon-flict countries in the past decade.3 The resources

Figure A.5: Africa Improves Relative to FY98–02, But Continues to Lag


40% 50% 60% 70% 80% 90% 100%

Africa

Latin America andthe Caribbean

South Asia

Middle East andNorth Africa

Bank-wide



a. Share of projects with outcomes moderately satisfactory or better(weighted by disbursement)

Regi

on

FY98–02 FY03–07

b. Distribution of disbursements, by Region,FY98–07

Middle Eastand North Africa

5%

South Asia15%

Africa14%


24%


16%

Latin Americaand the

Caribbean26%


8 3

40% 50% 60% 70% 80% 90% 100%

IMPROVERS

Sect

or b

oard

40% 50% 60% 70% 80% 90% 100%

DECLINERS

a. Share of projects with outcomes moderately satisfactory or better(weighted by disbursement)

FY98–02 FY03–07

Environment

Economic policy

Energy and mining

Bank-wide

Agriculture and rural development

Water supply and sanitation

Financial and private sector development

Transport

Social protection

Education

Urban development

Public sector governance

Health, nutrition, and population

b. Disbursements by Sector Board, FY98–07

Agriculture andrural development

10.8%

Education8.5%

Energy and mining11.5%

Environment2.0%

Economic policy10.9%

Financialand private

sectordevelopment

13.5%

Globalinformation

and technology0.8%

Health,nutrition, and

population6.6%

Povertyreduction

1.2%

Public sectorgovernance

7.5%

Social development0.4%

Social protection6.2%

Transport12.0%

Urbandevelopment

4.0%

Water supplyand

sanitation4.0%

Figure A.6: Trends in Sectoral Performance



8 4

devoted to Special Financing Grants exitingbetween fiscal 2003 and fiscal 2007 were $493million, or 0.5 percent of the Bank’s overallproject portfolio during this period.4

Figure A.7: Long-Term Trends in Development Policy Lending and Investment Lending


20

40

60

80

100

1992 1995 1998 2001 2004 2007


a. Development policy lending b. Investment lending

Shar

e of

pro

ject

s w

ith o

utco

mes

mod

erat

ely

satis

fact

ory

or b

ette

r (%

)

Shar

e of

pro

ject

s w

ith o

utco

mes

mod

erat

ely

satis

fact

ory

or b

ette

r (%

)20

40

60

80

100

1992 1995 1998 2001 2004 2007


By project Weighted by disbursement

Figure A.8: Outcome Performance of New Lending Instruments


40

60

80

100

Learning andInnovation Loans

Bank-wide AdaptableProgram Loans

DevelopmentPolicy Loans

Poverty ReductionSupport Credit

New operations (FY03–07)

Shar

e of

pro

ject

s w

ith o

utco

mes

mod

erat

ely

satis

fact

ory

or b

ette

r



8 5

Exit FY98–02 Exit FY03–07Disbursements Disbursements

Number of (% of Outcome: Number of (% of Outcome:evaluated World Bank % evaluated World Bank %

Special program type projects portfolio) satisfactorya projects portfolio) satisfactorya

Global Environment Facility 39 0.1 79.5 39 0.46 84.6

Montreal Protocol Fund 4 0.0 100.0 5 0.05 100.0

Rainforest Initiative 1 0.0 100.0 2 0.00 100.0

Special Financing Grants 17 0.2 94.1 41 0.54 65.9

All Special Programs 61 0.3 85.2 87 1.0 77.0Source: World Bank database.

a. Projects whose outcomes are rated moderately satisfactory, satisfactory, or highly satisfactory are here referred to as “satisfactory.”

Table A.2: Bank-Managed Special Programs Are Performing on Par with Bank Average


8 6

Exit FY98–02

ID impact: Number Outcome: Sustainability: %

of Share % % likely substantial projects (%) satisfactorya or better or better

Sector BoardAgriculture and Rural Development 229 16 66.4 53.7 37.8Economic Policy 74 5 56.8 65.2 29.7Education 133 10 81.2 66.9 40.6Energy and Mining 153 11 68.0 59.4 43.3Environment 72 5 71.8 73.9 50.7Financial and Private Sector Development 159 11 64.9 66.4 38.8Gender and Development 0 na na naGlobal Information/Communications Technology 16 1 100.0 100.0 68.8Health, Nutrition, and Population 108 8 65.7 61.0 36.9Poverty Reduction 0 na na naPublic Sector Governance 98 7 84.4 79.8 54.2Social Development 1 0 100.0 100.0 100.0Social Protection 84 6 83.1 58.3 45.8Transport 137 10 88.0 78.4 64.7Urban Development 67 5 71.2 50.0 36.4Water Supply and Sanitation 67 5 65.2 46.9 33.3Overall Result 1,398 100 72.5 63.8 43.4Lending Instrument TypeDev Pol Lending 177 13 75.1 73.8 43.2Investment 1220 87 72.1 62.3 43.4Not Assigned 1 0 100.0 100.0 0.0Overall Result 1,398 100 72.5 63.8 43.4NetworkFinancial and Private Sector Development 172 12 66.5 64.7 38.8Human Development 325 23 76.6 62.8 40.8Poverty Reduction & Economic Management 172 12 72.4 73.4 43.5Sustainable Development 729 52 72.1 61.8 45.5Overall Result 1,398 100 72.5 63.8 43.4RegionAfrica 385 28 59.5 47.1 32.4East Asia and Pacific 206 15 76.2 64.4 45.5Europe and Central Asia 274 20 82.8 80.2 52.4Latin America and Caribbean 277 20 76.4 69.4 50.8Middle East and North Africa 105 8 73.8 64.9 39.8Not Assigned 0 0 na na naSouth Asia 151 11 74.2 64.8 41.1Overall Result 1,398 100 72.5 63.8 43.4Agreement TypeNot Assigned 1 0 100.0 100.0 100.0Global Environmental Facility 39 3 79.5 63.2 51.3IBRD 671 48 75.9 71.6 48.7IDA 665 48 67.9 55.5 37.4Montreal Protocol Fund 4 0 100.0 100.0 75.0Rainforest Initiative 1 0 100.0 100.0 0.0Special Financing Grants 17 1 94.1 71.4 41.2Overall Result 1,398 100 72.5 63.8 43.4Source: World Bank database.

na = not applicable.a. Projects whose outcomes are rated moderately satisfactory, satisfactory, or highly satisfactory are here referred to as “satisfactory.”

Table A.3: Outcome, Sustainability, and Institutional Development (ID) Impact by Various Dimensions, by Project, FY98–07


8 7

Exit FY03–07 Exit FY98–07

ID impact: ID impact: Number Outcome: Sustainability: % Number Outcome: Sustainability: %

of Share % % likely substantial of Share % % likely substantial projects (%) satisfactorya or better or better projects (%) satisfactorya or better or better

188 14 83.1 78.0 60.1 417 15 73.9 62.5 46.576 6 81.1 80.0 41.0 150 5 68.9 71.4 34.8

136 10 81.3 85.4 59.8 269 10 81.3 74.6 48.783 6 77.5 80.0 67.9 236 9 71.3 64.8 49.884 6 75.3 73.1 52.5 156 6 73.7 73.6 51.5

124 9 72.1 80.7 56.1 283 10 68.1 71.9 45.61 0 100.0 100.0 100.0 1 0 100.0 100.0 100.0

11 1 100.0 100.0 50.0 27 1 100.0 100.0 62.5109 8 61.5 72.6 59.2 217 8 63.6 65.9 46.421 2 81.0 90.9 53.8 21 1 81.0 90.9 53.8

124 9 64.5 74.4 41.9 222 8 73.3 77.1 48.123 2 71.4 66.7 42.9 24 1 72.7 69.2 46.796 7 83.0 74.5 50.0 180 7 83.1 65.0 47.7

119 9 92.4 86.5 62.7 256 9 90.0 81.4 63.978 6 79.2 77.0 44.8 145 5 75.5 63.2 40.671 5 84.3 78.8 59.6 138 5 75.0 61.2 45.5

1,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1

202 15 79.8 88.1 52.6 379 14 77.6 80.3 47.61,140 85 77.7 77.1 55.1 2,360 86 74.8 68.2 48.2

2 0 100.0 100.0 100.0 3 0 100.0 100.0 50.01,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1

124 9 72.1 80.7 56.1 296 11 68.9 70.5 45.2341 25 75.4 78.4 56.8 666 24 76.0 69.4 47.7222 17 71.9 77.7 42.9 394 14 72.1 75.5 43.2657 49 82.7 79.1 57.6 1,386 51 77.1 68.5 50.4

1,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1

316 24 67.1 66.3 47.7 701 26 62.9 54.2 38.3202 15 83.5 80.9 64.4 408 15 79.9 71.2 53.6282 21 83.2 89.6 62.9 556 20 83.0 84.4 57.0318 24 82.6 81.3 56.7 595 22 79.7 74.6 53.4111 8 73.6 70.9 33.7 216 8 73.7 67.6 37.0

1 0 100.0 na na 1 0 100.0 na na114 8.5 78.1 83.8 54.8 265 9.7 75.8 71.3 46.0

1,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1

0 0 na na na 1 0 100.0 100.0 100.039 3 84.6 76.2 53.8 78 3 82.1 67.8 52.3

603 45 82.3 83.9 57.6 1,274 46 78.9 76.6 52.4654 49 74.4 75.7 53.2 1,319 48 71.1 63.6 44.0

5 0 100.0 100.0 50.0 9 0 100.0 100.0 66.72 0 100.0 100.0 100.0 3 0 100.0 100.0 50.0

41 3.1 65.9 48.2 37.5 58 2.12 74.1 56.1 38.81,344 100 78.1 78.9 54.8 2,742 100 75.2 69.9 48.1


8 8

Exit FY98–02

ID impact: Disburse- Outcome: Sustainability: %

ments Share % % likely substantial ($m) (%) satisfactorya or better or better

Sector BoardAgriculture and Rural Development 11,669 11 77.8 67.3 45.4Economic Policy 11,497 11 51.9 77.5 37.7Education 8,125 8 86.6 76.4 44.7Energy and Mining 15,234 14 68.6 62.3 44.0Environment 1,517 1 58.8 84.9 37.4Financial and Private Sector Development 18,019 17 84.1 85.9 51.3Gender and Development 0 0 na na naGlobal Information/Communications Technology 1,297 1 100.0 100.0 61.5Health, Nutrition, and Population 5,622 5 77.9 74.5 45.8Poverty Reduction 0 0 na na naPublic Sector Governance 7,638 7 86.8 91.1 52.0Social Development 5 0 100.0 100.0 100.0Social Protection 6,289 6 84.5 77.1 40.9Transport 11,908 11 90.8 85.1 66.0Urban Development 3,868 4 84.6 56.7 40.8Water Supply and Sanitation 4,342 4 62.3 40.4 26.1Overall Result 107,031 100 77.5 74.9 46.8Lending Instrument TypeDev Pol Lending 38,247 36 77.6 84.8 46.0Investment 68,783 64 77.5 69.7 47.3Not Assigned 0 0 na na naOverall Result 107,031 100 77.5 74.9 46.8NetworkFinancial and Private Sector Development 19,658 18 84.2 82.3 48.9Human Development 20,036 19 83.5 76.1 43.8Poverty Reduction & Economic Management 19,135 18 65.8 83.0 43.4Sustainable Development 48,202 45 77.0 68.7 48.8Overall Result 107,031 100 77.5 74.9 46.8

RegionAfrica 13,143 12 60.0 48.5 28.7East Asia and Pacific 28,502 27 80.6 81.4 50.3Europe and Central Asia 19,187 18 75.3 82.6 51.4Latin America and Caribbean 25,040 23 86.2 80.1 54.5Middle East and North Africa 5,544 5 79.9 67.3 49.4Not Assigned 0 0 na na naSouth Asia 15,616 15 74.7 70.8 37.5Overall Result 107,031 100 77.5 74.9 46.8Agreement TypeNot Assigned 32 0 100.0 100.0 100.0Global Environment Facility 138 0 80.0 69.6 55.6IBRD 79,302 74 78.5 78.5 50.0IDA 27,332 26 74.6 64.9 38.1Montreal Protocol Fund 9 0 100.0 100.0 81.4Rainforest Initiative 0 0 na na naSpecial Financing Grants 219 0 96.8 69.2 33.4Overall Result 107,031 100 77.5 74.9 46.8Source: World Bank database.

na = not applicable.a. Projects whose outcomes are rated moderately satisfactory, satisfactory, or highly satisfactory are here referred to as “satisfactory.”

Table A.4: Outcome, Sustainability, and Institutional Development (ID) Impact by Various Dimensions, by Disbursement, FY98–07


8 9

Exit FY03–07 Exit FY98–07

ID impact: ID impact: Disburse- Outcome: Sustainability: % Disburse- Outcome: Sustainability: %

ments Share % % likely substantial ments Share % % likely substantial ($m) (%) satisfactorya or better or better ($m) (%) satisfactorya or better or better

9,800 11 85.2 82.2 70.3 21,469 11 81.2 73.0 55.110,081 11 65.9 55.6 27.3 21,578 11 58.4 67.6 33.08,798 10 84.7 95.5 70.2 16,922 9 85.6 85.0 56.37,635 8 80.4 76.3 74.4 22,869 12 72.5 65.7 51.42,393 3 66.4 76.9 40.4 3,911 2 63.0 80.3 39.18,847 10 92.1 95.6 81.0 26,866 14 86.8 89.0 61.4

3 0 100.0 100.0 100.0 3 0 100.0 100.0 100.0214 0 100.0 100.0 33.2 1,511 1 100.0 100.0 58.0

7,426 8 69.0 78.2 61.1 13,048 7 72.8 76.1 52.62,287 3 84.8 92.5 28.9 2,287 1 84.8 92.5 28.97,337 8 81.8 88.2 51.8 14,975 8 84.3 89.8 51.9

847 1 65.4 94.2 44.2 852 0 65.6 94.2 44.66,025 7 93.3 85.2 72.2 12,313 6 88.8 79.5 51.9

11,906 13 93.3 90.4 60.9 23,814 12 92.0 87.3 63.94,156 5 81.7 80.6 53.0 8,024 4 83.1 68.6 46.93,598 4 91.3 78.8 66.1 7,941 4 75.4 55.8 42.3

91,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8

30,125 33 82.2 81.9 53.1 68,372 34 79.6 83.7 48.961,228 67 83.1 82.9 62.7 130,011 66 80.1 74.9 53.4

0 0 na na na 0 0 na na na91,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8

8,847 10 92.1 95.6 81.0 28,505 14 86.7 86.2 59.122,248 24 81.8 87.8 67.9 42,284 21 82.6 80.9 54.019,708 22 74.0 70.7 35.7 38,843 20 70.0 77.2 39.840,549 44 85.6 82.8 62.8 88,751 45 80.9 74.1 54.291,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8

14,572 16 73.0 74.8 52.0 27,714 14 66.9 60.1 39.218,261 20 92.4 90.2 74.7 46,763 24 85.2 84.2 59.012,053 13 87.4 92.9 70.7 31,240 16 80.0 86.4 58.228,920 32 80.8 73.7 56.4 53,960 27 83.3 77.3 55.44,032 4 82.1 82.3 31.2 9,575 5 80.8 72.5 43.1

0 0 na na na 0 0 na na na13,491 15 80.8 87.8 48.5 29,107 15 77.5 77.5 41.891,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8

0 0 na na na 32 0 100.0 100.0 100.0422 0 90.9 91.8 52.3 560 0 88.2 84.1 53.3

58,907 64 84.7 82.9 62.4 138,209 70 81.1 80.1 54.631,487 34 79.4 82.1 53.8 58,820 30 77.1 72.5 45.3

43 0 100.0 100.0 4.0 52 0 100.0 100.0 21.20 0 na na na 0 0 na na na

493 1 66.9 44.3 25.8 712 0 76.1 52.8 28.591,353 100 82.8 82.5 59.3 198,383 100 80.0 77.9 51.8

9 1

The Bank has in place the policies andprocedures to monitor, self- evaluate, andconduct independent evaluation of its opera -tions. The Regions conduct the monitoring and self- evaluating of their lending, analytical, andadvisory activities, and country assistanceprograms under guidelines issued by theOperations Policy and Country Services Vice- Presidency. Monitoring and self- evaluation ofoperations Bank- wide is carried out by theQuality Assurance Group, Operations Policy andCountry Services, and other units. The Bank isalso strengthening its monitoring and self- evaluation activities financed through trust fundsand its involvement in global programs and partnerships.

Independent evaluation is carried out by IEG.The Director- General of Evaluation reportsdirectly to the Executive Board, which approvesthe Director- General’s mandate and IEG’s termsof reference. IEG validates the Bank’s self- evaluations, verifies their results, and assesses

the relevance, efficacy, and efficiency of Bankoperational activities and processes.

The Bank formally revised some of its policies tobetter monitor and evaluate the results of Bankoperations. The Bank issued a new OP/BP 8.60,Development Policy Lending, in 2004, requiringoutcomes and measurable indicators for M&E inpolicy loans/credits as well as investment projects.BP 2.11, Country Assistance Strategies, wasupdated in 2005, mainstreaming the results- basedcountry assistance strategy approach. In 2007, theBank revised OP 13.60, which had focused ondissemination and utilization of IEG findings,making it a policy on M&E. The new OP 13.60,Monitoring and Evaluation, includes a section onBank monitoring and self- evaluation and anotheron independent evaluation, outlining IEG’s role.OP/BP 14.40, Trust Funds, also provides guidanceon monitoring and evaluation for activities fallingunder that rubric, and has been updated tointroduce enhancements, including greatercoverage of periodic independent evaluation.

APPENDIX B: MONITORING AND EVALUATION OVERVIEW


9 2

MONITORING

Defining outcomes and Tracking and reporting Type of intervention or instrument monitoring indicators on implementation progress

Loans and credits Results frameworks in Project Appraisal Documents Implementation Status Report

(for investment loan/credits) and Program

Documents (for development policy loans/credits)

Analytical and advisory activities Concept paper specifies development objectives Activity update summary

and tracking indicators

Country programs Results framework in Country Assistance Strategy CAS Progress Report

Trust- funded activities Initiating Brief for Trust Fund includes program Grant reporting and monitoring system

objectives and accompanying performance for child trust fund/grant

indicators

Source: IEG.

Table B.1: Summary of Bank Monitoring and Evaluation

A P P E N D I X B : M O N I TO R I N G A N D E VA L U AT I O N OV E R V I E W

9 3

EVALUATION

Independent evaluation by IEG Self- evaluation Individual activity Aggregate level

• Ex post: Implementation Completion ICR reviews (all); Project Performance ARDE

and Results Reports Assessment Report (selected)

• Ex ante: QAG Quality- at- Entry Assessments.

• Bank- wide: QAG Annual Report on Portfolio

Performance

• Ex post: Activity completion summary Reviewed as part of sector/thematic and

• Country: QAG analytic and advisory activity country assistance evaluations

assessments

• Bank- wide: QAG Annual Report on Portfolio

Performance

• Ex post: CASCR • CASCR reviews; ARDE, sector/thematic

• Bank- wide: CAS retrospectives. • Country Assistance Evaluations evaluations

• Less than $1 million: grant report and monitoring

completion report

• More than $1 million: Implementation Completion

and Results Report

9 5

OverviewIEG contributes to the Bank’s effectiveness bysupporting the Board’s oversight function andpromoting learning within the Bank and thedevelopment community. One tool to gatherfeedback on IEG’s work is a client survey, albeitany such survey has limitations, with respect tosample size and scale of response. The mostrecent survey found that among respondents,both inside and outside the Bank, there wassome increase in awareness of IEG’s products.Compared with earlier years, an increasingproportion of respondents reported generalsatisfaction with the quality and timeliness of IEGproducts. But they gave IEG lower marks fordepth of analysis and incorporation of allavailable information. The use of evaluations foroversight among World Bank Executive Directorswho responded to the survey remains high. Useof evaluation findings in the design of newoperations by respondents who are operationalstaff was higher in 2007 than a year earlier, butremains low.

The lower ratings for operational use andsatisfaction with depth of analysis point to thechallenge of finding the right breadth and lengthfor IEG products. IEG’s work program for fiscal2009–11, discussed with CODE in May 2008,assembles a good mix of activities, includingincreased emphasis on newer products such asCASCR reviews, clustered Project PerformanceAssessment Reports, and quicker turnaroundreports on selected major issues of topicalinterest.

Beyond client feedback, the Management ActionRecord (MAR) allows IEG to track its recommenda-

tions from sector, thematic, and corporate evalua-tions. Bank management is accountable to theBoard for follow-up. The MAR tracks the level ofadoption and the status of individual recommenda-tions. It presents management’s ratings on thesetwo indicators and IEG’s assessment of the same.

The 2008 MAR shows a continuing high level ofagreement by the Bank with IEG’s recommenda-tions. Some 96 percent of IEG proposals made inthe last three years’ evaluations have beenaccepted by the Bank. In terms of the Bank thenadopting those recommendations and puttingthem into practice, some 95 percent have beenadopted with medium, substantial, or highratings, a level slightly above the previous year.However, the share of recommendations adoptedwith substantial or high ratings was 42 percent,which is below the level of previous years.

Going forward, the challenge is for IEG tocontinue producing high-quality evaluationswith sensible and practical recommendations,retain the high level of agreement on thoserecommendations, and for the Bank to lift itsintensity of adoption and implementation.

Improving IEG’s EffectivenessThis appendix provides an overview of IEG’s rolein improving the Bank’s development effective-ness. It includes a results framework that linksIEG’s mandate and objectives to its operations.Within this framework, the appendix updatesIEG’s efforts to increase its evaluation focus onresults. The appendix includes findings from theannual client survey, an update on the communi-cations and outreach strategy, and the status ofthe MAR.

APPENDIX C: IEG’S SELF-EVALUATION: IMPROVING EFFECTIVENESS


9 6

IEG’s Results FrameworkIEG has three functions in improving develop-ment effectiveness. First, it provides accountabil-ity by independently reporting on the resultsachieved by Bank operations. Second, it distillsthe Bank’s operational experience intoknowledge of what works and why, and makesthat knowledge widely available to Bank person-nel and the global development community.Third, it supports client governments with itstechnical knowledge on M&E through its evalua-tion capacity development activities.

IEG has a mandate to assess “whether the WorldBank Group’s programs and activities are produc-ing the expected results, including global,regional, and other programs in which the WorldBank Group is a participant.”1 By reporting theresults of its evaluations to the Board of Directorsand communicating the findings and lessonsfrom its work to Bank management, operationalstaff, and the development community, IEGexpects to increase the Bank’s effectiveness andinfluence Bank and client country decisions onpolicies, programs, and procedures. While fullyindependent,2 IEG is placed within the Bank tomaximize its operational effectiveness and toprovide operational staff and strategic decisionmakers with knowledge that helps them workmore effectively.

IEG’s results chain is summarized in figure C.1, andrelated to measures of performance that havebeen collected for this report. IEG’s outputs arethe findings, lessons, and recommendations fromits evaluations, and evaluation capacity support inclient countries. Dissemination efforts are theintermediate step between outputs and outcomes.For IEG’s accountability function, the intermediateoutcomes of these outputs are the use of evalua-tions by the Board to fulfill its oversight function,and the incorporation of IEG recommendations inBank internal policies and procedures. For IEG’sfunction as a knowledge provider, the intermedi-ate outcome of IEG’s outputs is the influence anduse of these outputs to improve the Bank’s policyadvice and program and project design. It alsoincludes use of these outputs by external partnersto improve their development work.

The final outcome for IEG outputs is the use ofIEG knowledge about what works and why,leading to improved effectiveness of Bankoperations, and development assistance ingeneral, in reducing poverty. For example, IEG’sevaluation of regional programs has contributedsignificantly to the recognition of the importanceof regional approaches for the delivery of globalpublic goods (GPGs) by the Bank.3 Outside theBank, the Bill and Melinda Gates Foundationused IEG’s recent evaluation of World BankAssistance to Agriculture in Sub-Saharan Africa(2007d) as one of the main sources for thedevelopment of its agriculture strategy. TheBank’s operational staff, management, Board, andexternal clients use IEG’s outputs to strengthenactions taken in client countries. Measures thatattribute achievement of the final outcome toIEG, however, are difficult to construct.

Underscoring the importance of the above resultsframework, the 2006 AROE recommended that“To further strengthen IEG’s contribution to theworkings of the results agenda in the World BankGroup, IEG should continue to follow its ownresults framework and monitor it through theAROE. Its focus on the usefulness of evaluationfindings for its core audiences should beenhanced: for the Board through oversight, formanagement through the incorporation ofrecommendations into Bank policies and strate-gies, for Bank staff through the use of evalua-tion findings for policy advice to country partnersand in project design, for external partnersthrough the use of evaluation findings to improvetheir programs and policies, and for thecountries more broadly. In playing this role, IEGshould specifically:

• Improve the timeliness of its evaluations,• Strengthen the operational relevance of the

findings, and• Increase access to and exchange of the les-

sons.”

This appendix describes the key links in IEG’sresults chain, giving special attention to therecommendations from last year, and highlightsrecent developments in the results of IEG’s

A P P E N D I X C : I E G ’ S S E L F- E VA L U AT I O N : I M P R OV I N G E F F E C T I V E N E S S

9 7

approach. First, it looks at the activities IEG hasundertaken to improve its output and the achieve-ment of outcomes; and second, it analyzes thechange in indicators that measure the quality ofoutputs and achievement of IEG’s outcomes.

IEG’s Outputs: Increasing Relevance andthe Focus on LearningAt the product level, IEG has continued its line ofquickly produced papers that address immediateneeds for evaluative findings and lessons ofexperience, in the form of notes, presentations,or briefing papers. IEG’s short papers, in 2007,on governance received especially positive feed -back. IEG’s work plan for fiscal 2009–11 contin-ues the shift to fewer and more influentialevaluations, including some shorter, topicalproducts, and an increasing number of evalua-tions conducted jointly across IEG (World Bank,International Finance Corporation, and Multilat-eral Investment Guarantee Association).

At the level of country evaluation, the adoptionof CASCR reviews4 represents a major shift in theBank’s evaluation system, because it ensuressystematic evaluation coverage of CAS results. InCASCR reviews, IEG reviews the relevance of theCAS to the country’s development priorities,implementation of the country program,achievement of CAS objectives, and the quality ofthe CASCR itself. IEG then gives a rating for CASimplementation and Bank performance.5 As ofMarch 2008, IEG has reviewed 59 CASCRs, 17 ofwhich were completed in fiscal 2007. Goingforward, IEG is planning to review up to 30CASCRs each year. Besides its rating function, theCASCR review is intended as a learning tool thatdistills lessons learned from implementation ofthe preceding CAS for incorporation into thedesign and implementation of the following CAS.To make its country reviews more useful andrelevant, IEG is also planning an enhancedCASCR review, which would combine the timely

Figure C.1: IEG Results Chain

Source: IEG.

Government institutions, policies,programs made more effective

IEG-WB budgetary resources (staff, consultants, and so forth)

—Findings, lessons, and recommendations

Accountability through use byBoard and senior management

Use of evaluation findings byBoard members to provideoversight functionEvaluation recommendations

operations and policy

Learning from evaluation byoperational staff

staff to improve policy advice toclients and program and projectdesignUse of evaluation findings by

programs and policies

References to IEG in themedia

evaluations (client survey)

Management ActionRecord

Directors and their advisorsClient survey on use and

evaluations

evaluations in Project Appraisal

Data sources

Dissemination of IEG products

Client survey on evaluationquality satisfaction

Inputs

Outputs

Final outcome

Intermediateoutcomes

Documents and sector strategies


9 8

delivery of IEG’s results, when the new CAS isdiscussed at the Board, with the insights of acountry mission which would allow a much morethorough assessment of CAS achievements andchallenges by IEG.

It has to be noted, however, that the existingprocess limits the effectiveness of the CASCRreview as a learning tool. IEG receives the finalCASCR for validation when the preparation ofthe new CAS is nearly complete. Therefore, itmay be too late for the country team to incorpo-rate much, if any, of IEG’s findings directly intothe CAS document. But lessons learned on theimplementation of the CAS should be consid-ered by the country team as it moves toimplement the new CAS. This issue has not beenaddressed so far. In the case of enhanced CASCRreviews, it would be even more importantbecause these would need more preparationtime.

To create incentives within the Bank for goodperformance in design, implementation, M&E,and development effectiveness, IEG has beengiving annual Good Practice Awards tooperations that exemplify strong performance inthese areas.6 In addition to providing incentivesfor high performance, the awards heighten theprofile of operations that offer examples of goodpractices. To foster the exchange of lessonsbetween operational staff and to highlight the

importance of mutual learning, IEG added alearning event to the 2007 and 2008 awardsceremonies (see box C.1).

Deepening Strategic Partnerships IEG’s success in achieving greater focus onresults and learning requires strategic partner-ships. For example, to capitalize on the impactevaluation expertise available in other groups inthe Bank, IEG has been collaborating with thethematic group for poverty analysis, monitoring,impact evaluation, and the DevelopmentEconomics Department. Together, the threegroups organized a two-day event in January2008, Making Smart Policy: Using ImpactEvaluation for Policy Making.

Evaluation capacity development. IEG is strength-ening its support of results-oriented monitoringand evaluation systems and capacities in clientcountries. As a result of its 2004 self-evaluation ofevaluation capacity development, IEG refocusedits high-intensity support on a few targeted,demonstration countries while maintaining low-intensity support to a much broader range ofcountries. In recent years, these efforts wereparticularly evident for Colombia, Mexico, andChina, and this support will now be extended toother regions. Also, in June 2008, IEG hosted theannual International Program in DevelopmentEvaluation Training (IPDET), in collaborationwith Carleton University, for the eighth time. This

Since 2004, the World Bank’s Independent EvaluationGroup (IEG) has selected a small number of winners,from among operations evaluated in the previous fiscalyear, for its Annual Good Practice Awards. The main ob-jective is to highlight exemplary design, implementation,and self-evaluation in Bank projects and country pro-grams, and to create incentives among Bank staff forgreater learning from evaluation, to enhance developmenteffectiveness.

To facilitate replication of these good practices inother operations, in 2007 and 2008, IEG invited Good

Practice Award winners to share their experiences andlessons learned with other Bank staff in a learning event,Secrets of Successful Operations. In these engagingdiscussions, Bank experts—including award winnersfrom the Europe and Central Asia and the Latin Americaand Caribbean Regions, who led the way in using eval-uation effectively—helped identify the challenges theyfaced in their work and what they did to address them.The experts offered lessons for future operations and par-ticipants were encouraged to identify specific lessons toapply to their own work.

Box C.1: IEG Good Practice Awards: Secrets of Successful Operations—An IEG Learning Event


9 9

four-week course draws broad interest fromevaluation professionals and policy makersworldwide. In addition, IEG has a formal partner-ship with the Chinese government to helpdevelop a regional center for developmentevaluation training in Shanghai, and a regionalversion of the development evaluation training isoffered there on a twice-yearly basis. Since itsinception in 2001, more than two thousandpractitioners have participated in the Interna-tional Program in Development EvaluationTraining. Evaluation capacity developmentproducts on M&E methodology, impact evalua-tion, and influential evaluations consistentlydraw strong interest among practitioners.

Harmonizing development evaluation. As the largestand oldest of the evaluation units in the multilat-eral development banks, IEG has taken a leader-ship role in harmonization efforts among theinternational evaluation community. IEG hasactively promoted harmonization of develop-ment evaluation methods through the multilat-eral development banks’ Evaluation CooperationGroup, the Development Assistance CommitteeEvaluation Network, and the United NationsEvaluation Group. The Evaluation CooperationGroup has developed good practice standards forevaluation of both public and private sectordevelopment work. Member institutions havebeen benchmarked against these standards.Beyond that, IEG led the work on a joint Evalua-tion Cooperation Group paper synthesizingfindings and lessons on the linkages betweeninfrastructure and environment operations. IEGtook the lead in creating the Network ofNetworks on Impact Evaluation, with memberssuch as the Development Assistance CommitteeEvaluation Network, the Evaluation CooperationGroup, and the United Nations Evaluation Group,as well as developing countries and NGOs. IEG isproviding the secretariat for this group, which isdeveloping good practice standards for develop-ment impact evaluation and promoting impactevaluations of development work. IEG also ledthe Development Assistance Committee Evalua-tion Network’s work on developing good practicestandards and guidelines for evaluating globaland regional programs.

Quality and Relevance of IEG Outputs:Results of the 2007 Client SurveyIEG measures the quality and relevance of itsoutputs as part of its annual internal and externalclient surveys. The 2007 surveys asked targetaudiences for their views of IEG productsprepared in 2006 and 2007, including 5 sectorand thematic studies, 5 corporate reports, 6CAEs, about 40 Project Performance AssessmentReports, and an evaluation capacity developmentreport. IEG surveyed a targeted sample of 4,218internal clients, consisting of Bank staff andExecutive Directors and their advisors. Theresponse rate was 24 percent, as compared with22 percent last year.7 The survey of externalclients on published evaluations approached asample of 6,238 individuals and achieved aresponse rate of 19 percent. Given theseresponse rates, it has to be noted that the surveyresults are indicative for respondents, but cannotbe generalized to the surveyed population.

Readership and awareness. Sixty-one percent ofBank staff who responded to the most recentannual client survey (2007) were aware of theevaluation for which they were surveyed. This isabove the 56 percent in 2006 and continues thetrend of increasing awareness of IEG productsamong Bank staff respondents, only 39 percentof whom indicated awareness in 2004. Amongrespondents to the external survey, 75 percentwere aware of the evaluation for which they weresurveyed, compared with 76 percent in 2006.

The quality of IEG evaluations. As shown in figureC.2, Bank staff were asked to rate their satisfactionwith IEG’s evaluation for 10 attributes of qualityon a six-point scale. Bank staff respondents weremore satisfied with the quality of IEG products in2007 than in 2006, across all attributes of quality.Bank staff respondents continue to report highestsatisfaction with the relevance of IEG’s evaluationsto their work, with 80 percent of respondentsrating it 4 or higher, the highest rating in the pastfour years. Bank staff respondents continue to beleast satisfied with the incorporation of allavailable information and the depth of analysis,with 63 percent and 66 percent of respondentsexpressing their satisfaction, respectively.


1 0 0

4.3 4.2 4.13.9 4.0

3.8 3.7 3.7 3.6 3.6

4.4 4.4 4.34.2 4.1 4.0 4.0 4.0 3.9 3.8

1

2

3

4

5

6

Relevance to

your work

Ease of

understandingaConcise

presentation

of conclusionsa

Usefulness

of

recommen-

dationsa

Timeliness Unbiased and

objective

analysisa

Strong link

between

conclusions

and evidencea

Transparency

and clarity

of the

methodologya

Depth of

analysisaIncorporation

of all

available

informationa

IEG

Clie

nt S

urve

y ra

ting

scal

e

2006 2007

Figure C.2: Bank Staff Satisfaction with Quality of IEG Evaluations Improved in 2007

Source: IEG data.Note: Bars show the means on a six-point scale, where 1 = highly unsatisfied, and 6 = highly satisfied. The sample size for 2006 is 292, for 2007 is 417.a. Difference is significant at 90 percent confidence level.

Due to changes in the survey methodology, alonger time series is only possible for a fewquality attributes. Figure C.3 shows that IEGcontinuously improved the timeliness andrelevance of its products to operational staff.Regarding views on the depth of analysis in IEG

evaluations, the decline in ratings stopped in2007, but there is room for improvement.

Executive Directors and their advisors whoresponded to the survey are very satisfied withthe quality of IEG products. As figure C.4 shows,

Figure C.3: Satisfaction of Bank Staff with Quality of IEG Evaluations, 2004–07

Source: IEG data.

7380

6771

76

56

67

80

66

76

0

10

20

30

40

50

60

70

80

90

Relevance of lessons to Bank'swork

Presentation of conclusions Link between conclusionsand evidence

2004 (rated good or excellent) 2005 (rated good or excellent)

2006 (rated 4, 5, or 6, out of 6) 2007 (rated 4, 5, or 6, out of 6)

Perc

ent


1 0 1

in 2007, Executive Directors and advisor respon-dents were most satisfied with IEG’s relevance totheir work (with a mean of 4.95, 88 percent ofrespondents rating it 4 or higher). Externalrespondents’ satisfaction with IEG’s quality wasvery high, with 89 percent of respondents ratingall attributes of IEG’s evaluations with a score of4 or higher.

In response to the question of whether evalua-tions influenced their understanding of thesubject area, 74 percent of Bank staff respondentsrated it 4 or higher, a marked increase over 58percent in 2006.8 Ninety-four percent ofExecutive Directors and their advisors whoresponded rated this aspect at 4 or higher,compared with 90 percent in 2006. Amongexternal respondents, 83 percent rated theinfluence of evaluations on their understandingof the subject area at 4 or higher, compared with81 percent in 2006. At the more practical level ofinfluencing Bank strategies and the design ofresults frameworks, Bank staff respondents ratedIEG’s influence a bit lower. On average, 58percent rated these options at 4 or higher, with

means of 3.85 for IEG’s influence on how outputsare linked to outcomes at the country level, and3.96 for IEG’s influence on sector strategies.

Use of evaluations for assessing the Bank’s sectorand country strategies is high among ExecutiveDirectors and their advisors, with more than 90percent rating it at 4 or higher, and policies andprocedures, with 87 percent assigning ratings of4 or higher. Executive Directors and theiradvisors who responded use evaluations less forassessing Bank projects (73 percent rating it 4 orhigher). Overall, Executive Director and advisorrespondents made more use of IEG evaluationsin 2007 than in 2006. The same holds true forBank staff respondents, whose self-reported usehas increased by 19 percent, on average, for alltypes of usage since 2006.9 Bank staff respon-dents use evaluations mostly for commenting onthe work of others, making a case for a particularcourse of action, and providing advice to clients,and less for modifying strategies or operations,or designing new projects or programs.However, in 2007, 45 percent of respondentsrated use for modifying ongoing operations at 4

4.9

4.04.3 4.2

4.8 4.8 4.8 4.7 4.74.5

4.9

4.4 4.3 4.3

4.84.6 4.5 4.5 4.4 4.3

1

2

3

4

5

6

Relevance to

your workTimelinessa

IEG

Clie

nt S

urve

y ra

ting

scal

e

Improvement over previous year Decline from previous year

Ease of

understanding

Concise

presentation

of conclusions

Usefulness

of

recommen-

dations

Unbiased and

objective

analysis

Strong link

between

conclusions

and evidencea

Transparency

and clarity

of the

methodology

Depth of

analysis

Incorporation

of all

available

information

2006 2007

Figure C.4: Satisfaction of Executive Directors with Quality of IEG Evaluations

Source: IEG data.Note: Bars show means on a six-point scale, where 1 = highly unsatisfied, and 6 = highly satisfied. Sample size for 2006 is 50, for 2007 is 58.a. Difference is significant at 90 percent confidence level.


1 0 2

or higher, compared with only 27 percent in2006. External clients use IEG evaluations mostlyfor research (76 percent), while 71 percent use itto refocus ongoing strategies or programs.Although use of evaluations was higher in 2007than in 2006, respondents continued to point toobstacles to usefulness, such as the lack ofspecificity of recommendations to suggest howto put them into practice, and lack of reflectionon the country context where the Bank operates,which would make findings relevant to Bankoperations on the ground.

When asked about how to improve the evaluationfor which they had been surveyed, 59 percent ofstaff respondents recommended that IEG makeits findings more operational, compared with 57percent in 2006. Broadening consultation withBank staff was the second recommendation madeby 47 percent of respondents (48 percent in2006). This was followed by improving depth ofanalysis (2007: 41 percent; 2006: 45 percent) andbroadening external consultation (2007: 40percent; 2006: 35 percent). Executive Directorsand their advisor respondents made similarsuggestions, but 54 percent emphasized theimportance of broadening external consultationmore than consultation with Bank staff, whichonly 27 percent of respondents chose as arecommendation to IEG. External respondents’main recommendation was to broaden externalconsultation (59 percent), followed by moreoperational findings (51 percent) and obtainingmore evidence (47 percent).

In 2007, the survey also included questionsabout the timeliness and relevance of four IEGquick products from 2006 and 2007. These weregenerally well received. Eighty-four percent ofBank staff and Executive Directors and advisorrespondents rated them at 4 or higher on timeli-ness, and 80 percent rated them at 4 or higher onrelevance. However, awareness of these noteswas low, with senior management respondents(42 percent indicated no awareness with any ofthe four notes) and staff, whose awareness wasbelow 40 percent for all but one note. ExecutiveDirectors and advisors who responded weregenerally well aware of these notes.

Communicating Knowledge from IEG EvaluationsEffective communication of IEG’s knowledge toBank staff, governments, other donors, and theinternational community is a crucial linkbetween evaluations and outcomes. Over the lasttwo years, IEG has undertaken extensive effortsto improve its outreach to Bank staff and thewider development community. After a series ofpilot initiatives, IEG mainstreamed several newapproaches on media outreach, e-mail market-ing, Web promotions and an improved HelpDesk, and successfully increased awarenessamong target audiences.

Awareness of evaluations, as reported by Bankstaff who responded to the 2007 client survey,rose from 39 percent in 2004, to 61 percent in2007. The number of follow-up inquiries to IEG’sHelp Desk increased tenfold during 2005 and hasstayed at the same high level since, with about2,400 inquiries each year. Help Desk inquiries areconcerned with evaluation methodology, adviceon M&E systems, and IEG’s product portfolio,suggesting that IEG’s outreach campaigns aretriggering follow-up questions among keyaudience segments.

Over the past two years, IEG organized 11 mediaoutreach campaigns to coincide with the releaseof its reports. These campaigns have producedextensive media coverage and generated anestimated 200 articles and reports in audiovisualmedia over the last two years.10 The growingnumber of references in the media to IEG evalua-tions indicates strong interest in the Bank’sperformance on key development initiatives.

IEG’s communication and outreach can bestrengthened by providing more summaries ofIEG findings. In the client surveys, about two-thirds of all client groups made this recommen-dation in both 2006 and 2007. Bank staff (39percent of respondents) and Executive Directorsand advisors (53 percent of respondents)continue to ask for more online accessibility toIEG content. Among external respondents, 60percent asked for more training/educationmaterial. Forty percent of Executive Directors and


1 0 3

advisor respondents made this recommendationas well. Going forward, IEG will work on its Website, focusing on the development of new searchtools that will make its content more accessible.

IEG Intermediate Outcomes: EvaluationRecommendations Incorporated intoBank Operations and PolicyIEG influences the Bank’s effectiveness throughrecommendations to management as part ofsector, thematic, and corporate evaluations, aswell as Country Assistance Evaluations. Bankmanagement is accountable to the Board forfollow-up. One of the intermediate outcomes forIEG is the extent to which management incorpo-rates IEG recommendations and findings inpolicy advice, program design, and projectdesign (see box C.2).

The Management Action Record allows IEG totrack its recommendations and to monitor thedegree to which they have been adopted bymanagement. The MAR tracks two indicators: thelevel of adoption11 and the status of individualrecommendations.12 The MAR presents manage-ment’s ratings on these two indicators and IEG’sassessment of the same. The MAR includes IEGrecommendations from the previous threecalendar years.

The MAR for 2008 tracks management’s progresson 57 recommendations. These include 22 newrecommendations from the six IEG studies (exclud-ing CAEs) presented to the Board in calendar year2007, and 35 recommendations carried forwardfrom calendar years 2005 and 2006.

The 2008 MAR shows a continuing high level ofagreement from the Bank with IEG’s recommen-dations. Of the 57 recommendations in the 2008MAR, 96 percent (55 recommendations) havebeen accepted by Bank management,13 similar tothe rate of acceptance in earlier years (95 percentin MAR 2007 and MAR 2006).

In the 2008 MAR, as shown in figure C.5, IEG ratedadoption medium or better for 95 percent of 52recommendations. This compares with 85 per -cent (47 recommendations) in the 2007 MAR and94 percent (72 recommendations) in the 2006MAR. The share of recommendations adoptedwith substantial or high ratings was 42 percent inthe 2008 MAR, down from 60 percent in the 2007MAR. Therefore, the distribution of adoptionratings among the top three categories (medium,substantial, and high) has shifted toward medium.

The Bank’s own assessment shows this shift andit is confirmed by IEG. In this regard, it is

• The recently announced credit line to deal with catastrophes,such as earthquakes and hurricanes in MICs, directly reflectsIEG’s recommendation that a new mechanism be establishedto meet urgent needs in the early days of a disaster response.

• The Bank’s recently promulgated strategy ”Strengtheningthe World Bank’s Rapid Response and Long-Term Engagementin Fragile States“ responds directly to IEG’s recommendationthat internal Bank support for fragile states—including staffnumbers, staff skills, guidance, and organizational incen-tives—needs to be strengthened.

• Prompted by IEG’s primary education study and the accom-panying volume of science-based sectoral knowledge, Bankprojects more often now aim to measure learning outcomes,and, in particular, the reading fluency acquired in the early

grades. A major donor-financed workshop in March 2008used findings and benchmarks identified by IEG.

• IEG’s review of lines of credit led to a Bank-wide effort to iden-tify and review their quality, sharing of experience amongdonors, and an Operations Policy and Country Services ini-tiative to require early identification of lines of credit.

• IEG’s evaluation of private sector development in the powersector contributed to a reassessment of the Bank’s approachto infrastructure.

• The evaluation of social development fed directly into thepreparation of the Bank’s Social Development strategy.

• The global programs review resulted in greater scrutiny andstreamlining of the Bank’s approach to and governance ofglobal programs.

Box C.2: Selected Impacts of IEG Evaluations


1 0 4

noteworthy that agreement on adoption ratingsbetween IEG and management was higher in2008 than in previous years. IEG and manage-ment agreed on the rating on level of adoptionfor 65 percent of recommendations (36 out of55).14 And the disconnect between high andsubstantial adoption ratings by IEG and manage-ment was 18 percent (IEG rated 42 percent ofrecommendations high or substantial; manage-ment 60 percent) in 2008, compared with 28percent in 2007 and 21 percent in 2006.

Going forward, the challenge is for IEG tocontinue producing high-quality evaluationswith sensible and practical recommendations,retain the high level of agreement on thoserecommendations, and for the Bank to lift up itsintensity of adoption and implementation.

Since 2006, recommendations that are olderthan three years are being retired from the MAR,subject to review by CODE and the Board.

IEG Intermediate Outcomes: Use ofEvaluation Findings by Bank Staff at theOperational LevelIEG’s performance on this intermediate outcomecan be measured by whether Bank staff use IEGfindings to improve Bank policy advice and

program and project designs. The 2006 AROEfound that IEG provides high-level knowledgethat is useful for assessing programs, givingadvice to clients, and making comments, butneeds to focus on influencing ongoing and futureoperations. Focus groups conducted for thatreport showed that Task Team Leaders are leastlikely to report that they incorporate evaluationfindings into planning and design. Theiroperational context requires information on howto conduct monitoring, establish indicators, andprepare projects for evaluation. Respondents inmanagement positions reported greater use andusefulness of IEG products. CODE members relyheavily on IEG’s reports, advice, and recommen-dations. These findings were also borne out inthe 2007 client survey in which 72 percent ofBank staff respondents reported using IEGevaluations for making comments and givingadvice to others. However, only 54 percent of staffreport using evaluations for designing newoperations, and 45 percent report using evalua-tions for modifying ongoing operations.Executive Directors and their advisors makeheavy use of evaluations (90 percent indicatingsubstantial use) for their oversight function ofsector and country strategies, and Bank policies,but less so for project-level oversight (76 percentof respondents indicating substantial use).

Figure C.5: Adoption of IEG Recommendations Has Declined, but Agreement withManagement Is Higher in 2008

Source: IEG data.

22% 16% 11%

44%44%

31%

27%25%

53%

6% 5%15%

0

20

40

60

80

100

2006 2007 2008

Management Action Record

Shar

e of

ratin

g ca

tego

ry (%

)

High Substantial Medium Negligible

Sample size = 77 Sample size = 55 Sample size = 55


1 0 5

To improve IEG’s impact on Bank operations, IEGwill need to continue giving greater emphasis toapplied learning and real-time use of evaluationfindings, to improve Bank performance. To meetthis challenge, IEG needs to define its stance onengagement and learning with guidelines andfunding for its staff. The more that operationalstaff are involved up front in the evaluation designand during the evaluation, the more likely the

evaluation will deliver operationally relevantresults and be geared toward learning. At the sametime, close engagement challenges the evaluator’sindependence and might distract from thenecessary accountability perspective. Finding theright balance and providing the proper incentivesto IEG staff for engagement with operational staffduring the evaluation and afterward for learningrequires clear directions from IEG leadership.

1 0 7

Both the demand and supply features of GPGscan provide an obstacle to their provision.Individual countries “demand” GPGs only to theextent that this demand serves a national interest.Moreover, the features of different GPGs implylarge variations in the manner through which aGPG should be supplied. This section provides aframework for considering how the Bank and theinternational community should arrange for themost efficient approach toward providing GPGs.

From the Demand SideThe extent of divergence between national andglobal benefit (or cost) of a GPG varies across aspectrum, and so the extent to which a particulargood is truly a GPG is not always easy to assessprecisely. Clean air provides a good example of aGPG. In contrast, some facets of communicablediseases are primarily national in scope, notwith-standing the fact that there may be other reasonswhy responses are supported by global partnerships.

This distinction is noteworthy. A country’sperception of the divergence between nationaland global benefits and costs of GPGs influenceswhether a country “demands” a GPG and,therefore, whether there is action on the countrylevel, the extent to which such engagementoccurs, and the types of instruments that areused. For the Bank, the level of overlap betweennational and global net benefits can be onedeterminant of the type of role it can play infostering GPGs, and the extent to which strategicintent is translated into action on the ground.

At one extreme, when the national benefits of theGPG coincide in large part with the benefitsaccruing to the global community (depicted infigure D.1), the Bank’s country-based model can

work well to encourage countries to integrate theGPG into their national programs because thecountry’s interests are already closely alignedwith the worldwide interests. For example, acountry will likely be more interested in control-ling HIV/AIDS within its borders if it has a highHIV prevalence rate (and hence the health andsocial consequences are already felt at thenational level). In such cases, the Bank shouldhave influential national counterparts who arereceptive to its support. In such circumstances,traditional Bank instruments such as IBRD andIDA lending can often be used.

At the other extreme, when the net benefits ofthe GPG are large at the dispersed global levelbut only minimally appropriable at the level of anindividual nation state—as is the case for climatechange—the Bank’s country-based model maynot be as useful because of that very divergence(depicted in figure D.2). In such instances, the

APPENDIX D: FEATURES OF GLOBAL PUBLIC GOODS

The Bank’s country-based model can

work well, for example, in the case

of an HIV/AIDS control program.

National net benefit Global net benefit

Figure D.1: Country Perception of Net Benefits WhenNational and Global Interests Coincide

Source: IEG.


1 0 8

Bank may be more successful in fostering GPGsby leveraging its convening power and providingconstructive advocacy at the global level.

Between the two extremes, when there is somelevel of convergence in national and global netbenefit, the Bank has a chance to promptnational action, but has to “push hard at thedoor.” In fact there are several levers the Bank

can use to, as it were, “push the circles together”and create a coincidence of national and globalinterest. Those levers include: (i) financinginstruments; (ii) global programs, and (iii)advocacy—including convening power andsupport for international frameworks.

From the Supply SideGPGs also have different features in the way theyhave to be supplied and this has implications forthe way in which the Bank (and other bodies) dealwith them. As shown in table D.1, some publicgoods are supplied through aggregate efforts,others by single best efforts. Some depend on theweakest link, others require coordination. Thesedistinctions are important because they revealdifferent incentive challenges. While the desire ofsome countries to coordinate may be strong, thesupply of public goods requiring an aggregateeffort is usually prone to free riding. Supply ofpublic goods requiring a single best effort can beeasier to achieve than supply of a weakest-linkpublic good, which can be undermined by a singlefailed state.

The Bank’s convening

power and global-level

advocacy may be more

effective, for example,

in addressing climate

change.

National net benefit Global net benefit

Figure D.2: Country Perception of Net Benefits WhenNational and Global Interests Diverge

Source: IEG.

A P P E N D I X D : F E AT U R E S O F G L O B A L P U B L I C G O O D S

1 0 9

Table D.1: Bank Uses Different Vehicles to Support the Provision of Different Types of GPGs

Source: IEG.

Single best effortThe action of a single actor

or country (or small group of

actors) provides a GPG.

For example, researchers in the United States developed two polio vaccines in the 1950s that were available to many other countries.

TradeThe Bank's research and ad-

vocacy on trade provides pres-

sure and evidence to encour-

age a prodevelopment Doha

round of WTO negotiations.

VaccinesThe Global Alliance for

Vaccines and Immunizations

conducts research to provide

new vaccines for communi-

cable diseases that could then

be shared with all countries.

Weakest linkInaction or a weak effort

by a single country leads

to undersupply of a GPG.

For example, the inability of a country to eradicate a communicable disease forces other countries to vaccinate against it.

Crises and systemic risk The Financial Sector

Assessment Program helps

countries identify

vulnerabilities in their

financial systems and

determine needed reforms in

order to avoid financial crises

and potential global contagion.

Aggregate effort A critical mass of countries

each contributes individually

to a GPG

For example, countries reduce or eliminate their production of ozone-depleting substances to preserve the global ozone layer.

Climate change The Carbon Fund mitigates

against climate change through

market-based mechanisms.

The Global Environment

Facility (GEF) fights climate

change and protects

biodiversity through grant

finance.

Coordination Countries act in harmony

to jointly achieve an

agreed upon GPG.

For example, countries agree to standards of measurement that facilitate international trade and cross-country comparisons

Avian influenzaThe Bank conducted research

that outlined the costs and

benefits of avian influenza

control, which served as an

umbrella for international

action. Advocacy andresearch

Country-basedinterventions

Supply-type of Global Public Good

Wor

ld B

ank-

supp

orte

d ac

tivity

1 1 1

IntroductionThis year’s Annual Review of DevelopmentEffectiveness (ARDE) tracks World Bank perform-ance and examines a particular thematic topic, theBank’s work in fostering global public goods(GPGs). Management welcomes the new two- partformat of the ARDE and considers the thematictopic timely and highly relevant. In Part I, theBank’s Independent Evaluation Group (IEG)reports on recent trends in the outcomes of Bankprojects and country programs and analyzesprogress in the quality and coverage of monitor-ing and evaluation (M&E) systems employed bythe Bank. In Part II, IEG reviews the Bank’s experi-ence with (a) country- based support for clientcontributions to the supply of GPGs and (b) itsadvocacy for action on GPGs. This appendixprovides brief responses to IEG’s findings andsuggestions in Part I and Part II, respectively.

Tracking Bank PerformanceManagement values the review of developmentoutcomes and the Bank’s M&E practice. IEG’sfeedback fosters learning from experience and isa factor in the medium- term improvement ofBank performance documented in the ARDE.Management agrees that the weakening ofdevelopment outcomes in projects that exitedthe portfolio in fiscal 2007 warrants attention andoutlines its actions below. Management alsoagrees with most of the suggestions to furtherimprove the quality and coverage of M&E inprojects, country programs, and global programsand partnerships (GPPs).

Project OutcomesThe ARDE confirms that the ratings for projectoutcomes have significantly improved over themedium term and have exceeded the Bank’s

performance benchmarks in each of the threeyears to end- fiscal 2007. Improvements havebeen particularly impressive in the Africa Regionand in the water supply and sanitation sector.The report goes on to highlight and discuss twosigns of weaknesses in the data for fiscal 2007,including a jump in the so- called disconnect—that is, the difference between the outcomeratings provided by Bank staff in the finalImplementation Status and Results reports(ISRs) for ongoing projects and IEG’s ratings inits reviews of the Bank’s ImplementationCompletion and Results reports (ICRs)—and adecline in the share of projects with satisfactoryoutcomes, from a high of 83 percent in fiscal2006 to 76 percent in fiscal 2007.

Actions in response to warning signals. Managementconcurs that a correct rating of likely outcomes inongoing projects is necessary for its ability to taketimely remedial action in problem projects, andthat the fall in IEG exit ratings and the increaseddisconnect in fiscal 2007 must be recognized as awarning sign. This parallels findings under ourmore detailed review of IDA controls as well asQAG reports and the India detailed implementa-tion review. Management is taking on these issuesin the context of investment lending reform,notably changes in our supervision practices. AsIEG notes, the more problematic areas are thenoninfrastructure sectors and low- incomecountries (LICs), especially fragile states. Manage-ment is considering a more customized approachfor implementation to reflect the complexities ofthese operational situations, moving beyond thetraditional notion of supervision and incorporat-ing the possibility of implementation supportdirected at capacity building. However, manage-ment is not waiting for the introduction of these

APPENDIX E: MANAGEMENT COMMENTS


1 1 2

reforms. Each Region has conducted its ownreview of ratings in ongoing operations. In theSouth Asia, East Asia and the Pacific, and LatinAmerica and Caribbean Regions, Regional vicepresidents have taken the lead in reviewing ISRs,with instructions for action by task teams. TheAfrica Region has adopted specific measures(involving country management, sector manage-ment, and the quality and knowledge servicesunits) to reduce the disconnect, ensure realism inthe ratings of ongoing operations, and strengthenaccountability for project quality. In addition, theupcoming Development Policy LendingRetrospective will review the Bank’s ratingpractice for development policy loan operationsand, as necessary, will recommend appropriateactions to ensure that management receivestimely information on weak performance in such operations.

Decline in fiscal 2007 project outcome ratings andincrease in disconnect— some observations.Effective support for members’ developmentefforts requires the Bank to take risks;consequently, not all the operations it supportswill achieve the desired outcomes. Against thisbackground, the share of Bank projects with asatisfactory development outcome in recentyears has risen to a reasonable level— a level atwhich year- to- year variations are to be expected.However, management agrees that vigilance iswarranted to ensure that the fiscal 2007 drop inratings signifies a variation around that trend andnot the beginning of a decline. IEG is to becommended for examining the possible impactof sample composition bias and of changes in theevaluation methodology on the fiscal 2007outcomes, and for recalling key factors aboutwhich the Bank must always be vigilant, includ-ing poor or overly complex project design,articulation of overly ambitious expectedoutcomes, implementation delays, andweaknesses in project supervision and staffperformance. In addition, management believesthe following considerations to be pertinent.

• Reduced outcome ratings for developmentpolicy loans contribute to the reported over-all drop in fiscal 2007 ratings. That reduction

likely reflects in part the temporary impact ofa recent decision to report on programmaticdevelopment policy loans only at the end of theseries. As a result, for a few years, one- off de-velopment policy loans will make up a largershare of the sample for which outcome ratingsare available. With programmatic developmentpolicy loans rated better than one- off devel-opment policy loans, on average, this shift incomposition has an adverse, but likely transi-tory, effect on the average rating for the sample.

• Vigilance about “overly complex” project de-signs must not discourage staff from respond-ing to client demand for sophisticated— and attimes necessarily complex— products, partic-ularly in middle- income countries (MICs). Therisk of not achieving satisfactory outcomes maybe higher in such circumstances, but takingthat risk is necessary for the Bank’s continuedrelevance to its members. The issue is not somuch reducing complexity as it is having clearobjectives and effective systems for trackingoperational performance.

• In addressing task team performance, it is im-portant to take into account the challengesthat result from the Bank’s commitment to in-creased harmonization and collaboration withdevelopment partners, particularly in LICs.Staff must devote more time to coordinationand are generally more dependent on the ac-tions of others for achieving results, notablypending greater harmonization around agreedresults frameworks.

• The jump in the disconnect can in part betraced to the introduction of a new evaluationmethodology in fiscal 2007, specifically to dif-ferent paces of implementing the agreed ratingsystem and rating criteria between IEG and thelarger and more layered Bank operational com-plex. The ARDE estimates this effect as ac-counting for one percentage point of the jumpin the disconnect, but it may have been higher.As more ISRs and ICRs apply the methodologyalready used in IEG’s reviews, the divergencein ratings can be expected to decline.

Country Program OutcomesThe ARDE reports the average developmentoutcome ratings for some two decades of

A P P E N D I X E : M A N AG E M E N T C O M M E N T S

1 1 3

country programs on the basis of IEG’s CountryAssistance Evaluations (CAEs), and, for morerecent country programs (starting in fiscal 1999),on the basis of IEG’s reviews of CountryAssistance Strategy Completion Reports(CASCRs). The ARDE observes that in both sets,average outcomes for MIC programs surpassthose for LIC programs by a considerable margin,and that higher outcome ratings for the morerecent country programs— those for whichCASCRs are available— reflect improvements inMIC programs only. It also compares averagecountry program outcomes with average projectoutcomes and suggests that the lower rating forcountry program outcomes may indicate a failureof such programs to exploit synergies betweenthe Bank’s development services. Managementoffers two comments.

• The gap between the outcomes of MIC and LICprograms is a matter of concern to manage-ment. However, averages over long durationsdo not shed light on the underlying factors.Management encourages IEG to analyzechanges in outcomes over time for subgroupsof countries, with the analysis taking accountof major changes, such as the introduction ofCAS results frameworks and changes in evalu-ation methodology. Moreover, managementwants to note that it has not yet introduced astandard rating methodology for CASCRs, andthe methodology used by IEG self- initiatedratings of CASCRs has been evolving and hasnot yet been finalized.

• IEG’s suggestion that lower outcome ratings forcountry programs than for projects indicatethat country programs fail to exploit synergiesamong Bank development services is not sub-stantiated in the report. The report acknowl-edges some of the factors that, inmanagement’s view, deprive the comparison ofinformative value (for example, different per-formance standards). Management notes thatperformance in country programs predatingthe introduction of CAS results frameworks infiscal 2005 is measured against objectives thatdo not clearly distinguish between countryoutcomes and CAS program outcomes. As pro-gram outcomes come to reflect more realisti-

cally the Bank’s role in supporting a country’sdevelopment strategy, outcome ratings tend toimprove. It is also important to recognize thatCAS program outcomes can change duringthe CAS period as the Bank responds flexiblyto changing client demand for its services.

Monitoring and EvaluationThe ARDE recognizes that as part of the resultsagenda the Bank has put in place strong policiesand procedures for the monitoring and evalua-tion (M&E) of project and country programoutcomes. For the projects and countryprograms reviewed, however, it reports a lowquality of M&E and results frameworks. Manage-ment agrees with IEG’s recommendations forimproving the quality and use of M&E, includingincreased focus on providing good baselineinformation for project outcomes, articulatingmore clearly the link between project outputsand targeted outcomes, simplifying the CASresults frameworks, and using them not only forevaluation but also for program management.Ongoing efforts in the Regions to improve M&Eare indeed focusing on these aspects, as ismanagement in its reviews of proposed develop-ment policy loan operations and CASs. Manage-ment believes that the ARDE could have givenmore recognition to initiatives such as RegionalM&E support for task teams, the development ofscorecards in a number of countries, and greateruse of M&E frameworks for managementpurposes (for example, for annual countryportfolio performance reviews in the LatinAmerica and the Caribbean Region). The ARDEmight have also given greater recognition to thecommitments and actions to date under theIDA14 and IDA15 Results Measurement Systems,notably in terms of baseline operational data.

Management does not share IEG’s view that staffincentive problems continue to be a majorobstacle to the improvement of M&E in projectsand country programs; rather, it believes that thenew policies and procedures and the ongoingefforts for enhancing practice are improvingquality in the more recent projects and programs.It would have been appropriate in this regard tohighlight the challenge of country capacity for


1 1 4

results management, since lack of data in poorercountries often is the most important obstacle toimproving M&E frameworks. Helping partnercountries overcome these data problems is aBank priority, notably through the MarrakechAction Plan for Statistics, the frameworkdiscussed by the Board in May 2006. While this isa long- term effort, there has been progress. A keytarget of this plan is to help all low- incomecountries develop a national strategy for improv-ing their statistical system. All but one Sub- Saharan African country (Somalia) have a strategyor are working on one. The Bank’s multidonorTrust Fund for Statistical Capacity Building hasprovided 20 new grants to countries for strategypreparation since 2006. Following the ThirdInternational Roundtable on Managing forDevelopment Results, in Hanoi, staff have beenworking closely with other development partnersthrough the Partnership for Statistics forDevelopment in the 21st Century to find theresources to scale up the implementation ofthese strategies. Part of the effort is mainstream-ing statistical capacity building in countryoperations. New statistical capacity- building(STATCAP) lending projects have been approvedfor Kenya and Russia, and pipeline countriesinclude Bolivia, Tanzania, and India. Two donors— the Netherlands and the United Kingdom— are working with the Bank toestablish a new Statistics for Results Facility, whichwill promote better coordinated efforts by donorsat the country level, and will provide grants tohelp finance the implementation of nationalstatistical improvement plans and link them moreclosely with national development strategies.Staff are working to launch this facility in Ghana atthe forthcoming High-Level Forum on Aid Effectiveness.

In discussing the M&E of projects and countryprograms reviewed, the ARDE could have paidmore attention to an obvious legacy effect. Thelarge majority of projects reviewed wereapproved prior to the adoption of the newpolicies and procedures; and nearly all results- based CASs reviewed were from the first round,in which results came from an existing portfolioof operations that had been approved prior to

that date. As noted in the report, in the smallsample of reviewed projects approved sincefiscal 2005, the share of projects with substantialor high M&E quality doubled. As more IEGreviews of results- based CASs from the secondround become available, the ratings for thequality of results frameworks are likely toimprove as well.

Management finds the ARDE’s review of experi-ence with the Bank’s impact evaluations helpfuland agrees that their increased use is importantfor strengthening the knowledge base forreporting on the Bank’s corporate results;however, management is also aware of practicalbarriers to impact evaluation work in the contextof specific operations. Management concurswith the emphasis in the report on strengthen-ing results reporting at the corporate level. Itrecently proposed to prepare a Bankwide resultsreport that will provide an overview of resultsachieved through Bank activities and report onprogress in the Bank’s results focus andmeasurement systems.

Shared Global ChallengesThe ARDE reviews the Bank’s experience withcountry program support for GPGs and offers anumber of recommendations on bridging the gapbetween global needs and country preferences. Italso reviews the Bank’s advocacy work on GPGs.

Country- Based Support for GPGsManagement welcomes the report’s manyvaluable insights into the challenges of support-ing countries’ contributions in cases whereglobal and country interests diverge significantlyand international agreement on a course ofcollective action has not yet been reached.Management also appreciates the careful reviewof the Bank’s work on the country level— in thecontext of country programs and through countrylevel activities of global programs and partnerships— and the proposals for strengthen-ing that work. Management agrees with many ofthe observations and assessments. Some,however, require comments, notably those onthe extent of Bank involvement on the countrylevel, the Bank’s ability to provide attractive


1 1 5

financial support for the GPG efforts of MICs, thereliance on the country program model, theattention paid to fostering GPGs in country andregional strategies, the deployment of globalprograms, and the option of setting asidecorporate administrative funds for high priorityGPG work at the country level.

Bank involvement in GPGs at the country level. TheARDE observes that GPGs other than environ-mental commons are not sufficientlyemphasized in CASs and that the extent of Bankinvolvement in GPG issues varies widely amongcountries. This observation fails to take intoaccount the framework for such Bank involve-ment.1 Since the Bank is only one player amongmany, the framework calls for identifying wherethere is a gap not being met by other agenciesand then filling the gap in areas where the Bankhas the capability and a comparative advantage.At the country level, country ownership andresponse to client demand is the primary princi-ple of Bank involvement— that is, the Bankworks with clients on issues of global or regionalconcern and supports their efforts at addressingsuch issues. In sum, the appropriate role for theBank at the country level is situation- specific tothe GPG being supported and to the countrycontext; hence, variations in the extent of Bankinvolvement at the country level among GPGsand among countries are to be expected. Forexample, on communicable diseases, substantialfunding is flowing from large vertical funds. Herethe challenge is to ensure that the funding isbalanced by Bank support for health sectorsystems and other government priorities for thedelivery of health services. While not presentedin CASs as support for a global public good, suchsupport by the Bank is key for bridging the gapbetween global and country interests in the areaof communicable diseases (that is, it providesindirect support to the respective GPG). Regard-ing climate change, pending a future globalclimate agreement, country ownership and aneffective country support program can be builtonly on demonstrated development opportuni-ties and advantages of a low- carbon, climate- resilient strategy tailored to specific country circumstances.

Financial support for MICs’ contributions to GPGs.The ARDE presents ample evidence of theeffectiveness of concessional finance in bridginggaps between global needs and country prefer-ences in International Development Association- eligible countries. With respect to InternationalBank for Reconstruction and Development(IBRD)-eligible countries, the report recognizesthe Global Environment Facility (GEF) as asource of concessional financing and carbonfinance as an innovative mechanism the Bankcan deploy also in MICs. But the report paysinsufficient attention to the Bank’s ability tofurther mobilize and innovate finance for scaling- up action on climate change, in cooperation withdevelopment partners and the private sector,and it also neglects the possible leveraging ofIBRD and International Finance Corporationfinancing and risk management services throughblending with concessional donor funds.Management considers overly pessimistic thereport’s conclusion with respect to future Bankwork on GPGs in MICs that “. . . the Bank has notbeen able to call on an attractive large- scalefunding program . . . to encourage comprehen-sive action on climate change.” Given that theWorld Bank Group only recently began full- fledged efforts to step up climate action to scale,the conclusion appears premature. For example,the Bank works with donor countries to mobilizesome $5 billion in new and additional financingfor the proposed Clean Technology Fund, withthe expectation that these new resources willleverage another $25–30 billion in financing for low- carbon investments.

Use of the country program model. Management isaware that country- based work on GPGs faceschallenges in cases where global and countryinterests diverge and there is no internationalframework for collective action, but it believesthat such challenges can be addressed. Manage-ment does not agree with the report’s broadconclusion that relying on the country programmodel for the Bank’s work on GPGs is a ”double- edged sword.” The Bank’s ultimate clients arepoor people. Using the country program model,the Bank is better able to provide analysis thatputs growth and poverty reduction at the core


1 1 6

and relates GPG challenges to this goal.Importantly, using the country program modelalso helps ensure country ownership of theactions supported by the Bank. These considera-tions argue for integrating the country- levelactivities of GPPs into country programs andagainst allowing them to bypass countryprograms. The example of Ethiopia’s healthbudget, which due to the proliferation of verticalhealth funds devotes some 50 percent ofresources to HIV/AIDS, is instructive. That theBank can balance resource flows by targeting itsresources to other health needs in the country isa considerable strength of the country programmodel. The key challenge, in management’sview, is to more thoroughly integrate GPGs intothe diagnosis of countries’ developmentchallenges and the dialogue with the govern-ment as part of the CAS process. With thisanalysis in place, the appropriate contribution ofglobal programs and trust funds as part of theCAS support program can then be determined,consistent with the plans by vice- presidentialunits for implementation of the Trust FundManagement Framework.

Country and regional strategies. The reportprovides valuable information on the treatmentof GPGs in CASs, Regional strategies, sectorstrategies, and Bankwide strategy documents. Inlight of recent developments, however, manage-ment finds too categorical the general statementin the report that the Bank’s attention to GPGs “.. . wanes as one moves down from corporatestrategies to sector or regional strategies, andthen down one level further to country strate-gies.” For example, the 2008 Latin America andCaribbean Regional strategy update emphasizessupport for clients’ efforts to address globalissues, and this priority is translated into specificactions planned in the recent round of CASs,including those for Brazil, Colombia, and Mexico.(In March 2008, the Board approved a $500million loan to Mexico to support implementa-tion of its National Climate Change Action Plan.)

Global programs. The report shows that there isroom for improvement in deploying global andregional programs at the country level. Manage-

ment welcomes the report’s proposal tostrengthen M&E quality in GPPs, though it notesthat an M&E framework for Development Grant Facility- supported global programs is in place andthe Consultative Group on International Agricul-ture Research program offers an example of goodpractice. Management appreciates IEG’s insightinto factors influencing the integration of globalprogram activities into country operations,notably the strong positive effect of substantialdeveloping country representation in the govern-ing body of a GPP in both enhancing legitimacyand fostering stronger linkages with countryoperations. But management also believes thatthe report should more clearly distinguish amongsubsets of GPPs (instead of treating GPPs as if theyare all interchangeably applicable at the countrylevel or all deal with GPG issues) and should alsorecognize that the Bank’s role at the country levelis limited in GPPs where the Bank is not animplementing agency. Management agrees thatbetter integration of global funds at the countrylevel is important. To that end, over the last twoyears, the Bank has been working closely with theDevelopment Assistance Committee of theOrganisation for Economic Co- operation andDevelopment and with bilateral donors andpartner countries on an initiative to better alignglobal programs with country operations. Thisinitiative has contributed to preparations for theAccra High- Level Meeting.

Strengthening country program support for GPGs.Management agrees with the report’s sugges-tions for strengthening the Bank’s country- basedsupport for GPGs, notably through improvedorganizational arrangements to coordinateglobal, regional, and national activities; betterdelivery of the Bank’s global knowledge tocountry teams and improved deployment ofnetwork anchor experts; support for clients’efforts to gain greater voice in shaping responsesto global issues; and the exercise of greaterselectivity in the Bank’s engagement in globalprograms. Management agrees on the desirabil-ity of strengthening incentives to deliver GPGs atthe country level where that is appropriate, buthas reservations about the suggested option ofsetting aside significant administrative funding at


1 1 7

the corporate level for allocation to high- priorityGPG work at the country level. This could distortincentives and encourage supply- driven initia-tives. A more effective approach— as alreadydemonstrated in the Latin American andCaribbean Region— is Regional managementcommitment and clear guidance to staff on theneed to be attuned to client demands forsupport for their GPG priorities. There alsoneeds to be an ongoing dialogue among thenetworks and Regions on staffing and resourceallocation, to address key corporate prioritiesthrough country operations where there isdemand at the country level. This would bestrengthened by recognition in staff perform-ance evaluations, as indicated in the report.

Advocacy on GPGsManagement appreciates IEG’s thoughtfulanalysis of the Bank’s successful advocacy workon GPGs and its achievements in creatinginnovative financing mechanisms. Managementagrees with most of the assessments, though itbelieves that the report could have providedmore recognition to the Bank’s performance inpreserving biodiversity, with funding from theGEF and own Bank resources. Two commentsare warranted.

• The report acknowledges the Bank’s many ac-tivities in support of the international climatechange agenda, including securing resourcesfor the GEF (the financial mechanism of theUnited Nations Framework Convention on Cli-mate Change), developing methodologies toput the Kyoto Protocol’s Clean DevelopmentMechanism into action, launching carbonfunds, and supporting the demonstration, de-ployment, and transfer of low- carbon tech-nologies and adaptation technologies. In thislight, management believes that the report’ssummary assessment that “. . . the extent towhich the Bank has been a leading influential

advocate on climate change is more debat-able” does not fully reflect this reality.

• Management agrees that increased Bank ef-forts are merited to strengthen the voice andrepresentation of developing countries in thegovernance of global programs. For example,the participation of developing countries inthe design of the Climate Investment Funds,including the design of a governance struc-ture with equal representation of donor and re-cipient countries, has been critical to providinglegitimacy for these funds. This participationis essential to set the stage at the global levelfor climate change mitigation programs at thecountry level.

ConclusionThis year’s ARDE has again provided a valuableservice by tracking development outcomes andimprovements in M&E frameworks, a servicethat helps the Bank improve its performance.Management commends IEG for an insightfulreview of the Bank’s experience with fosteringglobal public goods through support forcountries’ efforts and through constructiveadvocacy. In its lessons of experience, the reportappropriately focuses on ways to strengthen theBank’s countrylevel support for GPGs. Asdetailed above, management generally agreeswith the report’s suggestions, although it hasreservations about the option of setting asideadministrative funding at the corporate levelearmarked for countrylevel GPG work. Butmanagement also stresses the need for clarity onthe Bank’s role in supporting a country’s effortsat addressing issues of global or regionalconcern. Such support must build on the Bank’score mandate to foster the country’s growth andpoverty reduction goals, support measures thatare owned by the country, and take into account,for each GPG, the involvement of other agenciesand partners and the Bank’s capability andcomparative advantage.

1 1 9

Chapter 11. Until this year, IEG produced an Annual Review

of Operations Evaluation (AROE), which included analy-

sis of the Bank’s monitoring and evaluation systems, as

well as an IEG self-evaluation. In consultation with

CODE, the AROE and ARDE have been combined this

year into a single document that maintains the essen-

tial features of both.

Chapter 21. See, for instance, Wappenhans (1992) for an early

critique of this “lending culture.”

2. IEG’s measure of outcome considers three factors;

relevance, efficacy, and efficiency. Relevance measures

the expected development impact of a project design

by weighing the continuing relevance of a project’s ob-

jectives. Efficacy refers to the extent to which each ob-

jective was achieved, or expected to be achieved.

Efficiency measures the cost-effectiveness of a project,

based mainly on sectorwide best practices and indica-

tors where available. Combining these three factors,

overall outcome is rated on a 6-point scale, ranging

from highly satisfactory to highly unsatisfactory.

3. IEG’s sustainability measure assesses the re-

silience to risk of net benefit flows over time by an-

swering the following questions: At the time of

evaluation, what is the resilience to risks affecting future

net benefit flows? How sensitive is the intervention to

changes in the operating environment? Will the inter-

vention continue to produce net benefits as long as in-

tended, or even longer? How well will the intervention

weather shocks and changing circumstances?

4. Under the new harmonized evaluation criteria

for project evaluations, approved in October 2005, proj-

ects would no longer be rated for their sustainability and

institutional development impact. Therefore, the fiscal

2003–07 cohort includes about 50 percent of evaluated

fiscal 2006 projects, and seven projects in fiscal 2007.

5. The analysis excludes the sector boards on gen-

der, global information and technology, poverty re-

duction, and social development because IEG has

evaluated very few projects managed by these sector

boards.

6. Data for fiscal 2007 remains partial because only

three-quarters of the projects that closed in fiscal 2007

have been evaluated to date.

7. During implementation, the Bank manages a proj-

ect by means of an ongoing Implementation Status and

Results report (ISR). Upon completion of a project, the

Bank conducts a self-evaluation. This self-evaluation is

called an Implementation Completion and Results re-

port (ICR), which is then independently evaluated by

IEG, leading to an Implementation Completion and

Results review.

8. Project outcomes have also declined when

weighted by disbursements, from 90 percent of projects

rated moderately satisfactory or better in fiscal 2006, to

83 percent in fiscal 2007.

9. IEG, in agreement with the Bank, introduced an

explicit requirement for evidence to substantiate as-

sessments of performance in projects, which fully went

into effect at the beginning of fiscal 2007.

10. CASCR reviews tend to assess a shorter time-frame

for the period of a single CAS, generally between 3 and

5 years. Country Assistance Evaluations (CAEs) assess

Bank programs over a longer period, typically between

6 and 10 years, and involve an in-depth mission to the

country. This analysis draws on IEG reviews of 59 CASCRs

completed since fiscal 2004, and one CASCR completed

near the end of fiscal 2003.

Chapter 31. Prior to 2004, the Bank required log-frames in

their projects. The results framework places greater

emphasis on the outcomes, differentiating between

the final outcomes (the project development objec-

ENDNOTES

tives) and intermediate outcomes, which can be used

to monitor progress toward achieving the project de-

velopment objectives. The log-frame included inputs and

outputs as well as outcomes.

2. An M&E system with a “high” rating is expected

to strongly influence project performance, provide suf-

ficient information to satisfactorily assess the stated

project objectives, and contribute to testing the un-

derlying development model (or results chain). Con-

versely, a system rated “negligible” would have many

weaknesses, have very little impact on the project or pro-

gram, be insufficient to satisfactorily assess the stated

project objectives, and contribute little to testing the de-

velopment model.

3. As of February 2008.

4. BP 2.11, Country Assistance Strategies, was up-

dated in June 2005 to reflect the results-based CAS ap-

proach.

5. Data on Bank-supported evaluations were drawn

from the Development Impact Evaluation database,

May 13, 2008.

Chapter 61. In- person and telephone interviews of 23 World

Bank operational managers, including country and sec-

tor directors.

2. The “successor” to the CAS—the Country Part-

nership Strategy (CPS)—is used in more recent cases.

All references here to CASs, therefore, also cover CPSs.

3. In- person and telephone interviews of 23 World


tor directors.

4. Discussed by the Executive Board in May 2008.

5. Such as the World Bank’s sustainable develop-

ment strategy: Making Sustainable Commitments: An

Environment Strategy for the World Bank. 2001.

6. With the exception of the East Asia and Pacific and

the Europe and Central Asia environment strategies,

which have a strong focus on GPGs.

7. The Health, Nutrition, and Population strategy, al-

though it mentions the promotion of GPGs, provides

very little detail on its GPG strategy.

8. Here GPGs are defined to include all work on the

following Bank-defined themes: HIV/AIDS, other com-

municable diseases, biodiversity, climate change, water

resources management, other environment and natu-

ral resources management, international financial ar-

chitecture, regional integration, trade facilitation and

market access, and other trade and integration.

9. Estimates are drawn from the Global Public Goods

Working Group.



tor directors.



tor directors.

12. Information from Avian Flu Resources by World

Bank Operations Policy and Country Services (OPCS).

13. It should be noted that this estimate covers all

Bank-supported trade projects, a significant share of

which were directed at domestic trade issues, as opposed

to the global dimension of trade.

14. For shorthand, this section uses the term “global

programs,” and the points made largely refer also to re-

gional programs, unless otherwise stated.

15. Here the GPGs included fall under four of the

Bank’s five GPG areas, excluding creation and sharing

of knowledge.

16. Findings from an IEG focus group conducted with

eight Task Team Leaders from Global and Regional Part-

nership Programs directly involved with global public

goods, April 2, 2008.

Chapter 71. Making Trade Work for Poor People (Stern 2002)

was considered a timely input to the Doha Trade Min-

isterial, and Global Economic Prospects 2004: Realizing

the Development Promise of the Doha Agenda (World

Bank 2004c) was published ahead of the Cancun Trade

Ministerial. World Trade Organization members cited

these sources as contributing to their understanding of

the issues and to the debate (IEG 2006a).

2. The Deaton report (2006), which evaluated the

Bank’s research program from 1998 to 2005, noted that

“historically, the Bank has had a very active, vibrant and

influential research program on international trade and

trade policy . . . [although] suspects that it has not

been as influential in recent years as once was the case.”

3. For instance, the World Bank’s position on trade

policy—including pressuring the United States and Eu-

rope to reduce agricultural subsidies—was widely re-

ported in the popular press, including nearly 50

references in the Economist and the Financial Times

since January 2005. World Bank (2006a) discusses the

effectiveness of these public interventions more fully.

4. For example, an op-ed piece in the Financial

Times at the time of the Bali conference, late 2007.


1 2 0

5. Barrett (2006) shows that the incentives for co-

operation in R&D depend on the prospects of the tech-

nologies embodying R&D being diffused. If new

technologies are diffused to developing countries, rich

countries will have greater incentive to finance R&D.

6. Refer to the following on CGIAR’s Web site at

www.cgiar.org: “Research & Impact,” “CGIAR on Global

Issues,” “Global Climate Change: Can Agriculture Cope?”;

and the World Bank’s Web site on “Climate Change Re-

search: Sustainable Rural and Urban Development” at

http://go.worldbank.org/7Q6I1HUPZ0.

7. In-person and telephone interviews of 11 senior

experts from international institutions, international

NGOs, partner countries, and the World Bank.

8. This is in part because loans tend to be slower in

disbursing, and the program is still very new.

9. In detail, those roles are: lender, founder, mem-

ber of the governing body, convener, financial contrib-

utor, trust-fund trustee, house secretariat, implementing

agency, chair of the governing body, trust-fund manager,

and cosponsor (IEG 2004b).

10. Development Committee Communiqué, April 13,

2008.

Appendix A1. IEG’s institutional development impact measure

evaluates the extent to which an intervention improves

the ability of a country or region to make more efficient,

equitable, and sustainable use of its human, financial,

and natural resources. Such improvements can derive

from changes in values, customs, laws and regulations,

and organizational mandates.

2. The analysis excludes the sector boards on gen-

der, global information and technology, poverty re-

duction, and social development.

3. The five countries/areas that received Special Fi-

nancing Grants in the past decade are Bosnia and Herze-

govina, Kosovo, Serbia, Timor-Leste, and the West Bank

and Gaza.

4. Of this amount, $351 million financed operations

in the West Bank and Gaza.

Appendix C1. Mandate of the Director-General, Evaluation.

2. IEG’s independence has been validated according

to the standards of the OECD’s Development Assis-

tance Committee and the U.S. Government Accounta-

bility Office (among others), as documented in two

papers: IEG, “OED Reach: Independence of OED,” Feb-

ruary 23, 2003; and “2004 External Review of OED” on

IEG’s Web site at www.worldbank.org/ieg/intro.

3. The Bank’s new framework for global public goods

(World Bank 2007d) makes explicit reference to IEG’s

evaluation of regional programs as the foundation for

expanding the Bank’s work in regional programs.

4. Good examples for a CASCR and CASCR Review

are the Philippines CASCR from April 2005 and IEG’s re-

view from May 2005, and the Yemen CASCR from May

2006 and IEG’s review from June 2006. These CASCRs

won IEG 2006 and 2007 Good Practice Awards.

5. Going forward, CASCR ratings will become a

tier 2 indicator in the IDA15 Results Measurement Sys-

tem.

6. The award categories are Projects, Implementa-

tion Completion and Results reports, Country Pro-

grams, Country Assistance Strategy Completion Reports,

Monitoring and Evaluation, and Initiatives with Demon-

strated Impact/Results.

7. Although 24 percent is not a high response rate,

a comparison of respondent and nonrespondent char-

acteristics (grade level, region, headquarters/country of-

fice) shows that both groups are very similar. The only

dimension with a difference of more than 6 percent be-

tween respondents and nonrespondents is the share of

headquarters and country office staff.

8. It should be noted that this question included a

“not applicable” answer option in 2007, but not in 2006.

The percentages presented in the text include these re-

sponses in the denominator.

9. Previous note applies.

10. This compares with about seven reports about

IEG in the media each year between 1996 and 2004. The

IEG reports with the highest coverage were evaluation

of Bank assistance to middle-income countries, from

2007, and the evaluation of Bank assistance to fragile

states, from 2006. For both reports, IEG undertook

multiple media launches in Washington, D.C. and in the

field.

11. Level of adoption ratings are high—fully adopted;

substantial—largely adopted but not fully incorporated

into policy, strategy, or operations as yet; medium—

adopted in some operational and policy work but not

to a significant degree in key areas; and negligible—no

evidence or plan for adoption, or plans and actions for

adoption are in a very preliminary stage.

12. The status of recommendations is rated as: ac-

tive and remains actionable by management; complete

and archived in the e-MAR; obsolete or overtaken by

E N D N OT E S

1 2 1

events and archived in e-MAR; difference of opinion be-

tween management and IEG.

13 . Recommendations that were not accepted by

management will be excluded from the following data

analysis.

14. This excludes the four recommendations on

which there was a difference of opinion between IEG

and management. These did not receive an adoption rat-

ing and are excluded from the sample when compar-

ing levels of adoption.

Appendix E1. See World Bank 2007m.


1 2 2

1 2 3

Baer, Paul, Tom Athanasiou, and Sivan Kartha. 2007.

The Right to Development in a Climate Constrained

World: The Greenhouse Development Rights Frame-

work. Heinrich Böll Stiftung Publications on Ecology.

Berlin: Heinrich Böll Foundation.

Barrett, Scott. 2006. “Climate Treaties and ‘Breakthrough’

Technologies.” American Economic Review: Papers

and Proceedings 96 (2): 22–25.

———. 2007. Why Cooperate? The Incentive to Supply

Global Public Goods. New York: Oxford University

Press.

———. 2008. “The World Bank’s Role in Fostering

Global Public Goods: Climate Change and Essential

Medicines.” ARDE background paper. Available on

this evaluation’s Web site at www.worldbank.org/ieg.

Benedick, R.E. 1998. Ozone Diplomacy: New Directions

in Safeguarding the Planet. Cambridge, MA: Harvard

University Press.

Brahmbhatt, Milan. 2005. “Avian and Human Pandemic

Influenza— Economic and Social Impacts.” Speech

delivered during the “Global Partners Meeting on

Avian Influenza and Human Pandemic Influenza,”

hosted by the World Health Organization, Geneva,

Switzerland, November 7–9.

CAN- E (Climate Action Network Europe). 2002. “CAN

Europe Comments on PCF Plantar Project.” May 7.

Brussels: CAN- E. http://www.climnet.org/pubs/CAN

EuropePlantar.pdf.

Carr, Christopher, and Flavia Rosembuj. 2007 “World

Bank Experiences in Contracting for Emission Re-

ductions.” Environmental Liability, No. 2.

Deaton, Angus, Abhijit Banerjee, Nora Lustig, and Ken

Rogoff. 2006. An Evaluation of World Bank Re-

search, 1998–2005. Washington, DC: World Bank.

Ellis, Jane, and Sami Kamel. 2007. “Overcoming Barri-

ers to Clean Development Mechanism Projects.” Or-

ganisation for Economic Co- operation and

Development and International Energy Agency. Paris:

OECD. http://cd4cdm.org/Publications/Overcoming

BarriersCDMprojects.pdf.

GLCA (Global Leadership for Climate Action). 2007.

Framework for a Post-2012 Agreement on Climate

Change. http://www.unfoundation.org/files/pdf/2007/

GLCA_Framework2007.pdf.

House of Commons— International Development Com-

mittee. 2008. “DFID and the World Bank: Sixth Re-

port of Session 2007–08,” Vol. 1. London, UK: House

of Commons.

IEG (Independent Evaluation Group). 2002. The World

Bank’s Approach to Global Programs: An Inde-

pendent Evaluation. Phase 1, Report No. 26284-

GLB. Washington, DC: World Bank.

———. 2004a. Annual Report on Operations Evalua-

tion. IEG Study Series. Washington, DC: World Bank.

———. 2004b. Addressing the Challenges of Global-

ization: An Independent Evaluation of the World

Bank’s Approach to Global Programs. IEG Study Se-

ries. Washington, DC: World Bank.

———. 2004c. The CGIAR at 31—An Independent

Meta- Evaluation of the Consultative Group on In-

ternational Agricultural Research. Washington, DC:

World Bank.

———. 2005. Committing to Results: Improving the Ef-

fectiveness of HIV/AIDS Assistance. IEG Study Series.

Washington, DC: World Bank.

———. 2006a. Assessing World Bank Support for Trade

(1987–2004): An IEG Evaluation. IEG Study Series.


———. 2006b. Hazards of Nature, Risks to Develop-

ment: An IEG Evaluation of World Bank Assistance

for Natural Disasters. Washington, DC: World Bank.

———. 2006c. Annual Review of Development Effec-

tiveness: Getting Results. Washington, DC: World

Bank.

———. 2006d. Annual Report on Operations Evalua-

tion. IEG Study Series. Washington, DC: World Bank.

BIBLIOGRAPHY

———. 2006e. Engaging with Fragile States: An IEG Re-

view of World Bank Support to Low- Income Coun-

tries under Stress. Washington, DC: World Bank.

———. 2007a. Development Results in Middle- Income

Countries: An Evaluation of the World Bank’s Sup-

port. Washington, DC: World Bank.

———. 2007b. Sourcebook for Evaluating Global and

Regional Partnership Programs: Indicative Prin-

ciples and Standards. Washington, DC: World Bank.

———. 2007c. The Development Potential of Regional

Programs: An Evaluation of World Bank Support

of Multicountry Operations. Washington, DC: World

Bank.

———. 2007d. World Bank Assistance to Agriculture

in Sub- Saharan Africa: An IEG Review. Washington,

DC: World Bank.

———. 2008a. Supporting Environmental Sustain-

ability: An Evaluation of World Bank Group Expe-

rience, 1990–2007. IEG Study Series. Washington, DC:

World Bank.

———. 2008b. Progress Report on IEG’s Activities Re-

lating to Global and Regional Partnership Pro-

grams. CODE Subcommittee Discussion Paper.


———. 2008c. Public Sector Reform: What Works and

Why?: An IEG Evaluation of World Bank Support.


———. 2008d. Using Training to Build Capacity for De-

velopment: An Evaluation of Project- Based and

WBI Training. IEG Study Series. Washington, DC:

World Bank.

ITF (International Task Force on Global Public Goods).

2006. Meeting Global Challenges: International Co-

operation in the National Interest. Final report.

Stockholm: ITF.

Kaul, Inge, Isabelle Grunberg, and Marc Stern, eds. 1999.

Global Public Goods: International Cooperation

in the 21st Century. New York: Oxford University

Press.

Müller, Benito, and Harald Winkler. 2008. “One Step For-

ward, Two Steps Back? The Governance of the World

Bank Climate Investment Funds.” Energy and Envi-

ronment Comment. February. Oxford, UK: Oxford

Institute for Energy Studies. http://www.oxfordenergy

.org/pdfs/comment_0208- 1.pdf.

Osterholm, Michael. 2007. “Unprepared for a Pandemic

(H5N1 avian influenza virus).” Foreign Affairs 86

(2): 47–57.

Redman, Janet. 2008. World Bank: Climate Profiteer.

Washington, DC: Institute for Policy Studies.

http://www. ips- dc.org/reports/# 292.

Rodrik, Dani. 2006. “Goodbye Washington Consensus,

Hello Washington Confusion? A Review of the World

Bank’s Economic Growth in the 1990s: Learning

from a Decade of Reform.” Journal of Economic Lit-

erature 44 (4): 973–87.

Schneider, Lambert. 2007. Is the CDM Fulfilling Its En-

vironmental and Sustainable Development Ob-

jectives?: An Evaluation of the CDM and Options for

Improvement. Report prepared for the World Wildlife

Fund. November 5. Berlin: World Wildlife Fund.

http://www.wwf.org.uk/filelibrary/pdf/cdm_fill_objecti

ves.pdf

Smith, Richard, David Woodward, Arnab Acharya, and

Robert Beaglehole. 2004. “Communicable Disease

Control: A ‘Global Public Good’ Perspective.” Health

Policy and Planning 19 (5): 271–78,

Stern, Nicholas 2002. “Making Trade Work for Poor Peo-

ple.” Speech delivered at the National Council of Ap-

plied Economic Research, New Delhi, November

28. http://siteresources.worldbank.org/INTRES/

Resources/stern_speech_makingtrworkforpoor_nov

2002.pdf

———. 2006. The Economics of Climate Change: The

Stern Review. Cabinet Office, Her Majesty’s Treasury,

United Kingdom.

Sweden Ministry for Foreign Affairs. 2001. Financing and

Providing Global Public Goods: Expectations and

Prospects. Study 2001:2. Prepared by Francisco Sagasti

and Keith Bezanson, on behalf of the Institute of

Development Studies, Sussex, U.K.; Stockholm,

Sweden.

UNSIC (United Nations System Influenza Coordinator)

and World Bank. 2007. Responses to Avian Influenza

and State of Pandemic Readiness: Third Global

Progress Report. New York: UNSIC- World Bank.

Wappenhans, W.A. 1992. Report of the Portfolio Man-

agement Task Force. World Bank, Washington, DC.

Wheeler. David. 2008. “Crossroads at Mmamabula: Will

the World Bank Choose the Clean Energy Path?”

Working Paper No. 140. Center for Global Develop-

ment, Washington, DC.

Woods, Ngaire. Forthcoming. “Governance Matters: The

IMF and Sub- Saharan Africa.” In Finance, Development,

and the IMF, ed. James Boughton and Domenico

Lombardi. Oxford, UK: Oxford University Press.

Woods, Ngaire, and Domenico Lombardi. 2006. “Uneven

Patterns of Governance: How Developing Countries


1 2 4

Are Represented at the IMF.” Review of Interna-

tional Political Economy 13 (3): 480–515.

World Bank. 1997. “Toward Global Sustainability,” James

D. Wolfensohn’s speech to the United Nations Gen-

eral Assemply Special Session on the Environment,

New York, June 25.

———. 2000. Report of the Working Group on Im-

proving Quality of Monitoring and Evaluation in

Bank Financed Operations. Operations Policy and

Country Services Department. Washington, DC:

World Bank.

———. 2001. Making Sustainable Commitments: An

Environment Strategy for the World Bank. Wash-

ington, DC: World Bank.

———. 2003a. Country Assistance Strategies: Retro-

spective and Future Directions (CAS Retro III). Op-

erations Policy and Country Services Department.


———. 2003b. Putting Our Commitments to Work: An

Environment Strategy Implementation Progress Re-

port. Washington, DC: World Bank.

———. 2004a. “OP/BP 8.60, Development Policy Lend-

ing.” Operations Policy and Country Services De-

partment. World Bank, Washington, DC.

———. 2004b. Responding to Climate Change: Pro-

posed Action Plan for the World Bank in Latin

America. Washington, DC: World Bank.

———. 2004c. Global Economic Prospects 2004: Re-

alizing the Development Promise of the Doha

Agenda. Washington, DC: World Bank.

———. 2005a. Results Focus in Country Assistance

Strategies: A Stock Taking of Results- Based CASs.

Operations Policy and Country Services Department.


———. 2005b. Harmonized Evaluation Criteria for

ICR and OED Evaluations. OPCS Guidelines. Op-

erations Policy and Country Services Department.


———. 2005c. Improving Health, Nutrition, and Pop-

ulation Outcomes in Sub- Saharan Africa: The Role

of the World Bank. Washington, DC: World Bank.

———. 2005d. “Program Framework Document for

Proposed Loans/Credits/Grants in the Amount of

US$500 Million Equivalent for a Global Program for

Avian Influenza Control and Human Pandemic Pre-

paredness and Response.” Report No. 34386. Wash-

ington, DC: World Bank.

———. 2006a. “Strengthening the World Bank’s En-

gagement with IBRD Partner Countries, Develop-

ment Committee.” Report No. DC2006-0014, World

Bank, Washington, DC.

———. 2006b. “Avian and Human Influenza: Financing

Needs and Gaps.” Technical note prepared for the

International Pledging Conference on “Avian and

Human Influenza,” in Beijing, China, January 17–18.

———. 2006c. “Avian and Human Influenza: Multi-

Donor Financing Framework.” Technical note pre-

pared for the International Pledging Conference on

“Avian and Human Influenza,” in Beijing, China, Jan-

uary 17–18.

———. 2006d. Forest Carbon Partnership Facility: A

Framework for Piloting Activities to Reduce Emis-

sions from Deforestation and Forest Degradation.


———. 2006e. Protecting and Improving the Global

Commons: 15 Years of the World Bank Group Global

Environmental Facility Program. Washington, DC:

World Bank.

———. 2007a. Quality of Supervision in FY05–06

(QSA7). Quality Assurance Group. Washington, DC:

World Bank.

———. 2007b. Focus on Results: The IDA14 Results

Measurement System and Direction for IDA15. In-

ternational Development Association / Operations

Policy and Country Services Department. Washing-

ton, DC: World Bank.

———. 2007c. Note from the President of the World

Bank, Robert Zoellick, for the 76th Meeting of the De-

velopment Committee. October 19, World Bank,

Washington, DC.

———. 2007d. Global Public Goods: A Framework for

the Role of the World Bank. Global Programs and Part-

nerships (GPP). Washington, DC: World Bank.

———. 2007e. Meeting the Challenges of Global De-

velopment: A Long- Term Strategic Exercise for the

World Bank Group. Washington, DC: World Bank.

———. 2007f. Sector Implementation Strategy: Third

Review. Operations Policy and Country Services De-

livery Management. Washington, DC: World Bank.

———. 2007g. Focus on Sustainability 2005/2006.


———. 2007h. Clean Energy for Development Invest-

ment Framework: The World Bank Group Action

Plan. Washington, DC: World Bank.

———. 2007i. A Brief History of the Prototype Carbon

Fund: The PCF 1997—2007, Pioneers in a Decade

of Change. Carbon Finance Unit. Washington, DC:

World Bank.

B I B L I O G R A P H Y

1 2 5

———. 2007j. Carbon Finance for Sustainable De-

velopment: 2007. Washington, DC: World Bank.

———. 2007k. Healthy Development: The World Bank

Strategy for Health, Nutrition, and Population Re-

sults. Washington, DC: World Bank.

———. 2007l. IDA and Climate Change: Making Cli-

mate Action Work for Development. Washington,

DC: World Bank.

———. 2007m. Global Public Goods: A Framework for

the Role of the World Bank. Report no. DC2007-

0020. Background report for the Development Com-

mittee, World Bank, Washington, DC.

———. 2008a. Annual Report on Portfolio Performance

– 2007. Quality Assurance Group. Washington, DC:

World Bank.

———. 2008b. Quality at Entry in FY06–07 (QEA8). Qual-

ity Assurance Group. Washington, DC: World Bank.

———. 2008c. Additions to IDA Resources: Fifteenth Re-

plenishment. IDA: The Platform for Achieving Results

at the Country Level: Report from the Executive Di-

rectors of the International Development Association

to the Board of Governors. Approved by the Exec-

utive Directors of IDA, February 28. Washington,

DC: World Bank.

———. 2008d. “Consultation Draft on Climate Invest-

ment Funds.” Washington, DC: World Bank.


1 2 6

2006 Annual Report on Operations Evaluation

Annual Review of Development Effectiveness 2006: Getting Results

Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank’s Approach to Global Programs

Assessing World Bank Support for Trade, 1987–2004: An IEG Evaluation

Books, Buildings, and Learning Outcomes: An Impact Evaluation of World Bank Support to Basic Education in Ghana

Brazil: Forging a Strategic Partnership for Results—An OED Evaluation of World Bank Assistance

Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy

Capacity Building in Africa: An OED Evaluation of World Bank Support

China: An Evaluation of World Bank Assistance

The CGIAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research

Committing to Results: Improving the Effectiveness of HIV/AIDS Assistance—An OED Evaluation of the World Bank’sAssistance for HIV/AIDS Control

Country Assistance Evaluation Retrospective: OED Self-Evaluation

Debt Relief for the Poorest: An Evaluation Update of the HIPC Initiative

A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995–2005

The Development Potential of Regional Programs: An Evaluation of World Bank Support of Multicountry Operations

Development Results in Middle-Income Countries: An Evaluation of the World Bank’s Support

Economies in Transition: An OED Evaluation of World Bank Assistance

Engaging with Fragile States: An IEG Review of World Bank Support to Low-Income Countries Under Stress

The Effectiveness of World Bank Support for Community-Based and –Driven Development: An OED Evaluation

Evaluating a Decade of World Bank Gender Policy: 1990–99

Evaluation of World Bank Assistance to Pacific Member Countries, 1992–2002

Extractive Industries and Sustainable Development: An Evaluation of World Bank Group Experience

Financial Sector Assessment Program: IEG Review of the Joint World Bank and IMF Initiative

From Schooling Access to Learning Outcomes: An Unfinished Agenda—An Evaluation of World Bank Support to PrimaryEducation

Hazards of Nature, Risks to Development: An IEG Evaluation of World Bank Assistance for Natural Disasters

How to Build M&E Systems to Support Better Government

IEG Review of World Bank Assistance for Financial Sector Reform

Improving Investment Climates: An Evaluation of World Bank Group Assistance

Improving the Lives of the Poor Through Investment in Cities

Improving the World Bank’s Development Assistance: What Does Evaluation Show?

Maintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health andNutrition Outcomes in Bangladesh

New Renewable Energy: A Review of the World Bank’s Assistance

Pakistan: An Evaluation of the World Bank’s Assistance

Pension Reform and the Development of Pension Systems: An Evaluation of World Bank Assistance

Poland Country Assistance Review: Partnership in a Transition Economy

The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank’s Support Through 2003

The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support

Power for Development: A Review of the World Bank Group’s Experience with Private Participation in the ElectricitySector

Putting Social Development to Work for the Poor: An OED Review of World Bank Activities

Small States: Making the Most of Development Assistance—A Synthesis of World Bank Findings

Social Funds: Assessing Effectiveness

Sourcebook for Evaluating Global and Regional Partnership Programs

Water Management in Agriculture: Ten Years of World Bank Assistance, 1994–2004

World Bank Assistance to the Financial Sector: A Synthesis of IEG Evaluations

The World Bank in Turkey: 1993–2004—An IEG Country Assistance Evaluation

World Bank Lending for Lines of Credit: An IEG Evaluation

IEG PUBLICATIONS

All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual selection, please visithttp://www.worldbank.org/ieg

WORKING FOR A WORLD FREE OF POVERTY

The World Bank Group consists of five institutions—the International Bank for Reconstruction and Development(IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), theMultilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of InvestmentDisputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their envi-ronment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public andprivate sectors.

THE WORLD BANK GROUP

ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION

The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses onassessment of IFC’s work toward private sector development, and IEG-MIGA evaluates the contributions of MIGAguarantee projects and services. IEG reports directly to the Bank’s Board of Directors through the Director-General,Evaluation.

The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of theBank Group’s work, and to provide accountability in the achievement of its objectives. It also improves Bank Groupwork by identifying and disseminating the lessons learned from experience and by framing recommendations drawnfrom evaluation findings.

THE INDEPENDENT EVALUATION GROUP


THE WORLD BANK

THE WORLD BANK

Annual Review of Development Effectiveness


An

nu

al Review

of D

evelop

men

t Effectiven

ess 2008: Shared

Glo

bal C

hallen

gesT

he W

orld

Ban

k

2008

SKU 17714

ISBN 978-0-8213-7714-7

Ë|xHSKIMBy377147zv*:+:!:+:!
