eVALUatiOn Matters - African Development Bankidev.afdb.org/sites/default/files/documents/files... · Policy-Based Operations — Putting Knowledge to Work African Development Bank

Decision to implement

Mid-term evaluation

Post-implementation evaluation

Final impact evaluation

PROJECT CYCLE (Monitoring throughout)

DESIGN (Plan project, considering the evaluability of the design)

ANALYSIS

EVALUATION

IMPLEMENT AND MONITOR

Independent Development EvaluationAfrican Development Bank

idev.afdb.org

There are three types of influential evaluations: those that determine the common good (raising a social, economic, or political problem); those that help select a course of action, and those that help adapt the course of action (Gary T. Henry)

Making Evaluation Influential at the African Development Bank

Influential Evaluations: Illustrations from Multilateral Development Organizations – IFAD

Evaluation Influence and the Evaluator’s Independence:World Bank Group

21 2712

73

First Quarter 2015

eVALUatiOn MattersA Quarterly Knowledge Publication from Independent Development Evaluation at the African Development Bank Group

EvAluATIOn InfluEncE IN DEVELOPMENt OrGANIzAtIONs

eVALUatiOn Matters

Evaluations make a difference in development workThe impact of evaluations on development work is rarely overtly acknowledged. This sometimes leads to a questioning of the cost and usefulness of evaluations. Yet, there is ample evidence that evaluations contribute to the effectiveness of development work and often change the trajectory of development efforts. This issue of eVALUation Matters showcases evaluations that made a difference or had a lasting impact on development work.

A quarterly knowledge publication from Independent Development Evaluation at the African Development Bank Group

Understanding evaluation influenceEvaluation influence connotes a change in pre-and post-evaluation states that can be attributed to the evaluation and influence decisions regarding the implementation, expansion or funding of programs, policies or projects. There are three ways to categorise evaluation influence: (i) direct or instrumental, connoting the direct effect of an evaluation; (ii) conceptual, when an evaluation changes ways of thinking or provides new understanding, and (iii) political or symbolic, when the mere existence of an evaluations, and not the results, can be used to persuade or support a viewpoint (Johnson et. al., 2009). P. 7

is a quarterly publication from Independent Development Evaluation at the African Development Bank Group. It provides different perspectives and insights on evaluation and development issues.

Editor-in-Chief: Felicia Avwontom Produced under the guidance of the AfDB Evaluator General, Rakesh Nangia.

© 2015—African Development Bank (AfDB) African Development Bank GroupImmeuble du Centre de commerce international d’Abidjan (CCIA) Avenue Jean-Paul II01 BP 1387, Abidjan 01 Cote dIvoire Phone: +225 20 26 44 44 Fax: +225 20 21 31 00 Internet: www.afdb.org

Design: Phoenixdesignaid and Felicia AvwontomLayout: Phoenixdesignaid

12

6834

21

46

Evaluation Influence and the Evaluator’s Independence: the Evaluation of World Bank Group Guarantee Instruments Marvin Taylor-Dormond, Stoyan Tenev, and Monika Weber-Fahr, World Bank

Views on evaluation use and influence vary widely and reflect how one perceives the nature of evaluation and the role of the evaluator in the process of social change: At one end (Scriven, Campbell) is the view that truth and objectivity are the paramount concern of the evaluator. Use and influence are to be left to the market of ideas. At the other extreme is the view that use is everything (Patton).

Fragile states—Back from the EdgeAfrican Development Bank

Timing must be carefully calibrated for an evaluation to have influence. This was the case with the evaluation of the African Development Bank’s work in fragile states. The evaluation synthesis and accompanying materials were completed in time to serve as a building block for the new Bank strategy for fragile states and to provide information and insight to the High-Level Panel on Fragile States.

Influential Evaluations: Illustrations from Multilateral Development Organizations Oscar A. Garcia, International Fund for Agricultural Development

Evaluations can guide decision-making and are therefore vital for improving the quality and results of development work. Influential evaluations in multilateral development organizations are those that contribute to significant change and reform for better development effectiveness. This article reviews two influential evaluations based on their instrumental use and a strategic evaluation based on its use for advocacy purposes.

Non-profit organisations have various uses for evaluations, including,“ …program improvement, publicizing activities, reconsidering the mission, raising funds, motivating volunteers and staff, and determining expenditures” (Eckerd and Moulton, 2011: 101). p. 5

6

27

38

Evaluation Influence: An OverviewSeetharam Mukkavilli, African Development Bank

Evaluation influence connotes a change in pre-and post-evaluation states that can be empirically attributed to the evaluation and influence decisions regarding the implementation, expansion or funding of programs, policies or projects. There are three ways to categorise evaluation influence.

Making Evaluation Influential at the African Development Bank A conversation with rakesh Nangia, Evaluator-General, Independent Development EvaluationRakesh Nangia, Boubacar Ly and Foday Turay, African Development Bank

An influential evaluation is one that it is not merely accepted by the Board/CODE andManagement, but is owned and used by Management to bring about changes.

Fostering Inclusive Finance in Africa: An Evaluation of the African Development Bank’s Microfinance Policy, strategy, and Operations, 2000–2012 Albert Eneas Gakusi, African Development Bank

How good timing, good collaboration with subject matter experts, successful engagement of stakeholders and concrete and actionable recommendations helped the AfDB answer puzzling questions about its microfinance initiatives.

eVALUatiOn Matters

Content

Thematic Evaluation Project Cluster Evaluation

Regional Integration Stra

tegy

Evaluation

Project Perfo

rmance Evaluation

(Public Secto

r)Impact Evaluation

Project Performance Evaluation

(Private Sector)

Coun

try S

trate

gy E

valu

atio

n

Evaluation Synthesis

Corporate Evaluation

Sect

or E

valu

atio

nSe

ctor

Eva

luat

ion

Evaluation at the AfDB

Acknowledgments: IDEV is grateful to all contributors, reviewers, editors, and proofreaders who worked on this issue.

64

Trust Fund Management at the African Development Bank

An Independent evAluAtIon

About the AfDB: The overarching objective of the African Development Bank Group is to spur sustainable economic development and social progress in its regional member countries (RMCs), thus contributing to poverty reduction. The Bank Group achieves this objective by mobilizing and allocating resources for investment in RMCs; and providing policy advice and technical assistance to support development efforts.

The mission of the Operations Evaluation Department is to enhance the development effectiveness of AfDB initiatives in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge.

Director: Rakesh Nangia, [email protected], Project Level Evaluations: Mohamed Manai, [email protected]

Operations Evaluation Department, African Development BankBP 323, 1002 Tunis-Belvedere, TunisiaTel : (216) 71 102 841 Fax : (216) 71 194 460

Helpdesk: [email protected] Website: www.afdb.org/opev


An Independent evAluAtIon This evaluation reviews the establishment and implementation of 28 bilateral and thematic trust funds moni-tored and partly administered by the Bank’s Partnerships and Cooperation Unit (ORRU).

It assesses the procedural effectiveness of the implementation of the Trust Funds: it attempts to set forth the scope and scale of the procedural issues; distinguish at what stage in the process problems are most critical; and identify the key factors behind the problems and the implications for policy, process, organizational struc-ture and human resource reforms.

operations evaluation departmentAfrican Development Bank Group

2013

Trust Fu

nd M

anagem

ent at th

e African

Developm

ent Ban

k A

n In

depen

den

t ev

Alu

At

Ion

OPEV Trust Fund ENG_cover.indd 1 22/05/13 15.34

The mission of Independent Development Evaluation at the AfDB is to enhance the development effectiveness of the institution in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge

Evaluator General: Rakesh Nangia, [email protected]

Managers: Samer Hachem, [email protected] Rot-Munstermann, [email protected] Amira, [email protected]

Questions? Telephone: +225 2026 2041 Web: http://idev.afdb.orgWrite to us: [email protected] [email protected]

Copyright: © 2015—African Development Bank (AfDB)

68

46

Bringing the AfDB Closer to its Clients: Mid-term review of the Decentralization strategy and Process at the African Development BankAfrican Development Bank

The review asked five questions: Has the Roadmap been implemented as planned? Has decentralization generated the expected benefits? What has decentralization cost the Bank? Should the model for decentralization be changed? What should be the next steps for decentralization?

Independent Evaluation of the AfDB’s Non-sovereign Operations: An influential report, so where did we go from there? Hadiza Sidikou, African Development Bank

The individual, interpersonal and collective (as defined by Gary Henry) outcomes of the NSO evaluation and how the direct and indirect impact of the evaulation helped move the AfDB along the road towards mainstreaming private sector operations.

71

51

56

regional Integration — An Imperative for Greater Development Effectiveness Albert-Eneas Gakusi, African Development Bank

Timely planning to respond to a pressing need for answers; Interaction with stakeholders from the conceptual stage of the evaluation onward; Transparent communication and methodological approaches helped make for an influential evaluation.

Policy-Based Operations — Putting Knowledge to Work African Development Bank

This evaluation was well-timed to feed into the institution’s strategy for PBOs (AfDB Strategy on Policy-Based Operations, 2012), to inform the strategic approach to fragile states (particularly in contributions to the development of budget support to fragile states under the Fragile States Facility) and to governance. The findings and recommendations were also well-received because many fit with ideas for improvement that had echoed through operations for some time, but that had not had formal standing.

Financial struggles in Uganda: Who struggles more and why? Prof. Isabel Günther and Joeri Smits, Center for Development and Cooperation (NADEL), ETH Zürich, Switzerland. Prof. Eva Terberger and Thomas Gietzen †, KFW

This study was motivated by the failure of Schicks’ approach to capture the actual effects of taking up microcredit on house-holds’ financial burden. In addition to measuring the prevalence of financial struggle among households, the study tries to address the question of which effect formal microcredit has on households’ over-indebtedness by applying a control or comparison group design.

There are at least four types of influential evaluations, which can be distinguished based on their use: instrumental, in which the evaluation findings are used to change the evaluand or the conditions that it is working under; conceptual, in which the findings are used to gain conceptual knowledge; informative, in which the findings are used to acquire information; and strategic, in which the evaluation findings are used for advocacy (Levington (2003) and Højlund (2014)). p 20

FONDS

AFRICAIN DE DÉVELOPPEM

ENT

AF

RICAN D

EVELOPMENT FUND

BANQUE A

FRICAINE DE DÉVELOPPEM

ENT

FONDS


ENT

AF

RICAN D

EVELOPMENT FUND

BANQUE A


ENT

Evaluation of the Assistance of the African

Development Bank to Fragile States

Evaluation

of the A

ssistance of th

e African

Developm

ent Ban

k to Fragile States

Operations Evaluation DepartmentAfrican Development Bank Group

2012

This evaluation was undertaken--at the request of ADF deputies —to assess the Bank’s assistance to

fragile states over the 1999-2011 period. The evaluation examines the relevance, effectiveness, and

efficiency of the Bank’s assistance to fragile states. It is based on a literature review, a portfolio review, a

headquarters study, three field country case studies, and three desk country case studies.

Evaluation of the Assistance of the African Development Bank to Fragile States

About the AfDB: The overarching objective of the African Development Bank Group is to foster sustainable economic development and social

progress in its regional member countries (RMCs), thus contributing to poverty reduction. The Bank Group achieves this objective by mobilizing and

allocating resources for investment in RMCs; and providing policy advice and technical assistance to support development efforts.

The mission of the Operations Evaluation Department is to help the Bank to foster sustainable growth and poverty reduction in Africa through

independent and influential evaluations. Such evaluations assess the Bank Group’s policies, procedures and operations, review performance and

report on results in order to draw useful lessons and promote accountability. Director: Rakesh Nangia, [email protected]

Manager, Project Level Evaluations: Mohamed Manai, [email protected]

Manager, High Level Evaluations: Odile Keller, [email protected]

Operations Evaluation Department, African Development Bank

BP 323, 1002 Tunis-Belvedere, Tunisia

Tel : (216) 71 102 841 Fax : (216) 71 194 460Helpdesk: [email protected]

Website: www.afdb.org/opev

24. Fragile states - evaluation report - ENGLISH cover.indd 1

11/05/12 10.04

Independent Evaluation of Non-Sovereign Operations, 2006-2011

SUMMARY REPORT


2013


SUMMARY REPORT


The mission of the Operations Evaluation Department is to enhance the development effectiveness of AfDB initiatives in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge




This evaluation reviews the Bank’s portfolio of 137 investment operations and 38 technical assistance projects approved by the Board between 2006 and 2011. It assesses four aspects of the portfolio: the strategic alignment of the portfolio, the performance of the portfolio, the risk management framework and risk exposures, and institutional efficiency.

The evaluation highlights such positives as the catalytic effect of the Bank’s operations, the risk management framework and growth in private sector operations in LICs. It reports negatives related to the one-Bank approach, the reach to SMEs and MSMEs, and inefficiency in the approval process.

Indepen

dent Evalu

ation of N

on-Sovereign

Operation

s, 2006-2011 SU

MM

ARY

REPO

RT

OPEV NSO english cover.indd 1 22/05/13 11.18

64 Gender Mainstreaming — stuck at the starting Gate?African Development Bank

Timeliness is crucial in producing evaluations that will have substantial influence in improving the effectiveness of development interventions – this analysis came at the right time to make a difference.

Identification Preparation and

formulation

Review and approval

Implementation

Evaluation

Seetharam Mukkavilli, Chief Evaluation Officer, African Development Bank.

Evaluation Influence: An Overview


7The ultimate purpose of evaluation is social betterment, which makes evalua-tion an instrument of change (Henry, 2003). Typically, using evaluation has meant using its results for decision-making – including for developing policy. Non-profit organisations have various uses for evaluations, including, “…program improvement, publicizing activities, reconsidering the mission, raising funds, motivating volunteers and staff, and determining expenditures” (Eckerd and Moulton, 2011: 101). Donor organisations often include project evaluations in their contracts with implementing agencies such as civil society organisations to help determine whether and how to continue support or to provide assurance that the intervention was indeed carried out. Governmental departments and agencies undertake evaluations and share performance-related information to inform legislative and parlia-mentary fora as part of being accountable for taxpayers’ funds. Policymakers, in deciding whether or not to use evaluation information, can use a truth test (Weiss and Bucuvalas, 1981) that considers the research quality leading to the information and whether it can be challenged. Both the process and the findings of an evaluation must be defensible (Cooksy and Caracelli, 2005:31). Participatory evaluations are conducted in collaboration with stakeholders including target groups and end-users, to ‘empower’ and promote collective learning among stakeholders on what works.

Since the 1970s, ‘evaluation influence’ has been the subject of significant research and theo-retical interest (Herbert, 2014:389). It is being developed as an approach to understanding the impact of evaluation. Kirkhart (2000), for example, advocates looking at evaluation influence, although use and influence are complementary approaches to planning, conduct-ing, and studying evaluation. By taking evaluation influence as a frame of reference, evalu-ators can better describe and understand what occurs during and following an evaluation. Influence can also address situations where individuals are unaware of the role played by an evaluation in a change that it has provoked.

Understanding Evaluation InfluenceEvaluation influence connotes a change in pre-and post-evaluation states that can be empiri-cally attributed to the evaluation and influence decisions regarding the implementation, expansion or funding of programs, policies or projects. There are three ways to categorise evaluation influence: (i) Direct or instrumental, connoting the direct effect of an evaluation; (ii) conceptual, when an evaluation changes ways of thinking or provides new understand-ing, and (iii) political or symbolic, when the mere existence of an evaluations and not the results, can be used to persuade or support a viewpoint (Johnson et. al., 2009).

The influence of an evaluation varies by the type, focus, audience and timing of the evalua-tion. Elaborating on the timing of the evaluation, formative evaluations, conducted at the beginning or initial stages of implementation, aim to influence investment decisions, or to alter design, delivery models, priorities and implementation modalities. Mid-term evalu-ations, conducted at the mid-point of project or program cycle, aim to inform decisions about potentially necessary mid-course corrections. Summative evaluations conducted towards the end or after the conclusion of an intervention aim to ascertain the planned and unplanned short-term and intermediary level effects of interventions, also called out-put and outcome levels. Impact evaluations, conducted after the end of an intervention,

eVALUatiOn Matters

seek to ascertain positive and negative planned and unplanned long-term effects, often to influ-ence policy or strategic decisions about replication or the continuity of interventions.

Framework of Influence: Benefits Henry (2000) emphasises the significance of a frame-work of influence to consider the intended and unin-tended impacts of evaluation. This is particularly important when considering more indirect goals (for example, organizational learning and empow-erment) and puts the onus on evaluators to contrib-ute to better social conditions through their work. Herbert (2014:394) provides a framework of evalu-ation influence that includes a range of benefits as quoted hereunder:

• Influence provides a definition and a framework that reflects the full impact of evaluation and a cohesive way to organize theoretical and empiri-cal knowledge of the effect evaluation can have on programs.

• Influence allows for the study of implicit mecha-nisms that affect change, including processes at the individual, interpersonal, and collective levels.

• Influence frameworks are oriented around link-ages to more developed constructs in other fields of literature such as attitude change, priming, skill acquisition, and persuasion.

• Shifting to an influence framework allows for the study of pathways of influence and the study of situations where evaluation failed to affect change.

• Influence is built around social betterment as the ultimate goal of evaluation, rather than use.

Influence PathwaysMark and Henry (2004) define “influence path-ways” as the steps through which evaluations make

a difference. Various studies have attempted to trace these pathways. Based on a study of an emergency response evaluation, Oliver (2008) found that indi-viduals responsible for setting policy agendas were removed from the evaluation process. Evaluation influence was affected by the absence of a culture of learning, lack of institutional memory, and of oppor-tunities for staff to read past reports. Alexander (2003) noted, from an analysis of heath sector evalu-ations, that key differences in influence appear to be mediated by successful collaboration between evalu-ation and program staff. In a study of civil society organisations engaged in monitoring and evaluating government programmes in Ghana, Gildemyn (2014) found that interface meetings created an environ-ment in which influence mechanisms can occur. Appleton-Dyer et al. (2012) analyse evaluation influ-ence in public sector partnerships, noting that the focus, rationale, approach, internal/external evalu-ator role, skills, expertise, and cultural competence, as well as the quality, sophistication, credibility, timeliness, responsiveness and communication of the evaluation – all play a role on its influence. Influential evaluations have been broadly recognized for their effects on public policies and programs. A World Bank (2005:iv) examination of eight case studies looks at “where evaluations were highly cost-effective and of considerable practical utility to the intended users.” The cost-effectiveness of evalu-ations was assessed by identifying and quantifying potential benefits and impacts, and by carrying out an attribution analysis to identify the contribution of the specific evaluation to the changes. Each case study addressed five questions.

1. What were the impacts to which the evaluation contributed?

2. How were the findings and recommendations used?

3. How do we know that the benefits were due to the evaluation and not to some other factors?

4. How was the cost effectiveness of the evaluation measured?

5. What lessons can be learned for the design of future evaluations?

8

Influential evaluations can change attitudes and behaviours, justify policies and public expenditures, empower change agents, and place items on a public agenda (Henry, 2003).


Using a study of three social program evaluations recognized for their influence on public policies and programs, Henry (2003) notes that high techni-cal quality makes it likely that evaluations will be influential. His study seeks to illustrate the potency of tightly designed efficacy studies that yield posi-tive results, thorough and comprehensive quasi-experiments that yield null findings, and experi-ments that compare a prevalent program model to an alternative model. Conceptions of evaluation quality can vary by stakeholder group and type of evaluation. Stufflebeam (2001) discusses several evaluation models where influence is promoted and valued differently by approach or model.

Capacity BuildingEvaluation influence is related to evaluation capacity building (ECB). Stockdill, Baizerman, and Compton (2002:8) define evaluation capacity-building as, “A context-dependent intentional action system of guided processes and practices for bringing about and sustaining a state of affairs in which quality pro-gram evaluation and its appropriate uses are ordinary and on-going practices within and/or between one or more organizations/programs/sites.” They note that demand for ECB is linked to the demand for evaluation, and that its sustainability depends on the extent to which an evaluation is used. High quality evaluations are at the core of ECB (Henry 2003). Cousins et al. propose a common framework for ECB in carrying out and using evaluations. According to them, the knowledge and skills shaping influential evaluations are: “ethics; evaluation planning and design; data collection, data analysis and interpreta-tion; communication and interpersonal skills, and project management” (Cousins et. al., 2014:8).

Fostering InfluenceCousins and Leithwood (1986) consider the factors that contribute to the influence of an evaluation, including identifying stakeholder perception of the appropriateness of an evaluation approach; the evaluation’s methodological sophistication, and the intensity and user involvement in evaluation process. Patton (1997) refers to ‘process use,’ or the result of having participated in an evaluation

process. Preskill, Zuckerman, and Matthews (2003) identify five categories of factors that affect process use: (i) facilitation of evaluation processes; (ii) man-agement support; (iii) advisory group characteris-tics; (iv) frequency, methods, and quality of com-munications, and (v) organization characteristics. The International Initiative for Impact Evaluation (3ie) advocates having grantees use a policy influence plan to identify changes -- in attitude, discourse, procedures, legislation or behaviour -- of key targeted policy stake-holders or influencers such as policymakers, media, civil society organizations, professional associations, trade unions, religious groups, etc.

Lessons for promoting influential evaluations (World Bank, 2005) include • the importance of addressing current policy

concerns• policymaker commitment to using findings• an evaluator’s perceived credibility and

independence• the timing of the evaluation and the commu-

nication of its results to stakeholders to support decision-making

• not exaggerating the importance of an evaluation but making it one of several sources of information for policymaking

• continuous engagement of the client and other stakeholders in the evaluation process inclusion of varied dissemination and communication strate-gies for sharing results

• and strengthened scope and methodology of evalu-ation to cover implementation, outcomes, and the

9

While research can trace the pathways of influence by examining the nature and extent of the use of an evaluation, no clear answers exist for evaluating influence. How is it measured? How can evaluators play a key role in improving the quality of policy decisions?

eVALUatiOn Matters

use of mixed methods. Considering evaluation influence early can improve evaluation planning.

Limitations of Evaluation InfluenceEvaluation influence, as Alkin and White (1979) argue, may take months or even years to become clear. These authors criticise the short reference periods used to assess the use of evaluation. Weiss (1988) points out that it is unrealistic to expect evalu-ation results to be used routinely and asks evaluators not to use instrumental use as an indication of an evaluation’s success or quality. Evidence rooted in evaluation is one among many sources of informa-tion for policymaking including policy discourses and debates called ‘advice’, public campaigns and advocacy, lobbying, and negotiation (Jones, 2011:2).

ConclusionInfluential evaluations can change attitudes and behaviours, justify policies and public expenditures, empower change agents, and place items on a public agenda (Henry, 2003). Given multiple evaluation designs with varied implications for influence, it is vital to remember that evaluative evidence is one of many inputs for policymaking. Studies aimed at ascertaining the impact of evaluations require budg-etary resources and must take systematic, methodo-logically rigorous approaches that eliminate chance factors and establish causality.

While research can trace the pathways of influence by examining the nature and extent of the use of an evaluation, no clear answers exist for evaluating influ-ence. How is it measured? How can evaluators play a key role in improving the quality of policy decisions? What matters more to influence – messages from an evaluation or the messenger in the form of a known authority endorsing the message? Should evaluators directly engage with policy makers? How should studies that examine the influence of evaluations be financed?

Mark and Henry (2004) highlight the importance of sophisticated, credible, timely, responsive evaluation feedback and findings and their communication.

Johnson et al. (2009) suggest that an evaluator’s professional and cultural competences are impor-tant for evaluation influence.

ReferencesAlexander, H. (2003). “Health-service Evaluations:

Should We Expect the Results to Change Practice?” Evaluation 9:405-414.

Alkin, Daillak and White (1979). Using Evaluations: Does Evaluation Make a Difference? Thousand Oaks, C.A.: Sage

Appleton-Dyer, Sarah, Janet Clinton, Peter Carswell and Rob McNeill (2012). “Understanding Evaluation Inf luence Within Public Sector Partnerships: A Conceptual Model”, American Journal of Evaluation 33(4):532-546.

Cooksy, Leslie J. and Valerie J. Caracelli (2005). “Quality, Context, and Use. Issues in Achieving the Goals of Metaevaluation”, American Journal of Evaluation, 26:1, March 2005 31-42.

Cousins, J. B. and K. A. Leithwood (1986). “Current Empirical Research on Evaluation Utilization,” Review of Educational Research, 56:331-364.

Cousins, J. B., S. C. Goh, C. J. Elliott, and I. Bourgeois, (2014). “Framing the Capacity to Do and Use Evaluation.” In J. B. Cousins & I. Bourgeois (Eds.), Organizational Capacity to Do and Use Evaluation. New Directions for Evaluation 141:7-23.

Eckerd, Adam and Stephanie Moulton (2011), “Heterogeneous Roles and Heterogeneous Practices: Understanding the Adoption and Uses of Non-profit Performance Evaluations”, American Journal of Evaluation, 32(1), 98-117.

Gildemyn, M. (2014). “Understanding the Influence of Independent Civil Society Monitoring and Evaluation at the District Level: A Case Study of Ghana.” American Journal of Evaluation. 35:507-524.

Henry, G. T. (2000). “Why not use?” In V. J. Caracelli and H. Preskill (Eds.), “The Expanding Scope of Evaluation Use.” New Directions for Evaluation 88:85-98. San Francisco, CA: Jossey-Bass.

10


Henry, Gary T. (2003). “Influential Evaluations”, American Journal of Evaluation, 24(4):515-524.

Herbert, James Leslie (2014). “Researching Evaluation Influence: A Review of the Literature”, Evaluation Review. 38(5): 388-419.

International Initiative for Impact Evaluation (2015). Accessed on 20 February 2015. Policy Influence. www.3ieimpact.org/en/about/what-3ie-does/policy-influenc/policy-entrepreneurship. Website

Johnson, K., O. G. Lija, S. Toal, J. King, F. Lawrenz, and B. Volkov (2009). “Research on Evaluation Use: A Review of the Empirical Literature 1986-2005”. American Journal of Evaluation, 30:377-410.

Jones, Harry (2011). Background Note: A Guide to Monitoring and Evaluating Policy Influence, Overseas Development Institute, London.

Kirkhart, K. E. (2000). “Reconceptualising Evaluation Use: An Integrated Theory of Influence”. New Directions for Evaluation, 88:5-23.

Mark, M. M. and G. T. Henry, (2004). “The Mechanisms and Outcomes of Evaluation Influence”. American Journal of Evaluation, 10:35-48.

Oliver, M. L. (2008). Evaluation of Emergency Response: Humanitarian Aid Agencies and Evaluation Influence (Ph.D. in Public Policy). Georgia State University, Atlanta, GA.

Patton, M. Q. (1997). Utilization-Focused Evaluation: The New Century Text 3rd ed. Thousand Oaks, CA, London, and New Delhi: Sage.

Picciotto, Robert (2003). “International Trends and Development Evaluation: The Need for Ideas.” American Journal of Evaluation, 24(2):227-234.

Preskill, H., B. Zuckerman, and B. Matthews (2003). “An Exploratory Study of Process Use: Findings and Implications for Future Research.” American Journal of Evaluation, 24, 423-442.

Stockdill, S. H., M. Baizerman and D. Compton (2002). “Toward a Definition of the ECB Process:

A Conversation with the ECB Literature.” In D. Compton, M. Baizerman, and S. H. Stockdill (Eds.), New Directions in Evaluation 93:7-25. San Francisco, CA: Jossey-Bass.

Stufflebeam, D. L. (2001). “Evaluation Models” In New Directions for Evaluation, 89:7-98. San Francisco, CA: Jossey-Bass.

Weiss, C. H. (1988). “If Program Decisions Hinged only on Information: A Response to Patton.” American Journal of Evaluation 9:15-28.

Weiss, C. H. and M. J. Bucuvalas (1981). “Truth Tests and Utility Tests: Decision-makers’ Frame of Reference for Social Science Research.” In H. E. Freeman and M. A. Solomon (Eds.), Evaluation Studies Review Annual (Evaluation Studies Review Yearbook) 6:695-706. Beverly Hills, CA: Sage.

World Bank (2005), Influential Evaluations: Detailed Case Studies, Operations Evaluation Department, Washington, D.C.

Acknowledgement: I wish to thank Foday Turay, Chief Evaluation Officer, IDEV, for help with the reference material for the preparation of the paper; and Albert-Eneas Gakusi, Chief Evaluation Officer, IDEV, for his insightful comments on the draft ver-sion of the paper.

11

Seetharam Mukkavilli, Ph.D., has more than 30 years’ experience supporting and advising international and national organizations.

Seetharam’s areas of expertise include project planning, results-based management, program management, monitoring frameworks, program evaluation, country and joint evaluation, impact evaluation, survey research and web usability testing. He also trains civil servants, private-sector executives and NGO professionals in planning, monitoring and evaluation through short-term courses and workshops. His publications include six books and monographs besides 28 articles in journals. His certifications include Project Management Professional and Certified Quality Auditor.

PrOfIlE Of THE AuTHOr

eVALUatiOn Matters

A N N U A L R E P O R T

2014

expanding influence

Marvin Taylor-Dormond, Director; Independent Evaluation Group – Private Sector; Stoyan Tenev, Manager, Private Sector Evaluation and Monika Weber-Fahr, Chief Knowledge Officer and Senior Manager, World Bank

Evaluation Influence and the Evaluator’s Independence: The Evaluation of World Bank Group Guarantee Instruments


2014

expanding influence



2014

expanding influence

Use and Influence1

Views on evaluation use and influence vary widely.11

Ultimately they reflect how one perceives the nature of evaluation and the role of the evaluator in the process of social change. The main issues can be put into sharp relief by juxtaposing the two extreme positions.

At one end (Scriven, Campbell) is the view that truth and objectivity are the paramount concern of the eval-uator. Use and influence are to be left to the market of ideas. There is no need for the evaluator to be involved in facilitating use. Validity and use are independent and distinct from each other. In fact, proponents of this view argue that, aiming at use and influence may conflict with objectivity and independence.

At the other extreme is the view (an increasingly influential one) that use is everything (Patton). The validity of the evaluation itself depends on use and the effects of use (see American Evaluation Association’s Topical Interest Group website on eval-uation use, http://comm.eval.org/EvaluationUse/Home). Evaluation is an integral part of the causal link of social change, and the evaluator has respon-sibilities for use. This line of reasoning, if pursued consistently, leads to the view that evaluation is a key driver of social change, and the evaluator has direct responsibilities for social betterment.

The first is a narrow and humble view of evalua-tion. The second is a broad view of evaluation that expands the role of the evaluator toward that of a communicator and change agent, perhaps even an organizational improvement coach. For an interest-ing debate between proponents of these views see the 2009 Claremont debate (Donaldson, Patton, Fetterman, and Scriven 2010).

Within these two extreme viewpoints lies a grow-ing literature on use and influence. Evaluators broadly distinguish between two main types of use—process use and substantive use (use of

11 For a review see Shadish and others (1991) and Mark and Henry (2004) for a review of the literature.

findings). The substantive use, in turn, is divided into conceptual, instrumental, and symbolic, as well as internal and external—that is, beyond the particular program and for other similar pro-grams. Evaluators have also identified a long list of factors that are associated with use and influence such as timeliness, credibility, and interactions with the potential users (see Shadish, Cook, and Leviton 1991; Mark and Henry 2004; Fleischer and Christie 2009).12 2

It is important to note that the literature on use and influence deals with ideal use, “where evaluation findings are employed in legitimate ways to sup-port discrete decisions (instrumental use), foster learning (conceptual use), or symbolize or persuade (political use)” (Cousins 2004). In reality, there is use and there is misuse. Process misuse and misuse of the findings correspond to process and substan-tive use respectively. Examples of misuse include commissioning evaluation for political gain or for publicity only (Weiss 1973b), manipulation and coercion of the evaluator or inappropriate suppres-sion of findings (Cousins 2004), using evaluations to delay action (Weiss 1973), changing the wording or evaluative conclusions (Arkin and Coyle 1988), selectively reporting results (Arkin and Coyle 1988; Weiss and Bucuvalas1980). A small but important literature on misuse (Arkin and Coyle 1988; Patton 2008; Fleischer and Christoe 2009) has emerged over the years. What is missing, however, is a uni-fied theory that treats use and misuse in a single framework using a common set of factors to explain

12 Other fields exist, beyond the evaluation field, that worry about use and influence when it comes to knowledge— notably the com-munications field, the knowledge management practitioners, and the learning professionals. The knowledge management field is of particular interest here, not only because the notion of “turning knowledge into business value” is a key objective of any functioning knowledge management system (beyond “creating” and “sharing”) (see O’Dell and Hubert 2011), but also because it introduces a group of potential allies for evaluators to leverage when seeking to increase influence and use of evaluation knowledge. The learning field also provides interesting perspectives, notably about the power of emotional connections between an individual and an insight for the individual’s motivation to use the insight. The research on and increasing use of storytelling as a technique for “packaging” knowledge are testament for the connection being put to use (see Denning 2005).

eVALUatiOn Matters

both phenomena. Without attempting to offer such a theory here, we argue that an evaluator’s independ-ence should be part of such an integrated framework. Influence Presupposes Independence A realistic view of evaluation acknowledges that the process of evaluation is a two-way street—evalua-tors influence, but are also influenced. Some of the influencing experienced by evaluators is legitimate, benign, and even necessary for a good evaluation—and is consistent with independence. But some (as in some of the examples listed above) of it can be pernicious.

Evaluation influence presupposes independence. Only independent variables can be causes. Influence loses meaning if the evaluation simply echoes the preferences of the sponsor or the powerful stake-holders. In such a case, the findings and recommen-dations will be accepted and implemented. But to claim that this amounts to use and influence would be as meaningful as the claim of the king in Saint-Exupéry’s Little Prince (who first asks his subjects what they want to do and then orders them to do exactly what they want) that he has absolute power and is never disobeyed.

It is interesting to note that in spite of this, some influential theorists (Weiss, Patton) come close to such a view. For instance, Weiss says that evalua-tions are “most likely to affect decisions when the

researchers accept the values, assumptions, and objectives of the decision-maker” (Weiss 1973b, p. 41). This seems an exaggeration. Yes, evaluators have to under-stand the val-ues, assump-tions, and

objectives of the decision maker. And they must ensure that the decision maker feels they are under-stood—and therefore motivated to listen and engage. But understand does not necessarily mean accept. Such unconditional acceptance tends to blur the lines between the evaluator and the actor or decision maker. This may contradict the evaluator’s role and impartiality, and the work may lose its capacity to be instrumental, particularly in the case of conflicting interests among key stakeholders.On the contrary, independence seems a necessary condition for influence. Rarely do evaluations get done in fields or on subjects when they are con-sidered by only one stakeholder. Wherever there is more than one stakeholder, there will be differ-ent views and conflicting interests. And it is here where independence and impartiality are particu-larly instrumental—assuring key stakeholders that their diverse perspectives and objectives are listened to and taken into account.

But independence does not guarantee meaningful influence, because it can also be misused. Other factors such as timeliness, credibility, and engage-ment remain important. But they need to leverage independence and impartiality for meaningful influ-ence. Independence being a necessary—but not sufficient—condition for influence, the important question is: What are the steps to be taken, and by whom, to move to influence from there?

The Independent Evaluation of World Bank Group Guarantee Instruments—an Example The independent evaluation of the World Bank Group guarantee instruments (IEG 2009) illustrates some of the points made earlier.

ContextThe evaluation was in response to a request by the Board of Directors of the World Bank Group (the Board). The Board had two main concerns: (i) that the guarantee instruments of the Bank Group were being underutilized; and (ii) that there were overlaps and redundancies in the use of the instruments among the three Bank Group institutions—the

14

The World Bank Group

Guarantee Instruments

1990–2007An Independent Evaluation

The World Bank Group

Guarantee Instruments

1990–2007An Independent Evaluation


15International Bank for Reconstruction and Development/International Development Association (IBRD/IDA), the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA).

In the past, the Board had asked the managements of the three institutions to look into these issues but was not satisfied with how the problem was addressed. The perception was that the tough issues were avoided, and the Board could not get an accu-rate and frank assessment of the issues from the three managements. In other words, there was a case of conflicting interests among key stakeholders, which demanded an assessment from an independ-ent perspective.

What was at stake? What was at stake for the managements of the three institutions? All three Bank Group institutions offer guarantees, but the importance of the instrument differs for each institution. MIGA is a mono-product institution offering guarantees or Political Risk Insurance (PRI) only. IFC offers a full range of products from equity to loans including synthetic political risk insurance. A concordat between MIGA and IFC seeks to eliminate competition by limiting IFC’s ability to offer partial risk guarantees. For IBRD/IDA, the guarantee instrument is a niche product—very similar to MIGA’s PRI. It is neverthe-less IBRD/IDA’s main instrument for supporting private sector projects.

There was concern among the managements that the Board looking into the issues might be a precursor to organizational changes as a way of dealing with overlaps and perceived underutilization. There have been previous attempts, all aborted, to effect organi-zational changes—that is, to merge the guarantee function of the Bank with that of MIGA. MIGA’s level of business was stagnant and the organization was being perceived as not using its capital effectively. As a mono-product organization, MIGA felt squeezed by IFC and IBRD/IDA, who were viewed by MIGA as encroaching on its product space.

Given the stakes, there were efforts by the man-agements of the three institutions to influence the evaluation, and subsequently the findings, in certain directions.

The approach takenThe evaluation team started the process by engag-ing in discussions with Board members to under-stand their concerns and expectations. This was followed by extensive engagement with operational staff (including through a survey), existing and prospective World Bank Group guarantee clients, industry peers (Berne Union members), and other multilateral development banks (MDBs). This was the point in time when, as noted above, the evalu-ation team explored the “values, assumptions, and objectives” of the stakeholders, so that the evaluation could speak to them – without, however, adopting them directly. A key result of these consultations, given the context explained above, was the notion of producing a set of scenarios or models to address the issues, instead of a set of recommendations based on a single framework.

Main findings and recommendationsThe study asked three main questions: (1) Should the World Bank Group be in the guarantee business? (2) Have guarantee instruments in the three Bank Group institutions been used to their potential as reflected in Bank Group expectations and perceived demand? (3) Is the Bank Group appropriately organ-ized to deliver its range of guarantee products in an effective and efficient manner?

The evaluation gave an affirmative answer to the first question. Guarantees have been effective in promot-ing key Bank Group strategic objectives, particularly in facilitating the flow of investment to high-risk sectors and countries. The additionality—or unique contribution of these guarantee instruments—has derived from the Bank Group’s relationship with host countries, its capacity to absorb risks that the private sector is unwilling or unable to bear, and its focus on the objectives of poverty reduction and sustainable development.

eVALUatiOn Matters

16 The answer to the other two questions was essen-tially no, especially regarding the delivery of prod-ucts for political risk mitigation (PRM). The use of guarantee products in each of the three institutions has fallen short of reasonable expectations because of external (demand) and internal (supply) factors. Analysis based on (i) benchmarking with the growth of the market, particularly Berne Union members; (ii) the level and evolution of political risk globally and in developing countries by segments; (iii) natu-ral experiments such as the growth in IFC’s trade finance program indicating that when the product is tailored to client needs growth follows, pointed to a conclusion that internal factors were the binding constraint to a higher level of use.

A closer look at the internal factors for underutiliza-tion revealed (1) competition among the Bank Group institutions for the same clients and of the kind that often imposes additional transaction costs on clients and adds reputational risks for the Bank; (2) weaknesses in the marketing of Bank Group guaran-tees and PRM products that limit client awareness and choice; (3) a range of supply-driven policy and mandate restrictions that inhibit the deployment of Bank Group guarantee instruments in response to evolving client needs; (4) limited internal awareness, skills, or incentives in the World Bank and IFC to use guarantee instruments in relevant situations; and (5) inconsistent pricing of the Bank Group PRM products, which runs the risk of creating market distortions and product differentiation among Bank Group instruments based on price. These findings suggested opportunities to productively expand the use of guarantees.3

As indicated above, to overcome the problems identi-fied, the evaluation provided options and scenarios13 for organizational change, but emphasized the prior-ity of actions that needed to be taken, irrespective of organizational structures. The Board expected options from the Independent Evaluation Group

13 The evaluation analyzed the pros and cons of three alternative perspectives for organizational realignment: the client perspective, the country perspective, and the product perspective.

(IEG), not a single proposal for organizational change. This reflected a desire to preserve flexibil-ity in a potentially tense decision-making situation. The evaluation team did not see strong evidence that any particular organizational option would provide an automatic solution to the problems identified.

Under any scenario, the evaluation recommended that the Bank Group senior management take actions to introduce greater flexibility in the use of guarantee instruments in response to dynamic country and client needs and market developments by: (1) revising existing policies and regulations on guarantees to minimize supply-driven product restrictions and allow product differentiation on the basis of value added; (2) ensuring that adequate incentives exist for staff to offer the full array of Bank Group guarantees and PRM products to pri-vate sector clients; (3) establishing more systematic links between advisory services and the deployment of Bank Group PRM instruments and other prod-ucts, particularly in infrastructure; (4) following a consistent approach to pricing of PRM across its guarantee instruments to avoid potential distor-tions; and (5) strengthening internal awareness of the guarantee instruments and the skills for their use and reducing transaction costs where possible, keeping in mind the importance of maintaining adequate processes and regulations for risk manage-ment and safeguards.

Evidence of use and influence With respect to instrumental use of this evaluation, most of its recommendations have been accepted and largely implemented. Some notable actions taken by Bank Group management include: (i) a joint IFC/MIGA unit was established with responsibilities for marketing and selling MIGA products by IFC staff, leveraging IFC’s business development platform; (ii) the evaluation supported and accelerated a change in the MIGA convention that expanded MIGA’s offer-ings, particularly in introducing the non-honoring of sovereign obligations product; (iii) changes were made in the IBRD/IDA guarantee policies, intro-ducing the partial credit guarantee instrument to IDA countries.


17The use of guarantee instruments has expanded in recent years, although it is not clear to what extent this can be attributed to the actions taken. Particularly noteworthy has been the growth of MIGA projects referred to by IFC. Those accounted for more than a third of MIGA underwriting volume in FY2014 and more than half of MIGA’s projects in that year.

Table 1: Number and volume of projects referred to MIGA by IFC, FY11–14

FY11 FY12 FY13 FY14

Number of projects

2 6 3 12

MIGA guarantee amount, US$ million

161.3 415.5 368.0 1,078.9

Source: The World Bank Group.

MIGA has been recording an impressive growth in recent years, driven to a large extent by the new product that was established through the change in MIGA’s convention—the non-honoring of sovereign obligations (see figure 1).

Growth, although not as pronounced as in the case of MIGA, is also visible in IBRD/IDA guarantees. The growth has been particularly strong in IDA countries in Africa.

In IFC there was no major change. The use of guar-antees for trade finance operations has continued to grow rapidly. No significant innovations have been observed in other types of risk-sharing guarantee products. Risk-sharing facilities continue to experi-ence limited utilization. Most important, the evaluation focused the atten-tion on innovation, tailoring to client needs, and staff incentives and awareness. Management actions in this direction led to expansion in use, reduction of overlaps in product spaces, and reduc-tion of internal tensions.

In addition, there has been significant external use of the evaluation. Other MDBs offer similar products and face similar challenges to those experienced by the World Bank Group. They often look at the World Bank Group for leadership and innova-tion. The evaluation findings and recommenda-tions have been used in developing new guarantee policies at the Inter-American Development Bank (see http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=38197598). Similarly, the African Development Bank (AfDB 2010) and the Asian Development Bank have used the evaluation in developing risk mitigation initiatives, designing policies and approaches to mobilizing private resources for addressing development challenges

7



In IFC there was no major change. The use of guarantees for trade finance operations has continued to grow rapidly. No significant innovations have been observed in other types of risk-‐sharing guarantee products. Risk-‐sharing facilities continue to experience limited utilization.

Most important, the evaluation focused the attention on innovation, tailoring to client needs, and staff incentives and awareness. Management actions in this direction led to expansion in use, reduction of overlaps in product spaces, and reduction of internal tensions.

Figure 2: IBRD/IDA Guarantees, 2004–14


Figure 1: MIGA New Issuance (2004–14)


eVALUatiOn Matters

18

in the context of replenishments for their concession finance windows (AfDB 2010; ADB 2011).

Instrumental use has also been observed in impor-tant practice areas such as transparency in budget processes (Bachenberg and others 2011), energy efficiency (Blyth and Savage 2011) and infrastruc-ture finance (Hilmarsson 2012), where the use of guarantee instruments holds particular promise. Concerning process use and conceptual use, the results have been important as well. There was limited knowledge of the nature of the various guarantee instruments and how they compare across institutions among Bank Group staff. The evaluation introduced a useful taxonomy of the guarantee instruments of the Bank Group. It has

been used to stimulate discussions on innova-tive finance (Fininnov 2014) for development and promoting a better understanding of the financing instruments of the MDBs (Venugopal and others 2013).

This process and conceptual use was made possible by the World Bank Group perspective adopted by the evaluation, rather than the institution-by-institution approach used in the past. The evaluation elevated the issues related to the use of the guarantee instru-ments to the highest level of the policy agenda: the President of the World Bank Group appointed a high-level task force led by the chief financial officer to address the issues identified by the evaluation.

Conclusion Independence is a necessary condition for evalu-ation influence. Of course, influence can be exer-cised from multiple social and organizational roles, including ultimately being part of the decision making process itself. But to be influential on the basis of evaluative knowledge requires impartiality and objectivity, distance from the decision maker, and capacity to deal with conflicting interests. The IEG guarantees evaluation offers a good example of this statement. Designed to examine a number of issues affecting the delivery of guarantees by the Bank Group from an independent perspec-tive, the evaluation ended up having a significant instrumental use, especially in MIGA and on the

7



In IFC there was no major change. The use of guarantees for trade finance operations has continued to grow rapidly. No significant innovations have been observed in other types of risk-‐sharing guarantee products. Risk-‐sharing facilities continue to experience limited utilization.

Most important, the evaluation focused the attention on innovation, tailoring to client needs, and staff incentives and awareness. Management actions in this direction led to expansion in use, reduction of overlaps in product spaces, and reduction of internal tensions.





Independence is a necessary condition for evaluation influence. Of course, influence can be exercised from multiple social and organizational roles, including ultimately being part of the decision making process itself. But to be influential on the basis of evaluative knowledge requires impartiality and objectivity, distance from the decision maker, and capacity to deal with conflicting interests.

19


interaction among the three institutions, as well as important process and conceptual uses with respect to the taxonomy and management of guar-antees in the group.

ReferencesAf DB (African Development Bank). 2010.

“Leveraging ADF Resources

for Private Sector Development.” Discussion paper for ADF-12 replenishment, Abidjan, Côte D’Ivoire. http://www.afdb.org/fileadmin/uploads/afdb/Documents/Boards-Documents/100422%20DP%20New%20Finacing%20Instruments%20ADF-12.FINAL.EN.pd

———. 2013. “Initiative for Risk Mitigation.” http://www.google.com/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=19&ved=0CEgQFjAIOAo&url=http%3A%2F%2Fsustainabledevelopment.un.org%2FgetWSDoc.php%3Fid%3D3015&ei=UUV2VMDPGMyWNozdgcgC&usg=AFQjCNHzs_ryoZeOgppjTG-RkTj1Q52pIQ&sig2=-a9pHwfTTYNbTk_x4iXIFg

Alkin, Marvin C., and Karin Coyle. 1988. “Thoughts on Evaluation Misutilization (and Associated Topics).” Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA, April 5-9, 1988.

ADB (Asian Development Bank). 2011. “Private Sector Development and Private Sector Operations in ADF Countries.” ADF-11 replenishment, Dhaka, Bangladesh. http://www.adb.org/sites/default/files/page/47251/adf-xi-private-sector-development-operations.pdf

Bachenberg, Kelsey, Alizeh Bhojani, Elisa Casey, Youjin Choe, Manmeet Dhami, Cammie Dodson, Julian Fellerman, Neena Goswamy, Thayer Hastings, Kristen Hess, Naomi Joswiak, Giselle Lopez, Jeff Meigs, Akshika Patel, Brittney Riley, Emilia Sternberg, Cassie Tillman, and Sarah Van Houten. 2011. Show Me the Money: Achieving

Economic, Social, and Cultural Rights through Government Budgets. https://digital.lib.washing-ton.edu/researchworks/handle/1773/16494

Blyth, Will, and Matthew Savage. . 2011, “Financing Energy Efficiency: A Strategy for Reducing Lending Risk.” Chatam House, London. http://oxfordenergyassociates.com/wp-content/uploads/2013/03/Chatham-House-energy-efficiency-financing-risk-en.pdf

Cousins, Bradley. 2004. “Commentary: Minimizing Evaluation Misuse as Principled Practice.” American Journal of Evaluation 25 (3): 391–97.

Denning, Stephen. 2005., The Leaders Guide to Storytelling: Mastering the Art and Discipline of Business Narrative. San Francisco: John Wiley and Sons.

Donaldson, Stewart, Michael Patton, David Fetterman, and Michael Scriven. 2010. “The 2009 Claremont Debates: the Promise and Pitfalls of Utilization-focused and Empowerment Evaluation.” Journal of Multidisciplinary Evaluation 6 (13). http://scholarship.claremont.edu/cgu_fac_pub/563/

Fininnov. 2014. Forum des innovations financieres pour le development. http://www.fininnov.org/main.php?page=links http://www.fininnov.org/main.php?page=links

Fleischer, Dreolin, and Christina Christie. 2009. “Evaluation Use

Results from a Survey of U.S. American Evaluation Association Members.” American Journal of Evaluation 30 (2): 158–75.

Hilmarsson, Hilmar Þór. 2012. “Small States and Large Private Sector Investments in Infrastructure in Emerging Market Economies in Partnership with International Financial Institutions.” Innovation, Technology, and Knowledge Management 15: 139–58.

eVALUatiOn Matters

20 IEG (Independent Evaluation Group). 2009. The World Bank Group Guarantee Instruments, 1990-2007. Washington, DC: The World Bank. Inter-American Development Bank. 2013. “Proposed Policy for a Flexible Guarantee Instrument for Sovereign Guaranteed Operations..” http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=38197598

Mark, Melvin, and Gary Henry. 2004. “The Mechanisms and Outcomes of Evaluation Influence.” Evaluation 10(1): 35–57.

O’Dell, Carla, and Cindy Hubert. 2011. The New Edge in Knowledge: How Knowledge Management Is Changing the Way We Do Business. Hoboken, NJ: John Wiley and Sons.

Patton, Michael. 2008. Utilization-Focused Evaluation. 4th edition. Los Angeles: Sage Publications

Shadish, William Jr., Thomas Cook, and Laura Leviton. 1991. Foundations of Program Evaluation, Theories of Practice. London: Sage Publications.

Venugopal, Aman Srivastava, Cliford Polycarp, and Emily Taylor. 2013. Survey of Public Financing Institutions’ Use of Instruments. Washington, DC: World Resource Institute. http://pdf.wri.org/sur-vey_of_public_financing_institutions_use_of_instruments.pdf

Weiss, C.H., 1973a. “Where politics and evaluation research meet.” Evaluation 1: 37–45.

Marvin Taylor Dormond is Director, Independent Evaluation Group--Private Sector, World Bank Group. Marvin joined the World Bank after a career with the Central American Bank of Economic Integration, where he held several positions. He also served as Vice-Minister of Finance for the Republic of Costa Rica. Marvin holds a PhD in Economics from Carleton University and Ottawa University in Canada, an MA in International Affairs from the Norman Paterson School of International Affairs in Canada, and a BA in Economics from the University of Costa Rica.

Stoyan Tenev is Manager, Private Sector Evaluation, World Bank Group. He has held several positions at the IFC, including Head of Macro-evaluations, Chief Evaluation Officer, and lead economist. He was also Chief Evaluation Officer at the European Bank for Reconstruction and Development.

Monika Weber-Fahr is Chief Knowledge Officer and Sr. Manager at the World Bank Group’s Independent Evaluation Group (IEG). Over the past 15 years, Monika has worked in strategy development, innovation and organizational change in IFC, the World Bank and with the Boston Consulting Group (BCG). She holds an MSc in Economics and a PhD in Business Economics.

PrOfIlE Of THE AuTHOrs

———. 1973b. “Between the Cup and Lip.” Evaluation 1(2): 49–55.

Weiss, C.H., and M.J. Bucuvalas. 1980. Social Science Research and Decision-Making. Ncw York, NY: Columbia University Press.


Oscar A. Garcia, Director, Independent Office of Evaluation, International Fund for Agricultural Development (IFAD)

Influential Evaluations: Illustrations from Multilateral Development Organizations

eVALUatiOn Matters

Evaluations can guide decision-making and are therefore vital for improving the quality and results of development work. Influential evaluations in multilateral development organizations are those that contribute to significant change and reform for better development effectiveness.

There are at least four types of influential evaluations, which can be distinguished based on their use: instrumental, in which the evaluation findings are used to change the evaluand or the conditions that it is working under; conceptual, in which the findings are used to gain conceptual knowledge; informative, in which the findings are used to acquire information; and strategic, in which the evaluation findings are used for advocacy (Levington (2003) and Højlund (2014)).

There is ample evidence that evaluations can indeed play a central role in shaping policy and thematic priorities and improving the performance of operations and results on the ground. In this article I will review some influential evaluations based on their instrumental use. Additionally, I will provide an example of a strategic evaluation based on its use for advocacy purpose.

In IFAD, the most influential type of evaluations are the corporate-level evaluations done by the Independent Office of Evaluation (IOE). Corporate-level evaluations include evaluations of IFAD policies, strategies, operations and business processes. Over time, they have provoked far-reaching and systemic institutional change and transformation, contributing to better development effectiveness and results on the ground.

The main objective of corporate-level evaluations is to strengthen IFAD’s accountability frame-work and learning loops for enhanced rural transformation and better livelihoods. While IOE selects the topics for corporate-level evaluations, it ensures they are also a priority to the IFAD Management and its Governing Bodies. This is essential to ensure the full support and coopera-tion of key stakeholders and the ultimate users of corporate-level evaluations.

Depending on the topic covered and whether previous evaluative evidence is available, a corporate-level evaluation in IFAD takes on average from one year to eighteen months to complete. All corporate-level evaluations are presented to the Executive Board, together with a Management Response. The Executive Board, as the main user of evaluations, usually spends adequate time to review and discuss such evaluations, and to provide its strategic guidance on the way forward. All final reports, inclusive of IFAD’s Management Response, are fully disclosed and made publicly available.

In the past decade, three corporate-level evaluations stand out for having had a major impact on IFAD policies, strategies, business processes and operating model. These include the Independent External Evaluation of IFAD (2004/5), and the corporate-level evaluations on the Direct Supervision Pilot Programme (2004/5), and the Field Presence Pilot Programme (2006/7).

The Independent External Evaluation (IEE) was the first evaluation of its kind in IFAD, since the organization started its operations in 1978. The IEE involved the entire organization over a period of about 18 months and allowed staff and Member States to take stock and reflect on the relevance of IFAD’s mission and results, as well as its impact in reducing rural poverty. While the IEE concentrated on the results and impact of IFAD’s activities, it also sought to assess the

22


23effectiveness of key corporate and management pro-cesses, through which IFAD’s policies, programmes and projects have been developed and implemented.

The IEE was closely managed by IOE, but carried out by an external team of consultants. The Director of IOE provided a written progress report on the imple-mentation of the evaluation at the Executive Board sessions that were held throughout the evaluation period. This was important to keep Member States informed of the status of the evaluation and to dis-cuss key issues that were unfolding during the IEE.A steering committee, composed of representatives of nine IFAD Member States, was established to serve in an advisory capacity to the Director of IOE. The IEE benefited from the advice, guidance and com-ments provided by the steering committee, which endorsed the terms of reference for the IEE that were prepared by the Director of IOE, selected the team of consultants and provided comments on all draft IEE reports. It also ensured greater ownership by Member States in the process, and also made sure they duly followed-up to the IEE’s recommendations by the Management.

The main outcome of the IEE was the development by Management of an Action Plan to Enhance IFAD’s Development Effectiveness, approved by the Board, which laid the foundation for important structural and organizational changes, the intro-duction of new corporate policies, and the stream-lining of internal process and procedures. The Action Plan introduced a new strategic framework for the period 2007-2010 to orient the Fund’s invest-ments for rural poverty reduction. Such framework included: (i) a targeting policy to ensure IFAD’s resources were channelled to the most needed; (ii) the establishment of a quality assurance group in the Office of the President and Vice President responsible for quality at entry of new projects and programmes; (iii) the issuance of an annual report on development effectiveness; and (iv) the setting up of guidelines for results-based country strategies. These and other measures have, over the years, contributed to the improvement of results on the ground, as reported by IOE in its Annual Report on Results and Impact of IFAD Operations.

In 1997, the Governing Council adopted a resolu-tion on Loan Administration and Supervision of Project Implementation, together with a five-year plan of action, whereby it decided that 15 IFAD-initiated projects were to be directly supervised and administered by IFAD during a five-year period. The overarching objective of this “Direct Supervision Pilot Programme” was to enable IFAD to acquire first-hand knowledge from supervision

The Director of IOE provided a written progress report on the implementationof the evaluation at the Executive Board sessions that were held throughout the evaluation period. This was important to keep Member States informed of the status of the evaluation and to discuss key issues that were unfolding during IEE.

An Independent External Evaluationof the International Fund

for Agricultural DevelopmentOffice of Evaluation

eVALUatiOn Matters

activities and to incorporate lessons learned from on-going operations more effectively into its pro-ject design work. Prior to the launching of the Direct Supervision Pilot Programme, IFAD was not directly supervising the projects it funded. It used to delegate project supervision to selected cooperat-ing institutions, such as the United Nations Office for Project Services.

The corporate-level evaluation on the Direct Supervision Pilot Programme was completed in 2005. One important message that clearly emerged from the evaluation was that projects directly supervised performed better than those that were supervised by cooperating institutions. The analy-sis also showed that direct supervision allowed the Fund to expand its catalytic objectives of innova-tion, policy dialogue and partnership development. Therefore, wide support was expressed by partners for IFAD to undertake direct supervision.

The corporate-level evaluation therefore recom-mended IFAD to develop a comprehensive super-vision and implementation support policy, which ultimately translated in 2006 into IFAD’s decision to move to direct supervision of projects, which can

be considered one of the most far-reaching changes since the establishment of the Fund in 1977. In fact, the implementation of this evaluation recommen-dation required an amendment to the Agreement Establishing IFAD, by the Governing Council in 2006, which is reflection of the significant change this corporate-level evaluation induced.

Similarly, the Field Presence Pilot Programme was a three-year initiative launched in 2003 with the objective to enhance the effectiveness of IFAD operations by focusing on four interrelated dimen-sions: implementation support, policy dialogue, partnership-building and knowledge manage-ment. Until the Field Presence Pilot Programme was launched, IFAD did not have any permanent in-country presence or country office. All its activi-ties, such as project design and supervision, were managed by the staff based at its headquarters in Rome.

The corporate-level evaluation on the Field Presence Pilot Programme, completed in 2007, assessed the performance and impact of the pro-gramme in achieving IFAD’s overall objectives. While the focus was on the Field Presence Pilot Programme in 15 countries, the evaluation also examined the experience gained at the time with the only two out-posted country programme man-agers (CPMs) in Panama and Peru.

The methodology included, inter-alia, a review of IFAD’s work in 15 countries that were not cov-ered by the Field Presence Pilot Programme, thus allowing the evaluation to assess performance in the countries with and without country presence. Moreover, to enhance the evaluation’s evidence base and given that the evaluation was undertaken on three to four years after the pilot was launched, it also included a comprehensive benchmarking study. The aim of the latter was to learn from the efforts and experience of other multilateral and bilateral organizations and international non-gov-ernmental organizations (NGOs) that had already set up offices in recipient countries.

24

Evaluation of IFAD's Field PresencePilot Programme

C O R P O R A T E - L E V E L E V A L U A T I O N

July 2007

Via del Serafico 107 - 00142 Rome, Italy

Tel: +39 06 54592048 - Fax: +39 06 54593048

E-mail: [email protected]

Web: www.ifad.org/evaluation


25The evaluation concluded that the CPM out-posting model, with the required delegation of authority to advance IFAD’s objectives at the country level, emerged as an effective option and paved the way for the establishment of a fully-f ledged IFAD Country Presence Programme. Since then, IFAD has established around 40 country offices and one regional office in East and Southern Africa, increasing its presence where its beneficiaries need it most. In sum, the corporate-level evaluation on the Field Presence Pilot Programme, similar to the evaluation on the Direct Supervision Pilot Programme, laid the basis for sweeping transformation of IFAD’s organiza-tional architecture and business model.

Moving away from IFAD’s experience to reflect the experience of other multilateral and bilateral devel-opment cooperation agencies, the independent joint evaluation of the implementation of the Paris Declaration conducted in 2011 can be considered a very good example of an influential evaluation, based on its strategic use as an advocacy tool 1.

The evaluation looked at how the principles of aid effectiveness were put into practice by international development partners and what results this was hav-ing in developing countries2. The outcomes of this important international joint evaluation, conducted in two phases (Phase 1: 2007-8; Phase 2: 2009-11), included a synthesis report, 22 country-level evalua-tions, 18 donor studies and seven thematic reviews.

1 The participating countries were: Afghanistan, Bangladesh, Benin, Bolivia, Cambodia, Cameroon, Colombia, Cook Islands, Ghana, Indonesia, Malawi, Mali, Mozambique, Nepal, Philippines, Samoa, Senegal, South Africa, Uganda, Vietnam and Zambia; and the participating development partners were: ADB, AfDB, Australia, Austria, Germany, Denmark, Finland, France, Ireland, Japan, Luxemburg, Netherlands, New Zealand, Spain, Sweden, UK, and UNDG (IFAD, UNAIDS, UNECA, UNDP, UNFPA, UNIFEM).2 The principles of the Paris Declaration are the following: national ownership, alignment with country strategies, harmoni-zation of donors’ actions, managing for development results and mutual accountability. http://www.oecd.org/dac/effectiveness/parisdeclarationandaccraagendaforaction.htm

The first phase of the evaluation con-tributed constructively to the policy debates on aid effectiveness, including the Accra Agenda for Action. The second phase of the evaluation made a synthesis of country and donors headquarter evaluations, plus thematic studies and the evaluations of the first phase. The recommendations to policy makers in both, partner countries, as well in donor countries were presented at the 4th High Level Forum on Aid Effectiveness that took place in Busan, Republic of Korea in 2011.

Managing the evaluation process over four years was a major undertaking involving governments and evaluation departments of 28 countries. In this particular evaluation, the process was as important as the end results, with the recommendations ema-nating from the synthesis report. The establishment of an international reference group, a management group and several national advisory groups allowed for a wide dissemination and advocacy of the Paris principles, but also of international good practices, in conducting independent evaluations. The com-mon approach and methodology allowed for a better understanding of the challenges of implementing the principles of aid effectiveness in very diverse country contexts while reinforcing the principle of mutual accountability. Additionally, as in the case of IOE evaluations, the joint evaluation benefited from an independent assessment of the evaluation under-taken by Michael Quinn Patton from the Evaluator’s Institute that certified the credibility of the report by reviewing its data collection instruments, protocols

“This Evaluation Report provides a credible basis for a constructive discussion in respect of

the reforms to Aid Management by both Partner Countries and Development Partners in

accordance with the Principles enunciated in the Paris Declaration and the Accra Agenda.

The extensive country evaluations based on multiple sources of evidence and techniques,

and carried out in diverse and complex country contexts admirably succeed in testing the

operational commitment of the relevant actors responsible for ensuring improved Aid Effec-

tiveness, and identifies clear and useful norms of good practice to inform future action and

the way forward, in terms of what works and what does not work. An important conclusion of the Report is the realization that successful Aid Reform can only

be achieved through a long-term campaign driven by political commitment rather than

technocratic fixes. It should be stressed at the same time that this should not offer justifica-

tion for the slow pace of change registered to date. There is need in this regard to develop

robust criteria for constant monitoring of progress.”

The 2005 Paris Declaration on Aid Effectiveness calls for “…independent cross-country monitoring and evalu-ation processes to provide a more comprehensive understanding of how increased aid effectiveness contributes to meeting development objectives.” The first phase of the evaluation com-plemented the international monitoring work with a qualitative assessment of progress and obstacles in implement-ing the Declaration in its first two years. It focused on ways to strengthen the performance of both countries and aid providers, and prepared the ground for this second phase evaluation on the ef-fects of better aid in advancing develop-ment objectives.

The evaluation is a multi-partner effort. It comprises 22 country level evalua-tions of how the Declaration’s principles are being applied on the ground, and seven donor and agency studies (in addition to 11 carried out in the first phase) focusing on changes in their policies and guidelines. All the partici-pating countries, donors and agencies volunteered to take part.

The findings and recommendations will be of wide interest: First and foremost to the more than 170 au-thorities that have endorsed the Paris Declaration, primarily the governments of partner countries and ministers and senior managers responsible for development agencies. More broadly, the results should be useful to all who have a stake in ensuring more effective aid: other parts of governments, new and emerging donors, civil society and private sector actors in development, journalists and opinion leaders, as well as managers and operational staff in partner countries and development agencies.

The individual evaluation reports merit wide national and international atten-tion, in addition to the direct value they will have for the countries and agencies where they have been conducted. Their executive summaries are annexed to this report, and the full texts are avail-able in the enclosed CD-ROM.

The Evaluation o

f the Paris D

eclaration

FINA

L REP

OR

TM

AY

2011

Overall strategic guidance for the evaluation was provided by an international Reference Group with broad membership and co-chaired by Malawi and Sweden:

AfghanistanAfrican Development BankAsian Development BankAustralia

AustriaBangladeshBelgiumBeninBoliviaCambodiaCameroonCanadaCivil Society: Better AidCivil Society: Reality of AidColombia

Cook IslandsDenmarkFinlandFranceGAVIGermanyGhanaIndonesiaIreland

Ownership, Alignment, Harmonisation, Results and Accountability

Countries and agencies evaluated in Phase 1 and/or Phase 2Afghanistan • African Development Bank • Asian Development Bank Australia • Austria • Bangladesh • Benin • Bolivia • Cambodia • Cameroon Colombia • Cook Islands • Denmark • Finland • France • Germany Ghana • Indonesia • Ireland • Japan • Luxembourg • Malawi • Mali Mozambique • Nepal • Netherlands • New Zealand • Philippines Samoa • Senegal • South Africa • Spain • Sri Lanka • Sweden • Uganda United Kingdom • UNDP/UNDG • USA • Vietnam • Zambia

A small secretariat, the PDE Secretariat, hosted by the Danish Institute for International Studies was responsible for day-to-day coordination and management of the overall evaluation process. The Secretariat was overseen and guided by a small Management Group comprising Colombia, Malawi, the Netherlands (Co-chair), Sweden, USA, and Vietnam (Co-chair).

Financial support for the overall evaluation effort through a Trust Fund set up for this evaluation was provided by:

Asian Development BankAustraliaAustriaBelgiumCanadaDenmarkFinlandFranceGermanyIreland

Japan LuxembourgMalawiMaliMozambiqueNepalNetherlandsNew ZealandNorwayOECD/DACPhilippinesSamoaSenegalSouth AfricaSpainSwedenSwitzerlandUgandaUnited KingdomUNDPUSAVietnamWorld Bank/IEGZambia

The costs of the individual country and agency evaluations were covered by the individual coun-tries and agencies with additional contributions from the above donors either through the Trust Fund or through bilateral arrangements.

Japan NetherlandsNew ZealandNorwaySpainSwedenSwitzerlandUKUSA

The Evaluation of the Paris Declaration

FinalReport

Phase 2

Ms. Mary Chinery-HesseMember of the African Union Panel of the Wise and Former Chief Advisor to the President of Ghana

Lord Mark Malloch-BrownFormer Administrator of UNDP and Former UK Minister

Executive Summary

“This Evaluation Report provides a credible basis for a constructive discussion in respect of

the reforms to Aid Management by both Partner Countries and Development Partners in

accordance with the Principles enunciated in the Paris Declaration and the Accra Agenda.

The extensive country evaluations based on multiple sources of evidence and techniques,

and carried out in diverse and complex country contexts admirably succeed in testing the

operational commitment of the relevant actors responsible for ensuring improved Aid Effec-

tiveness, and identifies clear and useful norms of good practice to inform future action and

the way forward, in terms of what works and what does not work. An important conclusion of the Report is the realization that successful Aid Reform can only

be achieved through a long-term campaign driven by political commitment rather than

technocratic fixes. It should be stressed at the same time that this should not offer justifica-

tion for the slow pace of change registered to date. There is need in this regard to develop

robust criteria for constant monitoring of progress.”

The 2005 Paris Declaration on Aid Effectiveness calls for “…independent cross-country monitoring and evalu-ation processes to provide a more comprehensive understanding of how increased aid effectiveness contributes to meeting development objectives.” The first phase of the evaluation com-plemented the international monitoring work with a qualitative assessment of progress and obstacles in implement-ing the Declaration in its first two years. It focused on ways to strengthen the performance of both countries and aid providers, and prepared the ground for this second phase evaluation on the ef-fects of better aid in advancing develop-ment objectives.

The evaluation is a multi-partner effort. It comprises 22 country level evalua-tions of how the Declaration’s principles are being applied on the ground, and seven donor and agency studies (in addition to 11 carried out in the first phase) focusing on changes in their policies and guidelines. All the partici-pating countries, donors and agencies volunteered to take part.

The findings and recommendations will be of wide interest: First and foremost to the more than 170 au-thorities that have endorsed the Paris Declaration, primarily the governments of partner countries and ministers and senior managers responsible for development agencies. More broadly, the results should be useful to all who have a stake in ensuring more effective aid: other parts of governments, new and emerging donors, civil society and private sector actors in development, journalists and opinion leaders, as well as managers and operational staff in partner countries and development agencies.

The individual evaluation reports merit wide national and international atten-tion, in addition to the direct value they will have for the countries and agencies where they have been conducted. Their executive summaries are annexed to this report, and the full texts are avail-able in the enclosed CD-ROM.

The Evaluation o

f the Paris D

eclaration

FINA

L REP

OR

TM

AY

2011

Overall strategic guidance for the evaluation was provided by an international Reference Group with broad membership and co-chaired by Malawi and Sweden:

AfghanistanAfrican Development BankAsian Development BankAustralia

AustriaBangladeshBelgiumBeninBoliviaCambodiaCameroonCanadaCivil Society: Better AidCivil Society: Reality of AidColombia

Cook IslandsDenmarkFinlandFranceGAVIGermanyGhanaIndonesiaIreland

Ownership, Alignment, Harmonisation, Results and Accountability

Countries and agencies evaluated in Phase 1 and/or Phase 2Afghanistan • African Development Bank • Asian Development Bank Australia • Austria • Bangladesh • Benin • Bolivia • Cambodia • Cameroon Colombia • Cook Islands • Denmark • Finland • France • Germany Ghana • Indonesia • Ireland • Japan • Luxembourg • Malawi • Mali Mozambique • Nepal • Netherlands • New Zealand • Philippines Samoa • Senegal • South Africa • Spain • Sri Lanka • Sweden • Uganda United Kingdom • UNDP/UNDG • USA • Vietnam • Zambia

A small secretariat, the PDE Secretariat, hosted by the Danish Institute for International Studies was responsible for day-to-day coordination and management of the overall evaluation process. The Secretariat was overseen and guided by a small Management Group comprising Colombia, Malawi, the Netherlands (Co-chair), Sweden, USA, and Vietnam (Co-chair).

Financial support for the overall evaluation effort through a Trust Fund set up for this evaluation was provided by:

Asian Development BankAustraliaAustriaBelgiumCanadaDenmarkFinlandFranceGermanyIreland

Japan LuxembourgMalawiMaliMozambiqueNepalNetherlandsNew ZealandNorwayOECD/DACPhilippinesSamoaSenegalSouth AfricaSpainSwedenSwitzerlandUgandaUnited KingdomUNDPUSAVietnamWorld Bank/IEGZambia

The costs of the individual country and agency evaluations were covered by the individual coun-tries and agencies with additional contributions from the above donors either through the Trust Fund or through bilateral arrangements.

Japan NetherlandsNew ZealandNorwaySpainSwedenSwitzerlandUKUSA

The Evaluation of the Paris Declaration

FinalReport

Phase 2

Ms. Mary Chinery-HesseMember of the African Union Panel of the Wise and Former Chief Advisor to the President of Ghana

Lord Mark Malloch-BrownFormer Administrator of UNDP and Former UK Minister

eVALUatiOn Matters

and processes, as well as the report’s findings and technical notes.

In conclusion, there are three further aspects of influential evaluations that are worth highlighting to ensure their usefulness and credibility. Firstly, the timeliness of topics chosen is essential, so that evaluations can help the organization at the cor-rect juncture. If an evaluation is done too early or too late, it is not likely to have the same impact on organizational development and its overall activities.

Secondly, the evaluations must be linked to further institutional or policy making processes, such as the formulation of a new policy on the topic or planned transformations to the operating model. This will allow evaluation lessons and recommen-dations to feed more immediately into specific processes of awareness raising, reform and change for better development impact.

Thirdly, in all cases the evaluations were per-ceived to be of good quality, with sound methods and methodology. Experience has also shown that it is important for external reviewers with distinguished credentials to be associated with corporate-level evaluations. Their role is to review key evaluation deliverables and to prepare a short final report attesting to the quality of the evalua-tion in terms of the methodology adopted, process followed and final outcomes. In this regard, IOE systematically mobilizes the valuable insights and inputs of prominent evaluators and development practitioners.

For evaluations to be influential, among other issues, they must be used, address critical areas of importance to major stakeholders, be delivered in timely manner, and build on the end users’ inputs and concerns to change the conditions under which the evaluand is working. The exam-ples provided are an illustration that evaluations can guide decision-making and are critical for improving the quality and results of development work of multilateral organizations, as well as for

advancing the understanding and implementation of internationally agreed principles on topics such as aid effectiveness.

ReferencesHøjlund, S. 2014. Evaluation Use in Evaluation

Systems: The Case of the European Commission. In Evaluation: the international journal of theory, research and practice. Volume 20. London.

IFAD Independent Office of Evaluation. 2005. Independent External Evaluation (2004-2005). Rome.

IFAD Independent Office of Evaluation. 2005. Corporate-level evaluation of the Direct Supervision Pilot Programme, Rome.

IFAD Independent Office of Evaluation. 2007. Corporate-level evaluation of the Field Presence Pilot Programme. Rome.

IFAD Independent Office of Evaluation. 2013. IFAD’s institutional efficiency and efficiency of IFAD-funded operations: Corporate-level evaluation. Rome.

Levington, L.C. 2003. Evaluation Use: Advances, Challenges and Applications. American Journal of Evaluation. Vol.24/4: 525-535. Washington, D.C..

World Bank Group, OED Operations Evaluation Department. 2005. Inf luential Evaluations: detailed case studies. Washington DC.

Wood, B., D. Kabell, F. Sagasti, and N. Muwanga. 2008. Synthesis Report on the First Phase of the Evaluation of the Implementation of the Paris Declaration, Copenhagen..

Wood, B., J. Betts, F. Etta, J. Gayfer, D. Kabell, N. Ngwira, F, Sagasti, and M. Samaranayake. 2011. The Evaluation of the Paris Declaration, Final Report. Copenhagen.

26


Making Evaluation Influential at the African Development BankA conversation with Rakesh Nangia, AfDB Evaluator General

Rakesh Nangia, Boubacar Ly and Foday Turay, African Development Bank


eVALUatiOn Matters

Evaluations provide a useful basis for accounting to shareholders and to the public for investments and results. Evaluation results are used to influence policies, strategies, practices, processes, programs, and projects. Independent Development Evaluation (IDEV), the evaluation unit at the African Development Bank (AfDB), undertakes evaluations to help improve the development effectiveness of the AfDB in its RMCs. The conditions of demand and supply must be right for IDEV evaluations to be used and for them to be influential.

For this issue of eVALUation Matters, Foday Turay and Ly Boubacar interviewed Rakesh Nangia about the influence of IDEV evaluations and how key stakeholders can help make them more influential.

Since you joined IDEV, you have constantly pushed the department to produce influential evaluations. What constitutes an influential evaluation at the Bank?

Evaluator-General: Context is essential in understanding the influence of evaluation at the Bank. IDEV evaluations are submitted to the Bank’s Board of Executive Directors through the Committee on Operations and Development Effectiveness (CODE). The evaluations are meant for use by the Board as well as by management, staff and stakeholders. While evaluations are not subject to the approval of Bank management and staff, both groups are the key users of evaluations and thus remain the most important audience. This makes management critical for the influence of evaluations.

With this in mind, let me now focus on the key demand and supply factors that create an influential evaluation. On the demand side, audience ownership is essential for ensuring influence not only in terms of interest and appreciation but also in terms of future utiliza-tion of the findings and recommendations. Effective engagement is important, especially during the critical stages of the process when evaluation questions are being defined and later on when recommendations are being formulated. An influential evaluation is accepted by the Board/CODE, but management must own and use it to make a difference.

Audience demand can be explicit or latent. Some evaluation issues and information gaps are so obvious that an audience says: “Yes, we think an evaluation here would be helpful.” Other information gaps may be less obvious. Here, demand can also be created. Take

Apple, for example. Everyone was happy with CDs and functioning CD players so there was no explicit demand for a “music revolution,” yet once demand was created, the entire industry changed –from products to our listening behavior.

Given the different interests of our two primary audiences, there may occasionally be conflicting demands regarding evaluation. Management may prefer an evaluation or a focus on a particular issue in time, whereas Board members might say: “We need IDEV to shift this evaluation focus because we think that this is where the shoe is pinching”.

28

Thematic Evaluation Project Cluster Evaluation

Regional Integration Stra

tegy

Evaluation

Project Perfo

rmance Evaluation

(Public Secto

r)Impact Evaluation

Project Performance Evaluation

(Private Sector)

Coun

try S

trate

gy E

valu

atio

n

Evaluation Synthesis

Corporate Evaluation

Sect

or E

valu

atio

nSe

ctor

Eva

luat

ion


29We need to understand these differences, which must be made explicit and ironed out to ensure audience ownership of the evaluation.

Now, let us examine the supply side. There is a saying that where there is an effective evaluation demand, evaluation supply will respond accord-ingly. But supply is not just doing an evaluation; it is about appropriately designing, conducting, and managing an evaluation to ensure its qual-ity, acceptability, ownership, and use by its audi-ence. Both the evaluation team and its methods are vital for evaluation quality: our evaluation teams include experts in evaluation; their evalu-ation methods appropriately mix quantitative and qualitative methods that enhance the breadth and value of the evidence.

Ownership does not come from an “easy evalua-tion” that avoids hard-hitting results and seeks to placate its audience. In my experience, manage-ment needs candid, credible, evidence-based evalu-ations. They may not be happy with hard-hitting evaluations, but the key people in management prefer deep constructive criticism rather than no criticism or simple confirmation that “everything is going well”. However, the evaluation must be credible: evidence-based, established on the basis of sound methodology and rigorous analysis. So the “supply-side” is equally challenging and both supply and demand are interdependent. Do any recent IDEV evaluations come to mind when you think about influential evaluations?

Evaluator-General: The Fragile States (p. 34) and the NSO evaluations (p. 46) both come to mind. Both were timely, evidence-based, credible, and owned by senior management and CODE, who were not only highly interested but who also responded con-cretely. Senior management and the President made institutional and structural changes well beyond the reach of the recommendations. For the fragile states evaluation, the President established a High Level Panel for Fragile States chaired by Her Excellency,

President Ellen Johnson Sirleaf. And for the NSO evaluation, the Bank’s institutional structure itself was amended, with the creation of a new department with clear roles and responsibilities.

An influential evaluation implies varying degrees of engagement from IDEV and from the evalua-tion audiences. What roles do you expect IDEV management, Bank management and evaluation teams to play?

Role of IDEV Management Evaluator-General: I expect IDEV management to focus on four things:

1. Building the evaluation demand side especially by constructively engaging with Bank management to identify topics or issues where evaluation(s) would help enhance the design and delivery of Bank support to its RMCs. IDEV management has to put more effort into revealing hidden demand, enabling the identification of implicit and poten-tially valuable evaluations, and ensuring that Bank Management and CODE accept them.

2. Adopting a forward-looking approach in order to foresee the evaluative information that will be important for the institution. Take the example of inclusive and green growth, which are part of the Bank’s ten-year strategy. If the Bank does a highway project tomorrow, IDEV can help define how that project could differ from a traditional project, rather than simply acknowledging the decision as a managerial prerogative and evaluat-ing the project in five years. Chronic and emerg-ing development challenges have implications for latent and potential evaluation demands. IDEV management can actualize these demands and provide the basis for evaluation influence. It must also contribute to the follow-up of evaluation recommendations.

3. Guiding the evaluation team and ensuring that an evaluation is adequately funded. Guidance means empowering the task manager to effec-tively and efficiently deliver the evaluation on

eVALUatiOn Matters

30 time and providing the enabling environment for the evaluation team. Management must ensure the evaluation’s credibility, quality, and timeli-ness through fit-for-purpose evaluation govern-ance, management, team competencies, quality assurance, and funds.

4. Keeping its ears to the ground and reacting in real time--this is essential in a fast-changing environment.

Role of the Evaluation Team From the evaluation team, I expect professional-ism, a credible evaluation process, and high-qual-ity, actionable evaluation results. Professionalism includes being open-minded, objective, efficient, and judgmental only when there is available, cred-ible evidence. Professionalism lends credibility to the evaluation process and quality, thus contribut-ing to the acceptance and use of the evaluation. Professionalism is essential for asking and answer-ing the right questions, for using the right design, methods and processes, and for producing the right outputs at the right time for the audience. This means engaging with the audience and keeping it in mind and being evidence-focused throughout the evaluation process to bring out and report nothing but the truth.

Role of Bank Management I expect Bank management to engage in the evalu-ation effectively and constructively, especially in defining the questions, constructing and imple-menting the recommendations, and facilitating access to data and information for the evaluation. Engagement should be open, be based on the avail-able facts, and provide space for disagreement. Reference group meetings offer opportunities for fair discussions. Recommendations should be discussed and formulated jointly by Bank man-agement and the evaluation team on the basis of findings to ensure that the proposed recommen-dations are the most appropriate and actionable. Bank management should stick with even the hard-to-implement actions once they have been agreed.

It should implement their spirit and not just the letter. In the Fragile States and NSO evaluations, for example, management went well beyond the evaluation recommendations.

What recent IDEV evaluation comes to mind when you talk about hidden evaluation demands? And what is your take on public-private partnerships as embodying hidden evaluation demands?

Evaluator-General: IDEV’s recent evaluation of the Bank’s management of trust funds comes immedi-ately to mind. IDEV saw the need for this evaluation and was convinced that it would make sense to Bank management and to CODE. It constructively engaged management and CODE, managed to get their buy-in early, and successfully conducted and disseminated the evaluation. Another similar evalu-ation was one conducted on the Bank’s Economic and Sector Work portfolio.

With regard to PPPs in the Bank, this definitely embodies latent and potential evaluation demands because they are innovative; they are growing into a huge portfolio, and above all, may help provide responses to many unanswered issues and questions. Formative evaluations, forward-looking retrospec-tive evaluations, and evaluation syntheses from IDEV can help provide answers to these questions.

You have emphasized Bank management’s owner-ship of evaluation recommendations for influential evaluations. Can you elucidate further on this?

Evaluator-General: If we expect the spirit of the actions to be implemented, management must own the recommendations and go beyond their mechanical and literal implementation. Although the higher authority in CODE will make sure that an action gets implemented and IDEV will bring what is not being implemented to CODE’s attention, the implementation of the spirit of what is agreed is far more likely to take place if management owns the recommendations. And that gives us a very strong chance of successfully addressing the problems or


31underlying issues. Let us keep in mind that improv-ing development outcomes is the common objective and that it is in everybody’s interest.

In effect, you are saying that audience ownership of recommendations is fundamental for an evalu-ation to be influential, and that the evaluation team is to foster audience ownership?

Evaluator-General: Indeed. You never want to sur-prise your key evaluation audiences with results that put them in an embarrassing light. Let me give you an example. In my previous life, senior manage-ment, including myself, learned about the results of a hard-hitting evaluation from the media. There was no reason to bring an evidenced-based quality evaluation to senior management through an exter-nal source. For us to succeed, we must have honest, open dialogue with management in an environment of trust based on a clear understanding that all of us are trying to do the right thing for the Bank and for our RMCs. There is no room for a “Got-you” attitude.

How do you ensure that management implements the recommendations?

Evaluator-General: We use a two-pronged approach.

First, we promote management’s ownership of the recommendations. As I have already mentioned, IDEV should not be pushing its views about how to solve the issues or problems that it has identified, but rather bring its evaluation findings or diagnosis to the discussion table and help management define and formulate the appropriate actions to address them. Management knows the context, it knows what resources are available, and what is and is not feasible. Furthermore, there may be several ways to solve the problem and we need to find the appropriate path jointly. Actions agreed and owned by management make implementation far easier. Second, we will use the management action record system that is currently being tested, to track and report on the implementation of agreed actions to CODE for appropriate action.

FONDS


ENT

AF

RICAN D

EVELOPMENT FUND

BANQUE A


ENT

FONDS


ENT

AF

RICAN D

EVELOPMENT FUND

BANQUE A


ENT

Evaluation of the Assistance of the African

Development Bank to Fragile States

Evaluation

of the A

ssistance of th

e African

Developm

ent Ban

k to Fragile States


2012











report on results in order to draw useful lessons and promote accountability. Director: Rakesh Nangia, [email protected]


Manager, High Level Evaluations: Odile Keller, [email protected]



Tel : (216) 71 102 841 Fax : (216) 71 194 460Helpdesk: [email protected]


24. Fragile states - evaluation report - ENGLISH cover.indd 1

11/05/12 10.04


SUMMARY REPORT


2013


SUMMARY REPORT


The mission of the Operations Evaluation Department is to enhance the development effectiveness of AfDB initiatives in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge




This evaluation reviews the Bank’s portfolio of 137 investment operations and 38 technical assistance projects approved by the Board between 2006 and 2011. It assesses four aspects of the portfolio: the strategic alignment of the portfolio, the performance of the portfolio, the risk management framework and risk exposures, and institutional efficiency.

The evaluation highlights such positives as the catalytic effect of the Bank’s operations, the risk management framework and growth in private sector operations in LICs. It reports negatives related to the one-Bank approach, the reach to SMEs and MSMEs, and inefficiency in the approval process.

Indepen

dent Evalu

ation of N

on-Sovereign

Operation

s, 2006-2011 SU

MM

ARY

REPO

RT

OPEV NSO english cover.indd 1 22/05/13 11.18


An Independent evAluAtIon


The mission of the Operations Evaluation Department is to enhance the development effectiveness of AfDB initiatives in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge.





An Independent evAluAtIon This evaluation reviews the establishment and implementation of 28 bilateral and thematic trust funds moni-tored and partly administered by the Bank’s Partnerships and Cooperation Unit (ORRU).

It assesses the procedural effectiveness of the implementation of the Trust Funds: it attempts to set forth the scope and scale of the procedural issues; distinguish at what stage in the process problems are most critical; and identify the key factors behind the problems and the implications for policy, process, organizational struc-ture and human resource reforms.

operations evaluation departmentAfrican Development Bank Group

2013

Trust Fu

nd M

anagem

ent at th

e African

Developm

ent Ban

k A

n In

depen

den

t ev

Alu

At

Ion

OPEV Trust Fund ENG_cover.indd 1 22/05/13 15.34

eVALUatiOn Matters

32 Role of CODE (Executive Board of Directors) How do you envisage the role of CODE?

Evaluator-General: CODE is the authority to which we report and it has overarching responsi-bility for what IDEV does. CODE is both judge and jury. Fundamentally, CODE must accept and adopt an evaluation for it to be influential. They must be convinced. For this to happen, the evaluation must be of high quality; it must be credible, and withstand all questioning and internal and external scrutiny. So long as CODE can ensure that IDEV is adequately protected, IDEV can speak truth to power without fear of retribution.

Apart from ‘owning’ the evaluation, and protecting the evaluation function, what else do you expect from CODE?

Evaluator-General: I expect CODE’s use of the evaluation to go beyond the primary stage. Take our recent evaluation of the Bank’s support to

Botswana, a middle-income country, which has obvious lessons for the design of the next Bank country strategy paper for Botswana and for other middle-income countries. When the new CSP for Botswana was presented for discussion at the Board, all CODE members and many other Board members insisted that the lessons from the evaluation be incorporated into the new CSP. By drawing from some of the pertinent lessons from our Botswana CSP evaluation to inform the discussion of other middle-income country CSPs, CODE enhanced the influence of this evaluation and went beyond the primary stage of evaluation use. I am delighted to report that I see this happening increasingly in the Board room where EDs are drawing on lessons from evaluations in their deliberations. On the resource side, adequate resourcing and especially budgeting for the evaluation function is absolutely important for us since adequate resources must be provided for any evaluation to be credible, of high quality, and influential.

Role of Regional Member Countries

How about the role of RMCs, the final but not least important audience?

Evaluator-General: The RMC audience is exten-sive, as it includes primary, secondary, and tertiary beneficiaries, CSOs, other development partners, and political institutions. I expect government officials, in particular, to participate in making our evaluations influential by helping to define evaluation issues and questions, to discuss evalu-ation findings, and to enable the environment for effective and efficient data collection and analysis. I expect some RMC audiences to find our evaluations useful with respect to processes and/or findings. If the RMCs were to use our evaluations, their influence would be enhanced, but I have not seen many evaluations yet that have grabbed the inter-est of RMCs. We definitely need to raise our level of engagement with RMCs in our evaluations to increase their influence.

From experience to knowledge...

From knowledge to action...

From action to impact

Botswana:

Country Assistance Evaluation 2004–2013Summary Report

An ID

EV C

ount

ry S

trat

egy

Eval

uatio

n


33

Rakesh Nangia is the Evaluator General at the African Development Bank. Prior to joining the AfDB, he spent 25 years at the World Bank, where he held several positions including Director of Strategy and Operations for the Human Development Network and Acting Vice-President for the World Bank Institute. He attended the Indian Institute of Technology in Delhi and Harvard University and holds degrees in business administration and engineering.

Foday TURAY is a Chief Evaluation Officer, Independent Development Evaluation, African Development Bank. Prior to joining the AfDB, he worked as a Coordinator/Projects Officer (Food and Agriculture Organization of the United Nations), Senior Economist and Monitoring and Evaluation Officer (Government of Sierra Leone) and Consultant. Foday is trained in development evaluation, macroeconomic modeling and development oriented research, and holds a Ph.D. in Economics (University of Portsmouth, UK).

Boubacar LY joined IDEV in February 2014 as a Consultant. He provides assistance on Bank project and program evaluations and project completion report (PCR) validations. He is also involved in thematic, sector, and country strategies and programs evaluations. Before joining IDEV, he spent eight years in academia and in the banking sector in Toronto, Canada. He also interned with the AfDB in 2004, and with the World Bank in 2008. He holds an MA and a BA in International Development Studies, and a Specialized Honours BA in Economics.

PrOfIlE Of THE AuTHOrs

Any additional thoughts on enhancing the influ-ence of IDEV evaluations in RMCs?

Evaluator-General: My hope is that we increase the sphere of influence of our evaluations especially in advocating themes relevant to RMC develop-ment— and effective advocacy requires credible, quality evidence. Evaluation can contribute to this objective.

• I want our evaluations to inform discussion fora. Let us take the case of governance and gender. I used several gender evaluations to inform a recent senior management discussion on gender

at headquarters, and the discussion was won-derful and instructive. From an evaluation per-spective, having information from evaluation synthesis and/or other evaluations about what did or did not work and why contributes to dis-cussions on pertinent themes concerning RMCs

• In addition, evaluations can help support the advocacy function of the Bank’s field offices. Our challenge is to pull pertinent knowledge from our evaluations and from those of others and to ensure that this is easily available to colleagues in the Bank’s field offices. I certainly see this as helping greatly with the policy dialog.

34

eVALUatiOn Matters

fragile states—Back from the Edge

35Fragile states represent one of the most compelling challenges facing the world today, and the African Development Bank plays an important role in meeting this challenge. At the time the Evaluation of the Assistance of the African Development Bank to Fragile States was conducted, about 40 percent of all African countries were classified as fragile states, which presented the Bank with a formidable task, with high stakes. It was important that the Bank get it right. To address this challenge, IDEV carried out—at the request of African Development Fund deputies— an evaluation to assess the Bank’s work (over the 1999–2011 period) in meeting the needs of member states struggling with fragility. The evaluation paid close attention to the period since the 2008 adoption of the Strategy for Enhanced Engagement in Fragile States.

What makes this a Fit-For-Purpose Evaluation?For an evaluation to have influence, its timing must be carefully calibrated – this was the case with the evaluation of the Bank’s work in fragile states. The evaluation synthesis and accompanying materials were completed in time to serve as a building block for the new Bank strategy for fragile states, and to provide information and insight to the High-Level Panel on Fragile States.The evaluation methodology met good practice standards, but, equally important, the evaluation team put the unique nature of fragile states front and center in the evaluation. Fragile states present a complex and dynamic environment for development assistance. As Ellen Johnson Sirleaf, chair of the High-Level Panel on Fragile States noted, “The special circumstances of fragile states need to be addressed. They are simply not the same as countries that are normal” (http://www.afdb.org/fileadmin/uploads/afdb/Documents/Generic-Documents/Fact%20sheet.pdf). The evaluation team took this into account, which allowed them to create an evaluation that will have long-lasting influence.

How were the Findings of the Evaluation Used?The findings of the evaluation synthesis have already seen broad use across the Bank and the larger development community, as intended. The Bank responded to the evaluation, in part, by establishing the High-Level Panel on Fragile States, to review the Bank’s current arrangements and update operational frameworks for engaging in fragile states. The Panel’s final report, Ending Conflict and Building Peace in Africa, a Call to Action; the Development Effectiveness Review 2012—Fragile States and Conflict-Affected Countries; and Addressing Fragility and Building Resilience in Africa: The African Development Bank Group Strategy, 2014–2019 all made good use of the information provided by the evaluation. The Bank also expressed its intention to monitor and evaluate performance in fragile states, supported by IDEV, to ensure continuous learning and accountability.

Lessons for Fit-For-Purpose EvaluationsThe fragile states evaluation synthesis underlined the importance of building an evaluation design that takes full account of the unique nature of the issues confront-ing fragile states. In these countries, it cannot be business as usual.

The imposition of the “fragility lens” allowed evaluators to see the need for flexibility and speed in dealing with these nations, as well as the importance of strengthening programming an internal organization at the Bank to maximize effectiveness, along with heightened results monitoring.

This experience illustrates the importance of engaging all stakeholders in the evaluation process, from selection of the topic to production of the final report. The broad acceptance of the findings and recommendations of this

FONDS

AFRICAIN DE DÉVELOPPE

MENT

AFRI

CAN DE

VELOPMENT FUND

BANQUE

AFRIC

AINE DE DÉVELOPPEMENT

FONDS

AFRICAIN DE DÉVELOPPE

MENT

AFRI

CAN DE

VELOPMENT FUND

BANQUE

AFRIC

AINE DE DÉVELOPPEMENT


Evaluation

of the A

ssistance of th

e African

Developm

ent Ban

k to Fragile States


2012











report on results in order to draw useful lessons and promote accountability. Director: Rakesh Nangia, [email protected], Project Level Evaluations: Mohamed Manai, [email protected]

Manager, High Level Evaluations: Odile Keller, [email protected] Evaluation Department, African Development BankBP 323, 1002 Tunis-Belvedere, TunisiaTel : (216) 71 102 841 Fax : (216) 71 194 460Helpdesk: [email protected] Website: www.afdb.org/opev

36

eVALUatiOn Matters

Main Findings, Recommendations and MethodologyFindings A number of the findings of the evaluation were quite positive. It noted that over the past decade, the Bank has developed a more explicit and systematic approach to working in fragile states. It has raised awareness of the special needs of fragile states in its activities, and allocated substantial additional financial resources to respond to those needs. Since 2008, the Bank has taken a number of steps to improve and differentiate its support to fragile states. The eligibility criteria have been applied transparently and, on occasion, a flex-ible approach has been adopted in response to need.

The Bank’s instruments and modalities have responded to a good range of country needs and capacities, augmented by new support introduced in 2004 with the Post-Conflict Countries Facility, and further strengthened in 2008 with the Fragile States Facility. The Bank’s regular and special programs have delivered significant results in arrears clearance, infrastructure rehabilitation, and some areas of capacity development. The Bank’s use of budget support has been support-ive of country governments emerging from con-flict. Nevertheless, by the AfDB’s conventional efficiency measures, the picture is mixed–fragile situations require quicker and more flexible action.

Recommendations1. A broader programmatic approach is needed for fragile and stabilizing situa-tions, to engage where the essential func-tions and resiliency of state, society, and/

or the economy are severely impaired or critically vulnerable to shocks, or where recovery from major shocks is still under way. • Establish a small number of key objectives

and criteria for AfDB assistance, and then allocate the available supplementary African Development Fund (ADF), AfDB, and other resources responsively (as it has with arrears clearance) on a rolling, merit-based alloca-tion basis.

• Objectives and criteria should be dictated by more in-depth assessments of needs in individual fragile and stabilizing situations and by the Bank’s demonstrated strengths in relevant areas.

• This responsive funding should not be sub-jected to any standard timelines for exit, but be available for shorter- or longer-term projects. It should be allocated through more frequent (perhaps quarterly) assessments of context and the strength of proposals emerging from countries and teams.

• Given the high stakes and difficult judgments involved, these allocation decisions would need to be made at a high level, with input from specialized staff.

2. Streamline and reallocate responsibilities within the Bank’s structures to enable an effective institutional response to fragility issues. • The Bank’s country offices and regional

and sectoral departments should have adequate responsibility and accountability (and adequate resources) for planning and

evaluation is attributable in significant part to such engagement.


37

Main Findings, Recommendations and Methodology

implementing programs in fragile and stabiliz-ing situations (including capacity-building and technical assistance support), and for applying the necessary analytical work and strategic guidance for these activities. A review of the accountabil-ity and incentive mechanisms for regional and sectoral departments should be undertaken to encourage more analytical work and adaptive approaches needed in fragile states.

• The Fragile States Unit should be relieved of its current responsibilities for directly managing the technical assistance and capacity building activities under Pillar III, as well as the vague and unrealistic “coordination” and “facilitation” roles assigned in 2008. It should be re-tasked to become a dedicated knowledge resource, with a role in resource allocation to maintain its operational links and its influence in effectively mainstream-ing knowledge. It should incorporate the latest practically oriented international guidance and the Bank’s own experience to rapidly generate practical guidance and operational tools adapted to the AfDB’s needs and current capabilities.

3. Consider which complex is more likely to provide the leadership for the implementa-tion of the necessary organizational changes required for the Bank to deliver on both the valid commitments of its 2008 strat-egy and the major revisions now needed, and to ensure the continuing coordination required. The Fragile States Unit should be posi-tioned in this complex.

4. With its Africa-wide responsibilities and need to leverage a useful strategic role in all fragile and stabilizing African countries and regions, the Bank should practice and promote more concerted, harmonized, and coordinated international efforts. It has a unique potential to become a working champion of

partnership, of practical experience-sharing rooted in African conditions, and of responding to condi-tions of fragility across borders. • The Bank must invest more effort in existing

donor coordination frameworks, especially at a strategic level, and actively help build them elsewhere; push ahead the decentralization pro-cess in fragile states and empower the country field offices with responsibility, decision-making authority, and resources.

5. The Bank should prepare an operational plan to deliver the cross-cutting changes required by the 2008 strategy, which extend from better external partnerships to stronger ana-lytical work, training, and adequate incentives for staff to work in fragile states.

A Methodology that Works The evaluation is built on the evidence from a port-folio review, a headquarters-based study, three country field studies (Liberia, Guinea Conakry, and Democratic Republic of Congo), and three country desk studies (Comoros, Côte d’Ivoire, and Republic of Congo). In keeping with its terms of reference, the evaluation focuses on assessing the Bank’s perfor-mance as measured against the stated objectives and standards established in the 2008 Fragile States Strategy, as well as the international Fragile States Principles and evolving standards of good practice. It examines the relevance, efficiency, organizational effectiveness, quality, and results of the Bank’s work in these countries.

This summary was prepared by Caroline McCuen (Consultant)

eVALUatiOn Matters

Fostering Inclusive Finance in Africa: An Evaluation of the African Development Bank’s Microfinance Policy, Strategy, and Operations, 2000–2012

Albert-Eneas Gakusi, Chief Evaluation Officer, African Development Bank

No projects

< UA 2 million

< UA 2 to 5 million

< UA 2 to 5 million

< UA 2 to 5 million

No projects

< UA 2 million

< UA 2 to 5 million

< UA 2 to 5 million

< UA 2 to 5 million

Fostering Inclusive Finance in Africa:

An Evaluation of the Bank’s

Microfinance Policy,

Strategy and Operations, 2000–2012Summary Report

From experience to knowledge...

From knowledge to action...

From action to impact

Independent Development Evaluation

African Development Bank

An ID

EV S

ecto

r Eva

luat

ion

39Since the 1970s, the Bank has implemented different kinds of microfinance projects. These projects were often components or subcomponents of large, non–financial sector pro-jects. They were dispersed among different departments without any effective coordination or adequate information system to assess if they were succeeding or not. In 2011, the President of the Bank set up a Bank-wide task force to conduct a review of the Bank’s microfinance activities covering the 2006–2011 period, which corresponds to the implementation period of the microfinance policy and strategy adopted in 2006.3 The task force recommended that: “In order to improve the Bank’s performance on microfinance activities, it is necessary to conduct an independent and comprehensive review of microfinance activities.” 4

The Purpose and Scope of the Evaluation12

The Evaluation of the African Development Bank’s Microfinance Policy, Strategy, and Operations, 2000–2012, was launched in response to the above recommendation. It covers the policy and strategies as well as operations approved between 2000 and 2012. It assesses- The extent to which the Bank’s microfinance vision, policies, and strategies are clear

and relevant in order for the Bank’s activities in this sector to contribute to its objec-tives of economic growth and poverty reduction;

- The relevance, effectiveness, efficiency, and sustainability of the microfinance projects;- The performance of the different intervention models; and - The appropriateness of the Bank’s institutional arrangements and staffing to deliver

microfinance projects.

What made the Evaluation Influential?The evaluation provided a response to puzzling questions in the Bank. The evaluation responded to a request from senior management to comprehensively clarify the status of microfinance in the Bank. At that time, the private sector group leading microfinance activities was dissatisfied with these activities, which were not adapted to Bank processes

3 African Development Bank, Microfinance Policy and Strategy, Operations Policies and Review Department (POPR), Tunis, 2006.4 African Development Bank, Stock-Taking Exercise of the Bank’s Microfinance Activities, January 2012.

Lessons Learned for Fit-For-Purpose EvaluationsA number of elements contributed to the usefulness of this evaluation, including:

• It responded to institutionwide concerns • The evaluation team partnered with a specialized institution to benefit from the latter’s

technical assistance and its network of seasoned consultants• Creating an enabling atmosphere and benefitting from structural changes• Internal research capacity and analytical leadership to foster close collaboration with

external consultants and external peer reviewers with specialized skills in the topic of the evaluation and in evaluation.

• Consistent and transparent communication and methodological approaches.

Albert-Eneas Gakusi, Evaluation Team Leader


eVALUatiOn Matters

that were more convenient for big transactions. In addition, little was known about the relevance, the performance, and the effectiveness of the projects implemented by the public sector departments. The coordination of the activities of the private sector and those of the public sector scattered in different departments was not working. There were questions about whether the Bank should finance microfinance activities at all. At the same time, senior manage-ment and Board members thought that microfinance activities responded to the Bank’ objective to reduce poverty in Africa. The need to know, the dissatisfac-tion, and the belief in microfinance as a development instrument led to wide buy-in of the evaluation by all stakeholders within the Bank. Fruitful collaboration between IDEV and CGAP. Because the IDEV team was not experienced in microfinance, it partnered with CGAP, the world-wide, leading think tank on microfinance, which has carried out several pertinent portfolio reviews for the Bank and other development partners. CGAP provided experts who acted as technical advisors at different phases of the evaluation and shared the contacts of its first-rate consultants in the areas of policy analysis, portfolio review, and case studies. It also helped recruit highly quali-fied and experienced external peer reviewers. At the same time, the IDEV team was committed to producing the needed responses to the uneasy situation of microfinance in the Bank. Finally, the evaluation team benefited from the strong support of highly qualified inclusive finance experts at the Bank’s headquarters and in its field offices.

Successful engagement of stakeholders and good timing. At the beginning of the evaluation, IDEV invited the chief executive officer of CGAP to address AfDB Board members and all staff on the topic of Financial Access for the Poor, Why It Matters and Where Are We in the Journey? The presentation triggered an engaged and sustained discussion. It also revealed the interest and the wide support for microfinance among AfDB staff, management, and Board members. This created a positive atmosphere that facilitated the work of the evaluation team. At the same time, the Bank was preparing its Ten-Year Strategy, 2014–2013, in which microfinance is considered an activity to be supported by the Bank.

Structural changes in the Bank. While the evalu-ation was on-going, the Bank’s senior management decided to create a new department in the financial sector as part of an institutional restructuring exercise. The new department prepared a policy and strategy with two pillars, one pillar consist-ing in increasing access of the underserved to the full range of financial services. In keeping with the evaluation recommendations, a microfinance division covering all microfinance activities was created. There were several meetings between the evaluation team and the management of the Financial Sector Department to share and discuss emerging findings and recommendations, which informed the preparation of the policy and strategy of the department.

Timing in favor of inclusive finance. Financial services for the poor have undergone consider-able changes over the past 30 years. In the late 1990s, the focus of funding agencies shifted from microcredit (small-value loans to the poor) to microfinance, covering a broad array of products including payments, savings, remittances, and insurance. In recent years, the focus has shifted to inclusive finance, which consists of building a robust financial ecosystem comprising tradi-tional banks and microfinance institutions, money transfer companies, postal networks, and mobile

40

The need to know, the dissatisfaction,and the belief in microfinance as a development instrument led to wide buy-in of the evaluation by all stakeholders within the Bank... ... created a positive atmosphere that facilitated the work of the evaluation team.


network operators. These ecosystems also include an enabling legal environment (regulation and supervision) and a solid market infrastructure: demand-side data; financial capability training programs; credit information sharing systems; support networks; and audit, rating, and training institutions. In this context, global and national-level policy makers as well as regional development Banks have been embracing financial inclusion as an important development priority. While this evaluation focused on the Bank’s microfinance operations, its recommendations take into account the shift to inclusive finance in order to place the Bank’s interventions in line with the global finan-cial landscape.

Concrete and actionable recommendations. Finally, the evaluation identified five models of intervention to assess performance through the desk reviews, the survey, and the case studies. The results pointed to successful models to scale up, as well as models that should be scaled down or abandoned. Findings and recommendations were shared during a Bank-wide seminar, where task managers, who had participated in the evaluation by filling out the questionnaire and responding to interviews, shared their views.

Use of the Evaluation’s Recommendations3

In the usefulness trilogy of Gary T. Henry,10 which consists of determining the common good, select-ing a course of action, and adapting the course of action, the evaluation helped the Bank determine what to do and what not to do, while showing which models to finance and which to exit. It confirmed the relevancy of financing microfinance for the Bank, based on its mandate and its strategic policy documents.

Management expressed overall satisfaction with the evaluation findings and all recommendations. It asserted that it had addressed many of the issues

10 Gary T. Henry, 2000, “Why Not Use,” New Direction of Evaluation 88: 85-98; 2003, “Influential Evaluations,” American Evaluation Association 24 (4): 515–24.

raised by the evaluation in the Bank’s Financial Sector Development Policy and Strategy 2014–2019 and that corrective measures had already been undertaken. In accordance with the evaluation’s recommendations, all microfinance activities were brought into a single division in the new Financial Sector Department.

The evaluation’s findings were shared with experts from leading funding agencies involved in finan-cial inclusion during a learning event organized by CGAP in Paris in January 2014: Becoming a Learning Organization. The contribution to this event consisted in presenting to the audience the role played by the portfolio review in discover-ing patterns that would be of assistance in taking corrective actions: the portfolio review showed differences in performance according to the depart-ments, whether targeting public or private sectors, the type of instruments used, and the mode of financing (direct or indirect).

41

Management expressed overall satisfaction with the evaluation findings and all recommendations. It asserted that it had addressed many of the issues raised by the evaluation in the Bank’s Financial Sector Development Policy and Strategy 2014–2019 and that corrective measures had already been undertaken. In accordance with the evaluation’s recommendations, all microfinance activities were brought into a single division in the new Financial Sector Department.

eVALUatiOn Matters

42

Methodology of the EvaluationThe evaluation compares the situation before and after 2006. The evaluation team faced research challenges related to lack of data and insufficient reliability of existing data. Thus, several sources of information were cross-referenced to produce find-ings and to make recommendations for policy and operations improvement. The evaluation conducted:

- A thorough review of all microfinance-relevant databases and documents, including poli-cies and strategies, appraisal reports, Project Completion Reports, supervision reports, and previous reviews conducted by the Bank and the Consultative Group to Assist the Poor (CGAP);

- Interviews with Bank project executive directors, senior management, managers, task managers, and officials from other funding agencies;

- Seven country case studies covering 16 of a total of 94 projects, including private and public projects, stand-alone and component projects, two post-conflict countries (Democratic Republic of Congo and Liberia), countries with mature microfinance sectors (Egypt, Ghana, Morocco, Tanzania), and one country where the microfinance sector is in crisis (Mali);

- A review of relevant literature on microfinance;- A seminar to present and discuss the findings

and recommendations emerging from the desk reviews and interviews;

- An electronic survey targeting the task manag-ers of 25 approved and ongoing projects that were assessed through a scoring exercise as very satisfac-tory, satisfactory, moderate, or weak/unsatisfactory, with a view to addressing weaknesses of available information on key indicators used in the micro-finance industry.

Because the evaluation team agreed to hold task man-agers accountable, informing them that it would give low ratings to projects without requisite information that should be available at their level, the response rate was maximal. This helped the team collect enough quantitative and qualitative information—mainly for quality at entry and project performance assess-ment. As is often the case in microfinance, the evalu-ation could not determine the extent to which the projects made a difference in the life of the intended beneficiaries. The information available rarely goes beyond the assessment of the quality of the projects; their implementation efficiency and their outputs; the organizational, technical, and financial soundness of the institutions with which the Bank works; and their sustainability.5 There are, however, an increasing number of evaluation studies that measure effects that can be traced to microfinance projects that would not have occurred without them. These are studies at the micro and local economic levels using randomized control trials (RCTs) or quasi-randomized evalua-tion approaches.6 These studies suggest that financial services do have a positive impact on a variety of microeconomic indicators—such as self-employ-ment business activities, household consumption, and well-being—but reveal no significant changes in some others—such as health, education, or women’s empowerment.7

5 Evaluation Cooperation Group (ECG), Making Microfinance Work: Evidence from Evaluations, Manila, 2010.6 Robert Cull, Tilman Ehrbeck, and Nina Holle, Financial Inclusion and Development: Recent Impact Evidence, CGAP Focus Note, 92, April 2014.7 Bauchet, Jonathan, Cristobal Marshall, Laura Starita, Jeanette Thomas, and Anna Yalouris, 2011, “Latest Findings from Randomized Evaluations of Microfinance.” Forum 2. Washington, D.C.: CGAP, Financial Access Initiative, Innovations for Poverty Action, and Abdul Latif Jameel Poverty Action Lab.; Banerjee, Abhijit, Esther Duflo, Rachel Glennerster, and Cynthia Kinnan, 2015, “The Miracle of Microfinance? Evidence from a Randomized Evaluation.” American Economic Journal: Applied Economics 7(1): 22-53.

Methodology, Main Findings,and Recommendations


43Main FindingsThe Bank’s strategy documents reflected the broad outlines of relevant approaches in microfinance. The 2006 strategy brought greater strategic clarity, but it was overly ambitious and did not set out priorities aligned to the Bank’s capacity, internal resources, and business model. It did not make clear choices about where the Bank could be most effective. The portfolio was only slightly impacted by the new strategy. After 2006, there was an increase in stand-alone projects, with the Private Sector and Microfinance Department approving only stand-alone projects and the Human Development Department increasing its stand-alone projects. However, project components in larger, non–financial sector projects still represented 44 percent of the projects after the strategy was adopted, from 95 percent before the strategy. Debts remained the main financing instrument, despite the availability of other instruments and the diverse market needs in Africa. They represented 86 percent after the policy compared with 92 percent before the policy. The strategy did not have any effect on internal processes, staff capacity, resources allocation, or organizational efficiency and effectiveness.

While the Bank confirmed its commitment to microfinance and to follow best practices for its operations over time, the evaluation showed that the Bank’s assistance performance was marginal. Attainment of objectives and efficiency was weak for public sector projects and moderately satisfac-tory for private sector operations. While the private sector projects scored high on sustainability, with greenfields consistently demonstrating high likeli-hood of sustainability, the Bank has mainly worked with unsustainable public sector institutions, which received 91 percent of the approvals during 2000–12.

For greenfields, the exit strategies that envisioned a financial break-even point after three-to-five years proved unrealistic, especially in fragile states. While effectiveness was moderately satisfactory for private sector projects, activities supported through grants (management information systems,

training, technical assistance) were all conducted with positive effects on institutional strengthening (for instance, the management information sys-tem provided to Access Bank Congo). Case studies reported that targets were not always reached on time, because they were sometimes overly ambi-tious, given difficult country contexts (Democratic Republic of Congo, Liberia, and Tanzania). However, because the project sponsors were seasoned experts in inclusive finance, they were able to rapidly rectify problems. It is therefore likely that targets will be reached, although delays appear inevitable.8

For the public sector, the Bank continued to finance microfinance components in larger projects, despite their poor performance; and the apex lending institu-tions (wholesale institutions) without adequate mar-ket and retail microfinance institution (MFI) capacity assessment to absorb their funding resources.9 Of the 12 Project Completion Reports that provided infor-mation on microfinance effectiveness, only one rated effectiveness as satisfactory. Of the 10 public sector projects included in the case studies, only the budget support in Morocco was assessed as satisfactory in terms of effectiveness. There was weak support at the macro level, despite the relevance of support to the microfinance business environment in the Bank’s regional member countries. 45

Operations across the Bank were fragmented, with-out an adequate operational system or a well-func-tioning focal point to take effective responsibility for implementing the 2006 strategy. Moreover, the lack of an internal champion with sufficient resources, clear responsibilities, and sufficient staff consider-ably hindered the Bank’s capacity to successfully support the microfinance sector. Internal processes, including project approval, procurement, and staff

8 For a comprehensive evaluation of greenfields in Africa, see Earne, Julie, Tor Jansson, Antonique Koning, and Mark Flaming, Greenfield MFIs in Sub-Saharan Africa. A Business Model for Advancing Access to Finance, Consultative Group to Assist the Poor (CGAP) and International Finance Corporation (IFC), 2014.9 This point is well documented in CGAP, Good Practice Guidelines for Funders of Microfinance, 2006


eVALUatiOn Matters

44

AcknowledgmentsThis paper draws on 13 reports prepared by IDEV with the technical assistance of the Consultative Group to Assist the Poor (CGAP). It benefited from the expertise of many consultants, the Bank’s internal inclusive finance experts, and Alice Nègre and Mayada El-Zoghbi, who also played an instrumental role in the evaluation process. All reports will soon be available on the Bank’s internet site: http://operationsevaluation.afdb.org/en/evaluations-publications/.

incentives do not favor microfinance projects, which are small transactions compared with standard Bank projects. Microfinance operations are per-ceived as overly risky by the Credit Risk Management Division. The Bank lacks performance metrics and disciplined procedures for monitoring results and managing the information across the institution. The introduction of the ex-ante Additionality and Development Outcome Assessment (ADOA) for pri-vate sector operations has contributed to improve-ment in using standardized indicators and measur-ing additionality for private sector operations. The Bank is slow to adapt to global developments on monitoring and evaluation (M&E) in the micro-finance sector, placing the Bank further behind other funders, some of whom have developed very ambitious targets in terms of microfinance M&E.

Other funders active in inclusive finance in Africa have found ways to overcome obstacles with which the Bank is still grappling. While the Bank has significantly improved its strategic clarity over time, its staff capacity in 2012 was only half of that of early 2000. Its performance remained weak in comparison to that of sister institutions for qual-ity assurance, project identification, performance indicators, and knowledge management.

While the Bank has played a limited role in the area of policy advice and coordination with other partners at the country and regional levels, it has a number of comparative advantages to engage in financial inclusion. These include- The ability to convene and engage with African

governments and to influence policy - A commitment to and experience in regional

approaches with promise for involvement in sup-port of financial infrastructure on a regional basis

- A range of financing instruments to deploy across both public and private sector operations

- The ability to source long-term financing- A proven research capability in financial inclu-

sion in close collaboration with the Making Finance Work for Africa Project.

The Way ForwardThe evaluation makes the following recommendations:

1. Within the overall financial sector strategy, focus the inclusive finance strategy, select effective intervention models, and translate the strategy into a specific business plan.

2. Bring all microfinance activities under the new department of the financial sector.

3. Continue direct support to retail institutions, albeit with streamlined processes that enable small transactions and diversified institutions (greenfields, bank downscaling, insurance, pay-ment companies, and the like).

4. Seize the opportunities of indirect investments through sustainable financial intermediaries.

5. Continue working with governments, but only to support policy and financial infrastructure initiatives.

6. Consider avoiding: (a) indirect financing to retail-ers through government and state-owned apexes and (b) microfinance components in larger non–financial sector projects.

7. Improve the management of the trust funds to enable small transactions executed by third parties.

8 Enhance the Bank’s microfinance performance M&E and build on the knowledge from others working in this sector.

9. Develop and reinforce partnerships with selected funding agencies with relevant expertise.

eVALUatiOn Matters

Hadiza Sidikou, Principal Evaluation Officer, African Development Bank

Evaluation of AfDB Non-sovereign Operations: An influential evaluation, so where did we go from there?

Independent Evaluation of

Non-Sovereign Operations, 2006-2011

SUMMARY REPORT

Operations Evaluation Department

African Development Bank Group

2013


SUMMARY REPORT

About the AfDB: The overarching objective of the African Development Bank Group is to spur sustainable economic development and social progress in its

regional member countries (RMCs), thus contributing to poverty reduction. The Bank Group achieves this objective by mobilizing and allocating resources

for investment in RMCs; and providing policy advice and technical assistance to support development efforts.

The mission of the Operations Evaluation Department is to enhance the development effectiveness of AfDB initiatives in its regional member countries

through independent and instrumental evaluations and partnerships for sharing knowledge

Director: Rakesh Nangia, [email protected]




Tel : (216) 71 102 841 Fax : (216) 71 194 460

Helpdesk: [email protected]


This evaluation reviews the Bank’s portfolio of 137 investment operations and 38 technical assistance projects

approved by the Board between 2006 and 2011. It assesses four aspects of the portfolio: the strategic alignment

of the portfolio, the performance of the portfolio, the risk management framework and risk exposures, and

institutional efficiency.

The evaluation highlights such positives as the catalytic effect of the Bank’s operations, the risk management

framework and growth in private sector operations in LICs. It reports negatives related to the one-Bank

approach, the reach to SMEs and MSMEs, and inefficiency in the approval process.

Indepen

dent Evalu

ation of N

on-Sovereign

Operation

s, 2006-2011 SU

MM

ARY

REPORT

OPEV NSO english cover.indd 1

22/05/13 11.18


47Setting the path for change: The private sector is widely recognized as a key driver of economic growth and employment creation – thus contributing to poverty reduction. The African Development Bank (the Bank) launched its private sector window in 1991, however, its private sector operations were never independently assessed until an independent review of AfDB non sovereign operations (NSO) was undertaken in 2012-2013 to take stock of the Bank’s private sector operations (PSO) with a view to learning from experience. The portfolio reviewed represented a total approved project volume of UA 3.9 billion and a committed vol-ume of UA 3.5 billion. The review included a technical assessment organized in four separate workstreams: strategy, portfolio performance, risk exposure and institutional efficiency.

This article reviews the extent of the utilization and impact of the Independent Evaluation of AfDB Non-Sovereign Operations, 2006-2012 and how the changes – direct and indirect – it led to were influential and moved the AfDB along the road toward mainstreaming private sector operations. There is a clear correlation between the findings of the report and corporate initiatives that sought to address important gaps and shortcomings in this area.

The NSO evaluation outcomes in the Bank can be grouped in three categories: individual, interpersonal and collective, as defined by Gary Henry. Individual influence is when evaluation changes something within the individual, such as one’s thoughts, attitudes, beliefs, or actions. Interpersonal influence refers to changes triggered by interactions between individuals, such as when an evaluation’s findings are used to persuade others about the merit of a program or policy. Collective influence means changes in the deci-sions or practices of organizations or systems, such as when policy change happens as a result of an evaluation, or when a program is expanded, continued, or terminated 1.

Impact on Policy and Strategy (collective influence). The NSO evaluation provided an independent per-spective on the private sector development strategy (PSD) and its alignment with the Bank’s operations. For example it noted that “to date, the Bank’s private sector operations have been focused narrowly on investment activity rather than on broader market reforms. Most Bank support for private enterprises is via senior loans and credit lines, although the equity portfolio

1 Gary Henry was director of evaluation at the David and Lucile Packard Foundation, deputy secretary of education for the Commonwealth of Virginia, and chief methodologist with the Joint Legislative Audit and Review Commission for the Virginia General Assembly. He received the American Evaluation Association Award for Outstanding Evaluation in 1998 and is a former co-editor-in-chief of New Directions for Evaluation.

eVALUatiOn Matters

48 has grown rapidly in recent years. In comparison, it has made relatively little use of other instruments such as guarantees, trade finance facilities and technical assistance”. Consequently, the evalua-tion recommended a review of the PSD strategy and of financial sector support to ensure private sector interventions achieved their intended reach and development goals. Following the evalua-tion, Senior Management drew on these lessons to strengthen the weak institutional environment and diversify the Bank’s private sector opera-tions. In 2014, it also restructured the organiza-tion, creating new units by shifting existing staff. This consolidated existing resources to scale up private sector operations such as Trade Finance. In addition, two new Strategy and Policy docu-ments were developed taking into account the important findings of the report. The new Private Sector Development Strategy, was informed by the recommendations of the evaluation, which served as providing a good catalyst to further refine the key tools and processes that set the framework for the Bank’s NSOs, the Bank’s new Private Sector Development Policy, as well as a new Business Plan for NSOs. Similarly, drawing lessons from the NSO evaluation, the Bank’s developed a new Financial Sector Development Policy and Strategy to address some of the challenges highlighted in the report, in particular the need for greater attention to the development of the financial sector and financial intermediation, and on the prerequisite to main-stream PSD with country and sector strategies.

Impact on Institutional Efficiency (interpersonal influence). The NSO evaluation report has turned out to be a highly cost-effective management tool that has improved institutional procedures especially on origination of projects. In fact, new innovative and efficient ways of originating private sector operations were introduced to fill in the gap in terms of business processes. The report noted that average project approval times has nearly doubled from 7.1 months in 2007 to 13.6 months in 2011. It also highlighted that the level of proactive sourcing of projects was only 5% of the portfolio, a rate considerably less

that other MDBs which have significantly higher rates. This makes the Bank less competitive than other IFIs and limits the potential productivity of the teams processing PSO. Two years later, it is clear that the findings and recommendations of the evalu-ation made a significant contribution to fine-tune the Bank’s business processes to strengthen their effectiveness and value for money. Following the evaluation, a committee was set up to look into inefficiencies in order to reduce redundancies and to improve average time for project approval. The Bank’s new PSO guidelines propose a streamlined approval process, benchmarked with other IFIs, to ensure that each step retained in the process presents “value for money” for the Bank and its clients2.

Impact on Portfolio Performance (interpersonal and collective influ-ence). The assessment of the performance of the private sector portfolio concluded that the Bank had inadequate monitoring procedures for gather-ing credible results data during supervision – and that this limits its ability to judge outturn develop-ment effectiveness. It also had some reservations on the equity portfolio due to its young nature. The evaluation recommended a review of the Bank’s equity investments to identify and address the reasons for underperformance in some areas. In line with the recommendations, a Private Equity (PE) evaluation was recently conducted to assess the financial performance of the Bank as well as the risk profile and development outcomes.

The evaluation report also called for the fine-tuning of the Assessment of Development Outcomes and Additionality (ADOA) procedure and for a closer alignment between the ADOA framework and ex-post development outcomes indictors which would enable a fair evaluation of projects’ outputs and outcomes. In addition, Management also took note of and acted on the recommendation to include the monitoring and reporting of ADOA in the loan agreement to ensure clients collect and report the indicators. Coaching clients on development

2 PSD Strategy 2013-2017


49outcomes upfront in the loan process has created awareness and prompted the action of frequent reporting on development outcomes. This new tool will allow the Bank to truly optimize its monitor-ing and reporting mechanism on development outcomes indicators for private sector operations.

Impact on Risk Management (col-lective influence). The NSO evaluation high-lighted the need to align the PSD strategy with the risk management framework to focus more on Low Income Countries (LIC) and fragile states. The assessment of the portfolio reveals a high concen-tration in Middle Income Countries (MIC) such as Nigeria, South Africa or regional projects. As a result, Management has adopted some measures to allocate more headroom to the Bank’s strategic priorities while keeping its risk levels within pru-dential limits. At the same time, Management is also increasing use of alternative instruments, as well as seeking risk-sharing partners to leverage the Bank’s NSO portfolio and pipeline 3.

Impact on External Stakeholders. It is interesting to note that the NSO evaluation also had some external influence on the evaluation community and on other MDBs. The report was used by JICA4 as a benchmark for the best prac-tice methodology to inform their non-sovereign loans evaluation. The report and the methodol-ogy used contributed to a better understanding of the complexities of evaluating private sector operations. It serves the purpose of learning and building capacity for peer institutions such as JICA. The independent review built on established good practice and was conducted using a data-driven approach with a mix of qualitative and quantita-tive assessment of the 137 projects composing the portfolio between 2006 and 2011. The portfolio analysis was completed with a combination of statistical analysis, direct interviews, sampling, field missions and external benchmarking. These good practice standards as well the findings of the

3 PSD Strategy 2013-20174 JICA: Japan International Cooperation Agency

report were shared with JICA as a learning tool for similar evaluations.

Conclusion: Empowering communities should be the ultimate goal of evaluations in the service of social and organizational betterment. The role of evaluation and its contribution to the devel-opment agenda of institutions has been proven for years and in many occasions. In the case of the NSO report, it is clear that the adoption of the evaluation’s recommendations has actually yielded the desired results. The report has con-tributed to organizational changes by providing information for decision making. The use of the report went beyond just the sharing of its findings and recommendations. It has been influential at many levels because it informed new policy and strategy documents on private sector operations. Indeed, learning took place as a result of the evalu-ation, triggering interest in the subject area from both non-sovereign and sovereign Bank staff. As a proven change agent, the NSO evaluation has played an important role in the Bank, ensuring account-ability on the use of the private sector resources and promoting learning on experience gained since the Bank opened its non-sovereign window. This evaluation was certainly one of the most influential report in recent years. The evaluation presented a variety of good prac-tices and lessons learned with new perspectives on how best the Bank can respond to the needs of its Regional Member Countries (RMC) through the private sector window. It also raised awareness about private sector support to the development agenda and highlighted its crucial role in catalyz-ing more resources in Africa. So far at the AfDB, the potential of, the private sector has not been assessed through an independent review. Consequently, the institutional context in which the evaluation took place reflected a thirst for change, which had influenced behaviors, thinking, actions and policies in many aspects.

eVALUatiOn Matters

Hadizatou Sidikou is a Principal Evaluation Officer for Private Sector Operations in the Independent Development Evaluation Department (IDEV)of the African Development Bank. She previously worked as a Program Officer for the Southern Africa Region, and as a Senior Budget Officer in the Bank. Before joining the Bank, Hadizatou worked in Washington DC in both public and private sectors, at AIG and in international NGOs such as PSI, EGPAF and AED, working on USAID development projects around the world as a Financial Analyst and Senior Program Manager, respectively.

She holds an MBA degree from University of Maryland University College, a DESS (Advanced Degree) in Dynamics of Development and a Bachelor’s Degree in Business Administration both from the University Pierre Mendes France in Grenoble, France.

PrOfIlE Of THE AuTHOr

The Bank launched its private sector window in 1991, to finance private sector projects and enti-ties incorporated in its RMCs without sovereign guarantees. While such non-sovereign opera-tions (NSO) are carried out on market terms, an essential feature of NSO is to promote pri-vate sector development (PSD) and expansion contributing to sustained economic growth and poverty alleviation in RMCs. These operations are financed through loans, guarantees and equity participation, as well as selective support through technical assistance (TA). The NSO evaluation conducted in 2012-2013 responded to Senior Management’s request to independently assess the Bank’s private sector portfolio to take stock of private investments between 2006 and 2011. The portfolio represented a total approved project volume of UA 3.9B and a committed volume of UA 3.5B. The review includes a technical assess-ment organized in four separate work streams:

Strategy, Portfolio Performance, Risk Exposure and Institutional Efficiency.

➤ Strategy: To what extent are the Bank’s private sector operations aligned with its five strategic objectives?

➤ Portfolio Performance: What have been the catalytic and demonstration effects of the Bank? How successful have the Bank’s private sector operations been in contributing to devel-opment? How profitable is the Bank’s private sector portfolio?

➤ Risk Exposure: What are the implications of the Bank’s risk management policies for its private sector strategy?

➤ Institutional Efficiency: How do the Bank’s business processes and procedures compare with those of its peers? How adequate are the Bank’s resources and its institutional structure?


Policy-Based Operations — Putting Knowledge to Work

From the AfDB

52IDEV carried out a multi-faceted evaluation of Policy-Based Operations (PBOs) (An Independent Evaluation of Policy-Based Operations in the African Development Bank, 1999–2009) to inform the new Bank policy governing these operations. Both the evalua-tion and the new policy had been called for by the Bank’s stakeholders (the policy by its shareholders and the evaluation by the contributors to the African Development Fund). The goal of the evaluation was to determine how effectively PBOs had been managed over the period, with a focus on the Bank’s policies and procedures for the design and implementation of PBOs, and how these had translated into decisions about how PBOs are used in a variety of country settings.

During the review period, 120 PBOs totaling UA 6.1 billion were approved, representing a massive 31.3 percent of total AfDB approvals and 21.8 percent of total African Development Fund (ADF) approvals. How well were these funds used?

What makes this a Fit-For-Purpose Evaluation?This evaluation was well-timed to feed into the new Bank strategy for PBOs (Bank Group Strategy on Policy-Based Operations, 2012), as well as to inform the strategic approach to fragile states (particularly in contributions to the development of budget support to fragile states under the Fragile States Facility) and to governance. The find-ings and recommendations were also well-received because many fit with ideas for improvement that had echoed through operations for some time, but that had not had formal standing.

In addition, the evaluation was scoped to be ambitious—covering all PBOs over a decade, and therefore a major chunk of the Bank’s investments. This made it strategically important and relevant to the Bank and its stakeholders. Equally, it responded to a specific request from the Bank’s financial supporters. However, it was also realistic. It was designed to focus squarely on the policies, practices, and management of PBOs, without attempting to cover development impacts across the continent.

How were the Findings of the Evaluation Used?Following the evaluation, the Bank adopted a new, consolidated PBO policy. The policy’s development was led by the policy team, with strong involvement from the Governance Department (which had most experience in the use of PBOs), but also involving other departments. This was because—as recommended in the evaluation—one of the objectives

Lessons Learned for Fit-For-Purpose EvaluationThe PBO evaluation was an excellent example of synergy: one well-timed, strategically important evaluation can influence multiple policies and practices and offer the greatest value for the time and money expended. A follow-up evaluation would be needed to assess whether these changes are having the full intended positive impact on the PBOs launched since the new policy, but the vital signs are good.

eVALUatiOn Matters


53of the new policy was to make it Bank-wide and encourage use of the PBO instruments across sec-tors, not only governance. The Bank later developed guidelines to support the policy’s implementation, and country strategy paper guidance was also revised; however, planned training has not yet been provided. In addition, the standard project docu-mentation for PBOs has been revised, to reflect that the instrument is different from the Bank’s projects.

Together, the new policy and guidance address many of the shortcomings of the earlier approach noted in the evaluation: (1) absence of consistent, widely available guidance, (2) use of PBOs for sec-tors other than governance; (3) insufficient guidance for Country Strategy Papers on the role of PBOs, (4) complex and time-consuming design and appraisal procedures; (5) the need for the Bank to engage in policy dialogue; and (6) uncertainty about audit and fiduciary risk requirements. Parallel reforms to the Bank’s risk management framework are also helping the Bank to address fiduciary risk concerns more effectively.

In addition, while the evaluation’s main target was the new PBO policy, the findings also informed the Bank’s new policies for governance and for fragile states, and it is widely cited in Bank documents and by other development institutions.

OPERATIONS EVALUATION DEPARTMENT March 2011 vii

Executive Summary

Scope of the evaluationi. This evaluation assesses the African Develop-ment Bank’s (AfDB) use of policy-based operations (PBOs) over the period 1999-2009, during which 120 PBOs totalling UA 6.1 billion were approved, repre-senting 31.3 percent of total AfDB approvals and 21.8 percent of total African Development Fund (ADF) approvals. The evaluation examines how effectively the Bank has used PBOs to support national develop-ment objectives, with a focus on the Bank’s policies and procedures for PBOs. The evaluation is based on a review of literature and comparative experience with PBOs in other development agencies, a review of the Bank’s institutional and policy framework, six country case studies, and four case studies of other significant operations.

Significant progress has been made…ii. The evaluation concludes that the Bank has made substantial progress in its use of PBOs. In 1999, the Bank was heavily dependent on the IMF and the World Bank for analysis and design of its engagement in structural and sectoral adjustment operations. These operations often encountered implementation difficulties and delays related to weak country ownership and the unsuccessful attempt to leverage policy change through the use of complex loan conditionalities.

iii. The Bank now operates as a significant partner in joint donor budget support arrangements, with a largely successful record of engagement. The Bank has developed a cadre of staff (concentrated in the Bank’s Economic and Financial Management Department, OSGE) who have strong experience in the design and management of budget support operations. The establishment of Field Offices (even though decentralisation has progressed far more

slowly than planned) has significantly improved the Bank’s ability to engage in national policy and budget processes, and has strengthened the Bank’s monitoring and supervision of PBOs.

iv. The Bank has also developed a stronger organisa-tional capacity and structure for the design, appraisal, management, and monitoring of PBOs, although some aspects still require further development. In addition, the Bank proved highly responsive to the challenges of the global economic and financial crises as they affected the Bank’s Regional Member Countries (RMCs) during 2008 and 2009. The Bank was able to design and implement operations that not only met the urgent financial requirements of its clients, but also provided a platform for addressing longer-term structural reforms. The Bank has also made important contributions to budget support in fragile states.

… but some improvements are still requiredv. The evaluation identified some significant short-comings in the Bank’s policies and practices regard-ing PBOs, including in comparison to approaches used by other development agencies.

• First, the Bank has a proliferation of policies and guidance for PBOs that (unlike in other agencies) have not been consolidated and updated. Procedural requirements for PBOs are excessively complicated and divert manage-ment attention from focusing on analysis and design. Information systems for PBOs are weak, and uncertainty persists about how audit and fiduciary risk should be addressed.

• Second, the Bank lacks both a mechanism for a medium-term programmatic approach to budget

FONDS AFRICAIN DE DÉVELO

PPEMENT

AFRICA

N DEVELOPMENT FUND

BANQUE

AFRI

CAINE D

E DÉVELOPPEMENT

Evaluation of Policy-based Operations

in the African Development Bank,

1999-2009



2011

eVALUatiOn Matters

54

MethodologyThe findings of this evaluation are drawn from four main sources. The first is a review of litera-ture and the comparative experience with PBOs in the World Bank, Asian Development Bank, European Commission, and selected bilateral agencies.

The second is a review of the Bank’s institutional and policy framework for PBOs within the con-text of the Bank’s wider strategies and organiza-tional reforms over the evaluation period.

The third includes six country case studies: Burkina Faso, Ethiopia, Morocco, Rwanda, Sierra Leone, and Tanzania. These countries were selected because they represented a wide range of contexts in which the Bank provided PBOs and because they had all had PBOs over the entire evaluation period, so that changes in the Bank’s approach and performance could be assessed.

The fourth source consists of four additional case studies of large and significant PBOs, including the Botswana Economic Diversification Support Loan, the Nigeria Economic and Power Sector Reform Program, the Egypt Financial Sector Reform Loan, and the Democratic Republic of Congo Emergency Program to Mitigate the Impact of the Financial Crisis.

In 1999, the Bank was heavily dependent on the International Monetary Fund (IMF) and the World Bank for analysis and design of its engage-ment in structural and sectoral adjustment

operations. These operations often encountered implementation difficulties and delays related to weak country ownership and the unsuccessful attempts to leverage policy change through the use of complex loan conditionalities.

During the period covered by this evaluation, the Bank sought to develop a more effective strategic and organizational approach to the challenges presented by changes in the international and regional context, developments in the interna-tional aid architecture, and lessons from the Bank’s own experience. The international eco-nomic environment during the period generally supported growth and increasing international aid, until the impact of the international financial crisis hit at the end of the period.

The evaluation evidence demonstrates that the Bank has made substantial progress in its use of PBOs over the evaluation period. It has developed a stronger organizational capacity and structure for the design, appraisal, and management and monitoring of PBOs, and it is a major partner with other donors in joint budget support operations, with a largely successful record of engagement. But to build on these achievements, the Bank now faces significant challenges. These include the need to build country capacity to engage in mean-ingful policy dialogue, to build links between PBOs and the project investment portfolio, and to expand PBO knowledge and capacity throughout the Bank.



RecommendationsThe evaluation made three main recommenda-tions that would enable the Bank to use PBOs more effectively in support of its overall objectives:1. Define more clearly and authoritatively the role of PBOs in advancing the Bank’s overall mandate and objectives, as well as its engagement at the country level.

2. Produce a single, comprehensive policy and supporting guidance on PBOs, building on the 2004 policy on Direct Budget Support Lending.

3. Address the organizational and management implications of the new policy to support its effective implementation.

In addition, the following specific actions were recommended:• Determine how PBOs would be used to support

the Bank’s wider strategic objectives and the needs of its clients. To the extent that PBOs were to be used to strengthen engagement in policy dialogue, rather than just as a financ-ing mechanism, the Bank needed to build its capacity and procedures to fulfill this role.

• Consolidate Bank PBO policies and guidance into a single policy.

• Base guidance for the design of PBOs and the identification of results on a more fully developed model of their intervention logic.

• Identify potential synergies between engage-ment in general budget support and related PBOs and other parts of the Bank’s program, especially investment operations.

• To the extent that the Bank identifies poten-tial synergies and contributions it can make through policy dialogue or complementary activities, these should be explicitly built into the design of PBOs. Sufficient capacity and resources should be made available to ensure that these contributions are effective.

• Each sectoral area of the Bank should assess the scope for using sectoral PBOs, contribut-ing to multisectoral PBOs, and participating in Sector-wide Approaches.

• Examine how policies and procedures (espe-cially relating to the ADF) can be adapted to allow a more programmatic medium-term approach to budget support.

• Develop, fully document, and provide com-prehensive training for staff in a set of pro-cedures and timetables that are specifically tailored to PBOs.

• Review and substantially strengthen its infor-mation systems and procedures for PBOs.


Caroline McCuen (consultant) and Penelope Jackson, Principal Evaluation Officer, IDEV, contributed to this article.

eVALUatiOn Matters

Financial struggles in UgandaWho struggles more and why?

KFW Prof. Isabel Günther and Joeri Smits, Center for Development and Cooperation (NADEL), ETH Zürich, Switzerland. Prof. Eva Terberger and Thomas Gietzen


Over-Indebtedness in MicrofinanceThe world has seen a rapid spread of microfinance over the past 40 years. Access to formal financial ser-vices has opened up new opportunities for millions of low-income households and micro-entrepreneurs. Despite the expansion of microfinance, there remain many regions in the world where financial inclu-sion is still lagging behind. However, other regions have undergone a very different experience lately, sparking a discussion about the vices and virtues of microcredit, the financial service that has histori-cally been at the heart of microfinance (that also entails microsavings and microinsurance).

Right from the beginning, the microfinance move-ment was carried forward by the belief that overcom-ing entrepreneurial credit constraints will prove an effective way to spur economic development and fight poverty.

In recent years, however, even advocates of micro-finance are getting worried that in some areas too much credit is available. In particular, urban areas have started to witness increasingly saturated credit markets, and sometimes abundant credit supply. The risk of over-borrowing and household over-indebtedness is increasingly being recognized as a major issue by the microfinance community.

In 2014—after the experience of severe crises in microfinance—for example, in Andhra Pradesh (India) in 2010 or Morocco in 2009—the CFSI Survey for Microfinance Risk (CFSI 2014) ranked over-indebtedness, credit risk, and competition as the most important risks faced by the microfinance industry.

Over-indebtedness of microfinance clients is a potential downside risk of the rapid expansion of microcredit. The risk has always been intrinsically tied to borrowing. A business idea can fail or other external shocks can occur, leaving the client to face the debt burden without any of the expected gains. However, the crises in microfinance have brought the issue of over-indebtedness to the collective conscience.

Measurement of Over-Indebtedness There are several ways to define and measure the problem of over-indebtedness of microfinance clients. Repayment delays and delinquency (non performing loans, or NPLs) at formal institutions are a commonly applied, well-defined, and easily quantified measure. However, NPLs offer only an incomplete picture of over-indebtedness. On the one hand, delinquent borrowers may include those with temporary liquidity shortages that do not

57

2

Non-performing loan Performing loan

Unwilling to pay

Short-term liquidity shortage

Unable to pay

By living be-

low the breadline

No

problem

Figure 1: Client Types and Over-Indebtedness Over-indebted

borrowers may actually meet their re payment obligations at formal financial institutions, but at the cost of their household needing to cut back severely on food consumption, education, or even medical expenses, which clearly represents a state of too much debt. Figure 1 presents an overview of the different categories of borrowers.

To find an over-indebtedness measure that captures households’ suffering from debt-induced distress, Schicks (2013) was a first attempt to measure over-indebtedness by the unduly harsh sacrifices households make to repay their loans; instead of considering the financial institutions’ view of non-performing loans only. Typical household sacrifices include reduced food quantity/quality, reduced education expenses (such as taking children out of school) or depleting basic household assets/savings. Applying this measure and counting micro-borrowers as over-indebted when they suffered severe sacrifices or more than two stress events, Schicks (2013) finds that in urban Accra, Ghana, about one-third of all the microfinance borrowers interviewed must

be considered over-indebted; a number that —on first consideration— seems to warrant severe concern about the sustainability of microfinance lending in urban Accra.

There are, however, two problems with Schicks’ approach that suggest interpreting her results with care. First, when asking households whether they had to endure hardship to meet repayment obligations, respondents tended to confirm the interviewer’s concerns (confirmation bias). Second, Schicks’ study to measure the prevalence of over-indebtedness in Ghana did not feature a comparison or control group. All of the respondents in the study were borrowers of formal micro loans, and as such, the study is unable to isolate the effects of borrowing at microfinance institutions on over-indebtedness. Instead, the results capture the general struggles of poor households in Ghana trying to make ends meet. The exclusive attribution of stress events to the use of microcredit is highly questionable.

Financial Institution

Borrower

Figure 1: Client Types and Over-Indebtedness

Financial struggles in UgandaWho struggles more and why?

eVALUatiOn Matters

58 necessarily suffer from a structural debt problem. On the other hand, some borrowers may actually meet their re payment obligations at formal finan-cial institutions, but at the cost of their household needing to cut back severely on food consumption, education, or even medical expenses, which clearly represents a state of too much debt. Figure 1 presents an overview of the different categories of borrowers.

To find an over-indebtedness measure that cap-tures households’ suffering from debt-induced dis-tress, Schicks (2013) was a first attempt to measure over-indebtedness by the unduly harsh sacrifices households make to repay their loans; instead of considering the financial institutions’ view of non-performing loans only. Typical household sacri-fices include reduced food quantity/quality, reduced education expenses (such as taking children out of school) or depleting basic household assets/sav-ings. Applying this measure and counting micro-borrowers as over-indebted when they suffered severe sacrifices or more than two stress events, Schicks (2013) finds that in urban Accra, Ghana, about one-third of all the microfinance borrowers interviewed must be considered over-indebted; a number that —on first consideration— seems to warrant severe concern about the sustainability of microfinance lending in urban Accra.

There are, however, two problems with Schicks’ approach that suggest interpreting her results with care. First, when asking households whether they had to endure hardship to meet repayment obli-gations, respondents tended to confirm the inter-viewer’s concerns (confirmation bias). Second, Schicks’ study to measure the prevalence of over-indebtedness in Ghana did not feature a comparison or control group. All of the respondents in the study were borrowers of formal micro loans, and as such, the study is unable to isolate the effects of borrowing at microfinance institutions on over-indebtedness. Instead, the results capture the general struggles of poor households in Ghana trying to make ends meet. The exclusive attribution of stress events to the use of microcredit is highly questionable.

The Uganda StudyOur follow-up study in Uganda was motivated by the failure of Schicks’ approach to capture the actual effects of taking up microcredit on house-holds’ financial burden. In addition to measuring the prev-alence of financial struggle among households, our study tries to address the question of which effect formal microcredit has on households’ over-indebt-edness by applying a control or comparison group design. Using data on more than 1,500 Ugandan households that were a) borrowers at a formal insti-tution, b) semi-formal borrowers (borrowing from an institution that is not regulated by the central bank), c) informal borrowers (borrowers at infor-mal sources such as moneylenders, also including those with outstanding bills), or d) nonborrow-ers, we are able to distil differences with respect to the financial burden between these groups. We cooperated with one of Uganda’s major providers of microfinance services to sample formal borrowers from that institution. Compared to the total popula-tion with a share of about 9 percent of households with a formal loan (FIN- SCOPE Uganda 2013), this study deliberately over-sampled borrowers of formal loans. The comparison group households were chosen pseudo-randomly by interviewing the third-next household in the vicinity of the baseline sample of formal borrowers.

Table 1: Distribution of Borrower Types (Sample)

Credit use Number of respondents

Share of respondents(%)

Formal 238 17.8

Formal and informal

359 26.8

Formal and semi-formal

45 3.4%

Semi-formal 43 3.2

Informal 368 27.5

Nonborrower 285 21.3

Total 1339 100%


Data was collected during interviews in the fall of 2013 on specific stress events, households’ loans and liabilities, and other household features in and around Uganda’s capital of Kampala, which is widely perceived as a highly competitive market for micro-finance. The interview questions built on Schicks’ (2013) sacrifices approach. However, in order to avoid confirmation bias, no explicit link between the questions on sacrifices (distress events) and formal repayment obligations was made.

Financial StrugglingFigure 2 gives an overview of the most important distress events that were elicited and their occur-rence among formal borrowers and the full sample of households. Results suggest that some stress indi-cators reach alarming levels. The fact that slightly more than 50 percent of all respondents for which a household member fell sick were unable to pay for medicine or the doctor due to a lack of funds during the last month is a clear indication of severe financial struggling of the population. However, the small difference between formal borrowers and the rest of the population suggests that this may not be an immediate result of too much credit among formal borrowers. Measuring distress events for a single

group of borrowers is informative about the general financial struggle of that part of the population, but contains no information on whether taking up microcredit expedites household over-indebtedness.

Figure 3 depicts the distribution of the distress index that was constructed by the sum of confirmative answers to the aforementioned distress events in

59

4

Figure 2: Distress Events, Formal Borrowers and Full Sample

70,0%

58,4% 60,0% 55,6%

46,7% 50,0%

43,9% 43,4% Share of affirmative answers 44,4%

40,0%

30,0%

16,8% 20,0%

14,0% 11,8%

10,3% 10,0%

1,1% 0,6% 0,0%

During the last During the last month, unable to month, ate less or

pay for medicine/doctor due to lack of

funds (those who got ill/injured)

lower quality due to lack of funds

During the last 6 months, took children from

school due to lack of funds

During the last 6 months, assets

seized for failing repayment obligations

During the last 6 During the last 6 months, assets months, ran out of sold to meet repayment obligations

money before next revenues arrived at least

every other month

Formal borrowers only Full sample


Figure 3: Distress Index among Borrower Groups (including 95 percent confidence interval)

5

over-sampled borrowers of formal loans. The comparison group households were chosen pseudo-randomly by interviewing the third-next household in the vicinity of the baseline sample of formal borrowers.

Data was collected during interviews in the fall of 2013 on specific stress events, households’ loans and liabilities, and other household features in and around Uganda’s capital of Kampala, which is widely perceived as a highly competitive market for microfinance. The interview questions built on Schicks’ (2013) sacrifices approach. However, in order to avoid confirmation bias, no explicit link between the questions on sacrifices (distress events) and formal repayment obligations was made.

Financial struggling Figure 2 gives an overview of the most important distress events that were elicited and their occurrence among formal borrowers and the full sample of households. Results suggest that some stress indicators reach alarming levels. The fact that slightly more than 50 percent of all respondents for which a household member fell sick were unable to pay for medicine or the doctor due to a lack of funds during the last month is a clear indication of severe financial struggling of the population. However, the small difference between formal borrowers and the rest of the population suggests that this may not be an immediate result of too much credit among formal borrowers. Measuring distress events for a single group of borrowers is informative about the general financial struggle of that

part of the population, but contains no information on whether taking up microcredit expedites household over-indebtedness.

Figure 3: Distress Index among Borrower Groups (including 95 percent confidence interval)

Figure 3 depicts the distribution of the distress index that was constructed by the sum of confirmative answers to the aforementioned distress events in figure 2. We find that borrowers at informal sources and borrowers that borrow from both informal/semi-formal and formal institutions report, on average, to have undergone more stress events than borrowers at formal institutions and households that did not borrow at all. The descriptive evidence is thus not supportive of any claims that purport a direct negative effect of formal micro- loans on households’ financial struggles.

Household characteristics Descriptive statistics alone are not sufficiently reliable as a source of evidence, however. Why is that? We

Average of distress index by borrower type

2,5

2

1,5

1

0,5

0

"

43,9%

4


70,0%

58,4% 60,0% 55,6%

46,7% 50,0%


40,0%

30,0%

16,8% 20,0%

14,0% 11,8%

10,3% 10,0%

1,1% 0,6% 0,0%











every other month


4


70,0%

58,4% 60,0% 55,6%

46,7% 50,0%


40,0%

30,0%

16,8% 20,0%

14,0% 11,8%

10,3% 10,0%

1,1% 0,6% 0,0%











every other month


4


70,0%

58,4% 60,0% 55,6%

46,7% 50,0%


40,0%

30,0%

16,8% 20,0%

14,0% 11,8%

10,3% 10,0%

1,1% 0,6% 0,0%











every other month


4


70,0%

58,4% 60,0% 55,6%

46,7% 50,0%


40,0%

30,0%

16,8% 20,0%

14,0% 11,8%

10,3% 10,0%

1,1% 0,6% 0,0%











every other month


4


70,0%

58,4% 60,0% 55,6%

46,7% 50,0%


40,0%

30,0%

16,8% 20,0%

14,0% 11,8%

10,3% 10,0%

1,1% 0,6% 0,0%











every other month


4


70,0%

58,4% 60,0% 55,6%

46,7% 50,0%


40,0%

30,0%

16,8% 20,0%

14,0% 11,8%

10,3% 10,0%

1,1% 0,6% 0,0%











every other month


4


70,0%

58,4% 60,0% 55,6%

46,7% 50,0%


40,0%

30,0%

16,8% 20,0%

14,0% 11,8%

10,3% 10,0%

1,1% 0,6% 0,0%











every other month


"

eVALUatiOn Matters

60 figure 2. We find that borrowers at informal sources and borrowers that borrow from both informal/semi-formal and formal institutions report, on average, have undergone more stress events than borrowers at formal institutions and households that did not borrow at all. The descriptive evidence is thus not supportive of any claims that purport a direct negative effect of formal micro- loans on households’ financial struggles.

Household CharacteristicsDescriptive statistics alone are not sufficiently reli-able as a source of evidence, however. Why is that? We might imagine a multitude of household char-acteristics that influence both the uptake of credit and a household’s experience of distress events.

Table 2 gives an overview of the household charac-teristics in the interviewed sample. To pick out some examples: Not all interviewed households have the same income or the same level of education. It seems quite likely that households with higher incomes or higher education than average experience less financial stress than the average household, regard-less of whether they borrow and what type of loan, formal or informal, they take up. Another prominent example is the influence of financial literacy. Many studies suggest that financially literate households have a lower probability of running into severe debt problems. The table shows that financial literacy var-ies between households, and perhaps this variation is also systematically linked to belonging to a certain borrower group. As we could not observe financial literacy directly, but considered it to be an important source of influence on financial struggling, we had to measure financial literacy (and numeracy skills) with the help of simple tests that asked respondents to answer five basic financial knowledge questions (solve basic numerical problems).

Just like borrowing from formal, semi- formal, or informal sources, all these household features reported in table 2 potentially influence the amount of financial struggling. Our descriptive statistics are unable to separate the different sources of influ-ence, they rather report them blended. That is why econometric analysis is the obvious next step in our analysis. Econometric AnalysisGiven that we can observe and measure the relevant household characteristics, econometric analysis is able to disentangle the effects of borrowing from a certain source versus the influence of different household characteristics on financial distress. Specifically, econometric analysis answers the ques-tion whether being a formal borrower increases the chance of experiencing more distress events compared to the other borrower groups, control ling for the different characteristics between the groups that might also have an influence on who borrows from whom.

Table 2: Basic Sample Characteristics

Variable Mean Std. Dev.

Female 40 % 49 %

Age in years 38.71 11.34

Household size 5.09 2.69

No education or primary education not completed

15 % 36 %

Highest education: primary

31 % 46 %

Highest education: high school

39 % 49 %

Highest education: diploma

5.5 % 2.3 %

Highest education: university

2.8 % 1.7 %

Muslim 23 % 42 %

Mean numeracy score (1-4)

2.58 1.25

Mean financial literacy score (1-5)

2.89 1.37

Monthly household income (UGX)

1,802,907 5,254,306

Monthly household debt service (UGX)

290,252 881,700

Total amount borrowed (UGX)

1,338,927 2,263,799

Shock (experienced a negative shock recently)

55% 50%


61Table 3 reports results of a Poisson regression (a type of regression that is used for data that can be counted, like the sum of distress events) using various household characteristics as control vari-ables. Results for different comparison samples are reported in columns (1)-(3) and offer a number of interesting insights. First, we find that being a borrower at a formal institution does not increase the chances of experiencing financial distress in comparison to nonborrowers (column 1). Moreover, we find that compared to the group of informal and semi-formal borrowers, borrowers at formal finan-cial institutions are less likely to suffer from distress events (column 2). This suggests that even when we control for observable differences in household char-acteristics between the groups, there is evidence that the chances of struggling financially are significantly higher for semi-formal and informal borrowers than for borrowers from formal institutions.

Furthermore, we find that a higher number of outstanding loans, self-reported unstable income, larger house hold sizes, and the household head being female correlate with more financial strug-gling. Results on female household heads are pos-sibly driven by differences in socioeconomic status

between households with a male and with a female head. Most notably, however, financial literacy skills are negatively correlated to levels of financial dis-tress. This finding is in support of the efforts of the national and international community to promote financial literacy among clients of financial institu-tions in Uganda. The result suggests that financial literacy skills help households to achieve more sustainable planning of household finances.

Column 3 reports the results of a comparison of formal borrowers to nonborrowers that have recently applied for formal credit but have not actually bor-rowed yet (applicants). Using applicants, we try to create a control group that is likely to be similar in its characteristics as the households that have self-selected into the same type of borrowing. Results suggest that active borrowers, on average, undergo more distress events than applicants. This effect, however, is to some extent expected. Borrowing always entails risk and thus leads to a spread of the household income distribution; some borrowers’ businesses fail, some borrowers’ businesses fare well. A distress index mainly captures the downside of this spread in

Selection Bias and RCT DesignsNonborrowers, applicants for a loan, semi-formal borrowers, and informal borrowers are imperfect comparison groups for formal borrowers. Household characteristics that make some households apply for and take up a formal loan may also affect their behavior and financial outcomes, inde-pendent of their borrowing behavior. Hence, we might mistakenly accredit differences in financial struggling between the borrower-groups to the type of borrowing that are actually caused by dif-ferent household characteristics. Comparisons between the groups might also be biased because households tend to apply for certain types of credit after experiencing an adverse event—that is households in dire straits might self-select into certain borrower groups. A more rigorous randomized control trial (RCT) design would require randomly granting formal credit to some households, while denying credit to others that had also become borrowers at the formal institu-tion. For obvious reasons, this approach of denying people credit is often unfeasible. Moreover, a more thorough look at the question of over-indebtedness would ideally follow the households’ finances over a longer period of time. Econometric analysis is an applicable tool to correct at least for different observable (but not for unobservable) household characteristics, even if the results cannot match the rigor of an RCT.

eVALUatiOn Matters

62 Table 3: Results from a Poisson Regression

Outcome: Distress IndexResults report b/se

Formal vs. Nonborrowers

Formal vs.informal/semi-formal borrowers

Formal vs. applicants

Formal borrower=1 -0.1007 -0.1664* 0.3526**

0.11 0.09 0.13

Age in years -0.0052 0.0159 0.0103

0.01 0.02 0.02

Female=1 0.1667* 0.1907** 0.2129**

0.07 0.06 0.08

Household size 0.0169 0.0388**** 0.0185

0.01 0.01 0.02

Muslim=1 0.1474* 0.1233 -0.0498

0.07 0.07 0.09

Primary educ completed -0.0603 -0.0214 0.1422

0.08 0.08 0.09

O level in high school completed -0.0402 -0.077 0.0096

0.09 0.08 0.11

A level in high school completed -0.3981* -0.1908 -0.3528

0.17 0.154 0.21

Diploma -0.4477 -0.4425* -0.35

0.23 0.2 0.26

University degree -0.3273 -0.2431 0.1865

0.85 0.28 1.62

Wealth index -0.031** -0.0336** -0.0233*

0.01 0.01 0.01

Numeracy score -0.0136 -0.0345 -0.0445

0.03 0.03 0.03

Financial literacy score -0.1126**** -0.0406 -0.0636**

0.03 0.02 0.03

Number of loans/open bills outstanding 0.1443**** 0.1695**** -

0.07 0.06

Shock dummy 0.2955**** 0.3649**** 0.3707****

(experienced a negative shock recently) 0.07 0.07 0.08

Unstable income(as reported by respondent)

0.9462****0.08

0.3649****0.09

0.3707****0.1

N 877 973 627

Wald 151.45 169.69 95.32

* P<0.05, ** P<0.01, *** P<0.001, **** P<0.0005


the distribution (reflecting disproportionally the negative experiences of that part of the borrowing population that does not fare well) and we do not observe whether the increase in downside risk is counterbalanced—or more than counterbalanced—by gains on the upside.

ConclusionFinancial distress of microfinance clients has become a great concern within the microfinance community. This study was motivated by the defi-ciency of earlier attempts to find meaningful com-parison groups that facilitate an interpretation of the role of credit on microfinance clients’ financial burden measured at the house hold level.

We find that while a larger part of the population in urban Kampala, Uganda, must be considered financially struggling, there is no evidence that the uptake of formal credit from a microfinance institution fosters over-in debtedness systemati-cally. Particularly in comparison with other credit sources, formal loans are associated with a lower amount of stress imposed on borrowers.

This is by no means saying that clients of formal financial institutions do not struggle financially (and sometimes more than nonborrowers). It is of utmost importance that financial institutions pay great attention to the repayment capacities of their borrowers, and not everybody is in the position to handle credit. It suggests, however, that formaliza-tion of credit sources reduces financial distress.

In a competitive credit market as urban Uganda, our finding of a strong association between the num-ber of outstanding loans and experienced distress underlines the importance of credit information sharing among lenders.

Financial literacy stands out as a key policy vari-able to make sure that, in particular in the case of saturated credit markets, borrowers do not take up unduly large amounts of credit and do not take uninformed financial decisions. An increase in the overall level of financial literacy skills in the popula-tion is likely going to lead to a reduction in overall financial struggling. SourcesJessica Schicks. 2013. “The Sacrifices of Micro-

Borrowers in Ghana - A Customer-Protection Perspective on Measuring Over-Indebtedness,” Journal of Development Studies 49(9): 1238–55.

CFSI.2014. “Microfinance Banana Skins 2014, The CSFI Survey of Microfinance Risk.” www.csfi.org/files/Microfinance_Ba- nana_Skins_2014_-_WEB.pdf

The chances of struggling financially are significantly higher for semi-formal and informal borrowers than for borrowers from formal institutions.

63

eVALUatiOn Matters

Gender Mainstreaming — stuck at thestarting Gate?

March 2014

6316

99

eVALUatiOn MattersA Quarterly Knowledge Publication

of the Operations Evaluation Department

of the African Development Bank Group

“Today there is growing recognition that gender equality is not only a critical

human rights issue for women and girls, it is a prerequisite for the achieve-

ment of broader development goals, effective humanitarian response and

sustainable peace and security.”

Phumzile Mlambo-Ngcuka, United Nations Under-Secretary-General and Executive Director of UN Women, Page 52

Gender equality is

the goal; gender

mainstreaming is

the means

Evaluation as a tool

in gender

mainstreaming

South Africa and

Rwanda are among the

pioneers in the area

of gender-responsive

budgeting

Gender Inequality and You

eVA

LUatiO

n M

attersG

ender In

equality an

d You

FONDS AFRICAIN DE DÉVE

LOPPEMENT

AFRICA

N DEVE

LOPMENT FUND

BANQUEAFRI

CAINE

DE DÉVELOPPEMENT

About the AfDB: The overarching objective of the African Development Bank Group is to foster sustainable economic development and social progress in its

regional member countries (RMCs), thus contributing to poverty reduction. The Bank Group achieves this objective by mobilizing and allocating resources for

investment in RMCs and providing policy advice and technical assistance to support development efforts.

The mission of the Operations Evaluation Department is to enhance the development effectiveness of the AfDB in its regional member countries through

independent and instrumental evaluations and partnerships for sharing knowledge


Website: http://operationsevaluation.afdb.org/

Write to us: [email protected]

From Experience to Knowledge …

From Knowledge to Action

… From Action to Impact

Gender Inequality and You

OPEV magazine cover.indd 1

24/03/14 13.27

65Since the Beijing Fourth World Conference on Gender and Development in 1995, the majority of multilateral and bilateral donors have created gender policies and strategies to promote equality in the design and delivery of development assistance. A broad range of institutions have undertaken multiple evaluations of gender, which have shown that mainstreaming has not succeeded in making gender a universal concern, and that gender equality results have been fragmented and have not been scaled up. In addition, many of the evaluations were conducted in isolation—with the exception of the Norad synthesis of eight organizational evaluations (2006)— and lacked a broader comparative assessment of findings across organizations.

There was a strong perception that mainstreaming gender equality is consistently underper-forming across the majority of donor organizations. But in the absence of a comprehensive synthesis review, such perceptions did not reveal the root causes or lead to solutions, and thus failed to influence policy or operational change strategies.

In response to this evaluation gap, the African Development Bank’s Independent Evaluation Department (IDEV) undertook this evaluation synthesis to examine experiences in main-streaming gender equality across multilateral and bilateral donor organizations, and, in so doing, to highlight trends (commonalities and differences) in findings, challenges faced, and good practices. This synthesis looked at 26 thematic and country evaluations, undertaken between 1990 and 2010, that focused on gender and/or women.

What makes this a Fit-For-Purpose Evaluation?Timeliness is crucial in producing evaluations that will have substantial influence in improv-ing the effectiveness of development interventions, and this analysis came at the right time to make a difference. Interest in gender issues was rising once again—and news stories continued to highlight the costs of a lack of gender equality. The synthesis also addressed a major, long-standing evaluation gap, as noted above.

Influence was embedded throughout the process, from the choice of what to evaluate, how to design and undertake the evaluation, to outreach and follow-up. The evaluation was developed in consultation with the intended users, and IDEV staff participated in Bank work to further spread the findings of the evaluation, including jointly managing the review of gender equality results of AfDB-funded public sector operations.

Lessons Learned for Fit-For-Purpose EvaluationsThis evaluation synthesis illustrates the importance of timing—the need to provide evalu-ative evidence when it is needed, when there is an appetite for the information, and when stakeholders have signaled a willingness to use it. It also emphasizes the importance of moving beyond evaluation of policy and strategy to look at what is going in the real world, to engage fully with stakeholders. Is gender mainstreaming affecting the lives of men and women? If so, how? How can development and evaluation practices be changed to move toward the desired results?


eVALUatiOn Matters

To sharpen the evaluation focus on these ques-tions, a pre-synthesis review was conducted of existing data, together with consultations with Bank gender and results-based specialists. The objectives were reassessed and adjusted as the synthesis progressed. The theory of change model was another tool that kept the focus fixed on the questions before the evaluators and moved the evaluation toward clear, action-able recommendations. The evaluation also demonstrated good practices in communica-tion and outreach, maximizing IDEV’s efforts to include the entire development community in taking a fresh approach to gender.

How were the Findings of this Evaluation Used?The evaluation synthesis informed the design of the Bank’s first gender strategy, Investing in Gender Equality for Africa’s Transformation (2014), and it was cited by the Special Envoy for Gender (SEOG) as an important piece of knowledge. It has also been cited and shared by a variety of organiza-tions that are currently grappling with the issue of gender equality, including the Organisation for Economic Co-operation and Development; the Swiss Agency for Development and Cooperation; the U.N. Educational, Scientific, and Cultural Organization; the Local Development Network of Africa; and the German Federal Enterprise for International Development, among others. The evaluation synthesis also provided inputs for the MDG Achievement Fund Secretariat, UNDP and UN Women, Two Roads, One Goal: Dual Strategy for Gender Equality Programming in the Millennium Development Goals Achievement Fund.

To promote continued use and expansion of the synthesis findings, the Bank has used its commu-nications tools, including the popular publication Evaluation Matters, to expand the conversation about gender and held “Mainstreaming Gender into Evaluation—An Evaluation Community Practice Event “ in Tunis (2014) to foster the kind of results-oriented evaluations that are needed.

66

Methodology, Main Findings, & RecommendationsA methodology that worksThe synthesis approach followed well-established methodological guidelines for conducting syntheses of evaluation studies and involved consultations with Bank gender experts at all stages, from development of the approach paper to discussion of conclusions and options for gender mainstreaming. The meth-odology was also linked to a theory of change for mainstreaming gender equity that was developed early in the synthesis process, based on the definition and principles of mainstreaming laid out by the U.N. Economic and Social Change Council.

The theory of change approach is an evaluation tool that maps out the logical sequence of means-ends linkages underlying a project, program, or approach. This begins by defining the intended impact of the process, and then moves toward the outcomes that mainstreaming activities aim to deliver. Having identified the beginning and end of the results chain, the theory then identifies the stages that must be passed through to move from the outcomes to achieving those impacts. Finally, the theory incorporates the assumptions that need to hold true if progress is to continue, and the forces that must be active to drive the process forward.

Findings and Recommendations: From Theory to ActionFindingsThe evaluations reviewed all pointed to the same finding: gender mainstreaming is a complex undertaking that has not been widely carried


67

out by the development community. The the-ory of change identifies four key assumptions and related drivers that need to be present to achieve the mainstreaming of gender equality: (a) effective leadership, (b) adequate finan-cial and human resources, (c) availability of appropriate procedures and processes, and (d) appropriate organizational incentives and accountability structures.

RecommendationsTo address these stumbling blocks, the evalu-ation recommends the following steps:• Utilize good practices such as linking

accountability to the implementation of measures to support gender equality.

• Place gender specialists at headquarters and in-country.

• Use a gender action plan at the intervention level; carry out follow-up mid-term reviews and/or annual performance reports to keep the pressure on management.

• Draw on the positive experiences in the education and health sectors, transferring lessons and good practices systematically to other sectors.

In addition, the evaluation evidence suggests that the substantial challenges presented by gen-der mainstreaming may call for fresh options:

• Gender focusing. Focus on sectors where gender equality appears to be sufficiently embedded and has made some progress, and create linkages with related sectors.

• Women in Development (WID) Plus. Many of the evaluations found that gender equal-ity is boiled down to a women-centered or women’s empowerment approach in interventions. This option would make the approach strategically explicit, building on the experiences that have delivered results,

but incorporating more fundamental analysis of gender power structures, with the intention of positioning interventions to empower women economically and politically.

• Policy dialogue on gender equality in new aid modalities. Gender and women’s empowerment mainstreaming, at the operational level, have been aimed at traditional project-based modalities. The increasing use of new aid modalities has created an additional set of challenges for integrating gender equality into interventions.

• Improve results reporting and learning through more systematic integration of monitoring and evaluation. Improving results and learn-ing within organizations depends on enhanced monitoring and evaluation within the context of the options outlined above. Gender monitoring and evaluation would be strengthened in sectors where gender equality and women’s empower-ment are the focus.

67

Main

streamin

g Gen

der Equality: A

Road to R

esults or a R

oad to Now

here?

FONDS

AFRICAIN DE DÉVELOPP

EMENT

AFRI

CAN D

EVELOPMENT FUND

BANQUEAF

RICAINE DE DÉVELOPPEM

ENT

Gender mainstreaming is in large part a theory about how development assistance can be more effective, efficient, inclusive, and equitable

in its delivery and results. The concept began to be applied in a broader public policy setting at the Third World Conference on Women, which

took place in Nairobi in 1985, and was definitively adopted by almost every multilateral and bilateral donor organization following the Beijing

Conference in 1995.

This evaluation synthesis examines experiences in mainstreaming gender equality across multilateral and bilateral donor organizations; and

in so doing, highlight trends (commonalities and differences) in findings, challenges faced, and good practices. The 26 evaluations reviewed all point to a similar finding: gender mainstreaming is a complex undertaking that has not been widely carried

out by the development community.

For more information on the evaluation, see the Synthesis Report: Mainstreaming Gender Equality: A Road to Results or a Road to Nowhere?

Mainstreaming Gender Equality:A Road to Results or a Road to Nowhere?



allocating resources for investment in RMCs; and providing policy advice and technical assistance to support development efforts.The mission of the Operations Evaluation Department is to help the Bank to foster sustainable growth and poverty reduction in Africa through


report on results in order to draw useful lessons and promote accountability.

Director: Rakesh Nangia, [email protected], Project Level Evaluations: Mohamed Manai, [email protected], High Level Evaluations: Odile Keller, [email protected] Evaluation Department, African Development BankBP 323, 1002 Tunis-Belvedere, TunisiaTel : (216) 71 102 841 Fax : (216) 71 194 460


FONDS

AFRICAIN DE DÉVELOPP

EMENT

AFRI

CAN D

EVELOPMENT FUND

BANQUEAF

RICAINE DE DÉVELOPPEM

ENT


2012

Mainstreaming Gender Equality:A Road to Results or a Road to Nowhere?

Caroline McCuen (Consultant) contributed to this article.

eVALUatiOn Matters

From the AfDB

field OfficesAlgeria field office | Angola field office | Burkina

Faso field office | Chad field office | Cameroon

field office | Central African Republic field office

| DRC field office | Egypt field office | Ethiopia

field office | Gabon field office | Ghana field office

| Guinea-Bissau field office | Liberia field office |

Madagascar field office | Malawi field office | Mali

field office | Mauritius Liaison office | Morocco

field office | Mozambique field office | Nigeria field

office | Rwanda field office | São Tomé and Príncipe

field office | Senegal field office | Sierra Leone field

office | Sudan field office | Tanzania field office |

Togo Field Office | Uganda field office | Zambia

field office | Zimbabwe field office

Bringing the AfDB closer to its clients:Mid-Term Review of the Decentralization Strategy and Process at the African Development Bank

Why was the evaluation conducted?The African Development Bank launched a Decentralization Roadmap in 2011 as an ambitious effort to bring the Bank closer to its clients and improve performance. The review asked five questions. 1. Has the Roadmap been implemented as planned? 2. Has decentralization generated the expected benefits? 3. What has decentralization cost the Bank? 4. Should the model for decentralization (that is, the

pattern of regional and country offices) be changed? 5. What should be the next steps for decentralization?

Main recommendations:• Cost savings through policy changes. Align the benefits

package for new staff deployed to Regional Resource Centers (RRCs) to those at headquarters; create a new ‘regional professional’ staff category to enable local professionals to operate intraregionally; rationalize travel for official missions from headquarters to RRCs.

• Improve uptake of the 2012 Delegation of Authority Matrices (DAM). Ensure rapid roll-out of information technology for project procurement to match the 2012 DAM processes; implement trainings and a case-based manual for the 2012 DAM.

• Complete planned review of business processes. This should include a review of the 2012 DAM and the implications of opening regional hubs in all regions under the Focused Presence Model.

• Standardize selection criteria and process for resident representatives. In reviewing the criteria for selection of resident representatives, emphasize the ability to work effectively in the matrix structure and to engage a wide variety at stakeholders.

• Improve the way analytical work is conducted in the field. Develop clear guidelines for publishing and dis-seminating analytical work from the field.

• Create expectation for staff to spend time in the field. Formalize requirements for field experience.

• Better equip Bank staff through training opportunities.• Review decentralization strategy of the Private Sector

Department (OPSM). Commit to sending more private sector staff to the field, including at least one to each

regional hub; organize ‘lessons learned’ visit with IFC where decentralization of staff has led to greatly increased project origination.

How has Bank used the findings?In the short term (within a year), the Bank planned to: • Make detailed plans for the shift to the Focused Presence

Model, including deciding on locations of regional hubs, slimmed field offices, and liaison offices.

• Decide on the major cost-saving measures and policies to be adopted (such as elimination of housing benefits, creation of a regional professional staff category, and adjustment of promotion criteria to include field-based experience).

• Adjust guidelines and metrics for analytical work to emphasize demand-driven knowledge products and track shorter products such as policy notes.

• Complete reviews of business processes and operations.• Introduce measures recommended to strengthen and

improve adoption of the DAM.• Revise selection criteria for resident representatives.• Conduct a review of Private Sector staff decentralization.

In the medium term (between one and three years), the Bank planned to: • Implement the bulk of the shift to the Focused

Presence Model.• Implement the cost-savings measures adopted.• Roll out new training programs for staff, monitor feed-

back through surveys, and adjust based on the feedback.

In the long term (beyond three years), the Bank planned to: • Complete the office network for the Focused Presence

Model.• Continue training programs and other ongoing

measures.

The Decentralization Roadmap laid out a 15-point Action Plan. Most of the Action Plan has been completed or is on track, except for some actions on policies and opera-tions- related issues.

Eric McGraw (Consultant) contributed to this article.

69


Mid-Term Review of AfDB Decentralization Strategy and Process

eVALUatiOn Matters


regional Integration — An Imperative for Greater Development Effectiveness?

Albert-Eneas Gakusi, Chief Evaluation Officer, AfDB

Fostering Regional Integration in Africa: An Evaluation of the Bank’s Multinational Operations, 2000–2010

What made this evaluation influential?

- It responded to an existing demand, and it was produced when the need for information was compelling

- Early consultation and communication with operations departments – key stakeholders – was well received and secured the support of these departments

72 Regional Integration—The Path to Greater Effectiveness? Regional integra-tion is essential for Africa to realize its full growth potential, participate in the global economy, and share the benefits of an increasingly connected global marketplace. The African Development Bank has long understood this, and regional integration has been part of the Bank’s mandate since 1963. However, it is only since the early 2000s that the Bank has significantly increased its assistance to operations designed to foster regional integration as a means to achieve tangible national and regional development goals. But all has not gone smoothly. For example, political pledges have not always been kept and poor infrastructure has proved a persistent barrier to success.

Purpose of the EvaluationFurther to the increased assistance for multinational operations, the Bank’s stakeholders expressed interest in an independent evaluation to determine whether these operations were performing as expected; whether they had attained or were likely to attain their objectives; whether the results were efficiently achieved and were sustainable; and what more the Bank could do and what it might do better going forward. Against this backdrop, IDEV launched an evaluation, Fostering Regional Integration in Africa: An Evaluation of the Bank’s Multinational Operations, 2000–2010, in response to Board members’ and senior management ’s need to know: (a) the relevance and consistency of the Bank’s strategic and operational framework for fostering regional integration; and (b) the relevance, efficiency, effectiveness, and sustainability of the Bank’s multinational operations. The evaluation would serve as an input for the Bank’s new strategic approach to this work and support the Bank’s efforts to accelerate regional integration.

What made this Evaluation Influential?The evaluation responded to an existing demand, and it was produced when the need for infor-mation was compelling.1

There was strong demand to know if multinational projects were succeeding or not. There was a common dissatisfaction with these projects, which actually take much longer to design and are more demanding to implement than others.

The evaluation team ensured early consultation with the operations departments. This was well received and secured the support of these departments for the evaluation – sup-port which continued until the end of the evaluation pro-cess. Any anxiety about the evaluation disappeared once the operations departments realized the nonpunitive intent of the evaluation, which aimed at documenting and explaining the underlying causes of successes and failures in order to inform the new strategic framework and operations design and implementation.

1 On the importance of timing in evaluation, see Vinod Thomas and Xubei Luo, 2012, Multilateral Banks and the Development Process, chapter 9, “Tarry Not—For Timing is (Almost) Everything.” Transaction Publishers.

FONDS AFRICAIN DE DÉVELO

PPEMENT

AFRICA

N DEVEL

OPMENT FUND

BANQUEAFRI

CAINE D

E DÉVELOPPEMENT

Fostering Regional Integration in Africa:

An Evaluation of the Bank’s

Multinational Operations, 2000-2010



2012


73The evaluation process was open and sustained by timely communication with operations departments, management, Board members, and other stakehold-ers, including municipalities, communities, and the private sector involved in the operations to be evalu-ated. There was transparency regarding the methods used to collect information. This fostered continuous dialogue between the evaluation team and operations departments and led to useful feedback from project task managers and higher-level managers.

With respect to the evaluation team, there was complementarity of internal and external exper-tise, supported by external peer reviewers with strong hands-on experience in both regional inte-gration and evaluation.

Finally, a Bank-wide seminar was organized to share and discuss the emerging findings and rec-ommendations from the desk reviews and inter-views with the operations departments. Another seminar was organized before the finalization of the evaluation so that the operations departments could offer their views of the pertinence of the find-ings, as well as the usefulness and implementability of the recommendations.

How were the Findings of this Evaluation Used?According to Gary T. Henry,2 there are three types of inf luential evaluations: those that determine the common good (raising a social, economic, or

2 Gary T. Henry, 2000, “Why Not Use,” New Direction of Evaluation 88: 85-98; 2003, “Influential Evaluations,” American Evaluation Association 24 (4): 515–24.

political problem); those that help select a course of action, and those that help adapt the course of action. The evaluation of multinational opera-tions served both the second and third functions, while the first function constitutes the raison d’être of the Bank.

The recommendations of the evaluation did not come as a surprise, given the open evaluation process and fruitful collaboration between the evaluation team and the operations departments. The recommendations were well received by man-agement, who agreed to address them with few qualifications. Some of the recommendations have already been implemented or are under way, including improved support for soft infrastructure, clarification of concepts used, and refinement of the classification and selection criteria for regional operations, among others. The evaluation also became an input to knowledge products and events created to deal with regional integration, including the new Bank Group Regional Integration Policy and Strategy, 2014–2023 adopted on 15 November 2014, and learning events, including “On the Road to Transformation: Learning from Experience.” It also served as an input to the preparation of the Bank’s 10-year strategy, At the Center of Africa’s Transformation, Strategy for 2013–2011.

Lessons Learned for Fit-For-Purpose EvaluationsElements that contributed to the usefulness of this evaluation:

• Timely planning to respond to a pressing need for answers. • Interaction with stakeholders from the conceptual stage of the evaluation onward.• Synergy between a strong internal evaluation team, competent consultants who were eager to

deliver, and experienced external peer reviewers with relevant skills in the topic to be evaluated and in evaluation.

• Transparent communication and methodological approaches.

eVALUatiOn Matters

74

Main Findings and RecommendationsThe evaluation found that the Bank has developed an increasingly coherent strategic and operational framework to guide its assistance toward regional integration, but that the strategic framework required further focus and fine-tuning in its (a) clarification of concepts used within the Bank, such as regional integration and cooperation, multina-tional operations, regional operations, and public goods; (b) selection of the soft constraints to regional integration, which include institutional, regula-tory, and administrative bottlenecks that need to be addressed for infrastructure operations to fully benefit regional integration; (c) clarification of the role of private sector operations in complement-ing the Bank’s public sector operations to develop regional value chains, integrate financial markets, finance trade, and build the capacity of African regional institutions. At the same time, the evalu-ation noted that there was limited elaboration of the areas where the Bank could bring its strengths and value added in addressing specific regulatory and administrative constraints to the development of integrated regional markets.

The evaluation noted that the Bank’s capacity to implement its regional integration mandate sig-nificantly improved with the creation of NEPAD (New Partnership for Africa), Regional Integration and Trade Department (ONRI). However, the ambi-tious mandate of ONRI was not matched by exist-ing resources, and the roles and responsibilities of ONRI were not clearly delineated. The evaluation emphasized that the business model of the Bank was still constructed for single-country operations and was not adapted to the specific requirements of multinational operations. It also showed that a mechanism for systematic feedback and learn-ing from experience to influence the design of new multinational operations was not in place.

Methodology of the evaluationBecause of complexity of the questions to be addressed, as well as of the context of regional integration in Africa, where multiple financ-ing agencies intervene, the evaluation utilized a triangulation of different sources of information collected using several approaches. The evalua-tion was built on an early consultation with the operations departments to understand the issues in question, to formulate the evaluation ques-tions, and to define the methodology. Sources included: (a) a Policy and Strategy Review; (b) a Portfolio Review; (c) a Quality at Entry Review; (d) three case studies of multinational operations in the infrastructure sector in East Africa, Southern Africa, and West Africa; (e) three Project Performance Evaluations; (f) inter-views with selected development partners; (h) a literature review; (g) benchmarking with donor institutions; and (i) focus group interviews with task managers, managers at the Bank’s headquar-ters, and beneficiaries (regional institutions, the private sector, and communities) during field visits.


75

The evaluation observed that the Bank had increased its share of multinational operations from 6 percent (2000) to 15 percent (2010) of total Bank approvals, and these operations have responded to compelling needs while addressing key sectors for economic and social development, such as power, transport, educa-tion, health, and trade. The multinational operations generally achieved their objectives no less effectively than single-country operations, even though they are exposed to more risks. Key factors in perfor-mance include country commitment and ownership, implementation, and governance arrangements, as well as a conducive policy environment. It revealed that a great number of multinational operations are merely replicated in different countries, but did not have the requisite integration properties. They rep-resented 40 percent of the total of 201 multinational operations approved between 2000 and 2010.

Recommendations to the Bank:1. Clarify and strengthen the strategic focus of the approach to regional integration. Develop distinct definitions, consistent across the Bank, of regional operations and multinational operations contributing to regional integration.

2. Be more selective when addressing the soft con-straints and regional public goods (RPGs). For soft constraints, one option would be to concentrate on the bottlenecks of the regulatory and administrative framework that choke off regional integration. For RPGs, the Bank could define a limited number of areas where it has appropriate expertise and where it can contribute the most compared with other donors.

3. The Bank should better define the role of private sector operations in regional integration. Project appraisal reports of the private sector window often indicate a contribution to regional integration, although there is no clear results framework dem-onstrating a direct link with regional integration.

4. Establish a mechanism for systematic feed-back and learning from the Bank’s experience with multinational operations to influence the design of new operations, especially in relation to the key factors of performance. This mechanism should specify clear responsibilities for the collec-tion, validation, analysis, and use of the information for policy and progress formulation.

5. Clearly define the roles, responsibilities, and division of labor among ONRI, regional depart-ments, and sector departments. In addition, the responsibilities of field offices should be more clearly defined and their capacity strengthened to engage in strategic policy dialogue on regional integration issues.

6. Adapt the Bank’s tools and business model to the specificities of multinational operations. Necessary measures include: (a) defining a set of specific criteria for the Multinational Operations Readiness Review; (b) assigning overall responsi-bility for those operations to one task manager; (c) allocating more time and resources for design and supervision; (d) reconsidering the format of the Project Appraisal Report for multinational opera-tions; and (e) adapting the incentives for staff to engage in complex, cross-sectoral operations.

AcknowledgmentsThis paper draws on seven reports prepared by Independent Development Evaluation on multinational operations. It benefitted from useful comments by Laura Delponte, Samson Houetohossou, John Eriksson, and Fredrik Söderbaum. They worked on the evaluation of multinational operations financed by the African Development Bank as regional integration policy analyst, research analyst, and external peer reviewers, respectively. The evaluation report is avail-able on the AfDB internet site: http://operationsevaluation.afdb.org/en/evaluations-publications/

eVALUatiOn Matters

Recent IDEV Evaluation Reports

From experience to knowledge... From knowledge to action... From action to impact


Chad:Evaluation of the Bank Group

Country Strategy and Program in Chad

(2002–2012)Summary Report

An ID

EV C

ount

ry S

trat

egy

Eval

uatio

n

July 2014

Strategizing for the “Africa We Want”:

An Independent Evaluation of the Quality at Entry of Country

and Regional Integration Strategies

Summary Report


An ID

EV C

orpo

rate

Eva

luat

ion

January 2015



August 2014

An ID

EV C

orpo

rate

Eva

luat

ion

Operational Procurement Policies and Practices

of the African Development Bank: An Independent Evaluation

Summary Report


An ID

EV S

ecto

r Eva

luat

ion

December 2014

Transport in Africa: The African Development

Bank’s Intervention and Results for the Last Decade

Summary Evaluation Report



Botswana:Country Assistance Evaluation

2004–2013Summary Report

An ID

EV C

ount

ry S

trat

egy

Eval

uatio

n

No projects

< UA 2 million

< UA 2 to 5 million

< UA 2 to 5 million

< UA 2 to 5 million

No projects

< UA 2 million

< UA 2 to 5 million

< UA 2 to 5 million

< UA 2 to 5 million

Fostering Inclusive Finance in Africa:

An Evaluation of the Bank’s Microfinance Policy,

Strategy and Operations, 2000–2012Summary Report



An ID

EV S

ecto

r Eva

luat

ion

eVALUatiOn Matters

[email protected]


[email protected]

Identification Preparation and

formulation

Review and approval

Implementation

Evaluation

From Experience to Knowledge … From Knowledge to Action … From Action to Impact

idev.afdb.org


“ Independence is a necessary condition for evaluation influence. Of course, influence can be exercised from multiple social and organizational roles, including ultimately being part of the decision making process itself. But to be influential on the basis of evaluative knowledge requires impartiality and objectivity, distance from the decision maker, and capacity to deal with conflicting interests. P 16

EvAluATIOn InfluEncE IN DEVELOPMENt OrGANIzAtIONs