Final Program Assessment Report Spanish Impact Evaluation Fund (SIEF)

Final Report 1

June 2nd, 2012

1 This report was prepared by Osvaldo Nestor Feinstein, independent consultant. Very valuable inputs and support were provided by Carlos Asenjo Ruiz. Useful comments on an earlier version were received from Laura Rawlings, Adam Ross, Paloma Acevedo and Carlos Asenjo Ruiz. Thanks are also due to the persons interviewed, mentioned in Annex ii, for their time and insights. The views expressed in this report are those of the author and do not represent the views of the World Bank or of the Government of Spain.

0

ABBREVIATIONS AND ACRONYMS

BNPP Bank-Netherlands Partnership Program CCT Conditional Cash TransfersDAC Development Assistance CommitteeDEC Development Economics VicepresidencyDIME Development Impact Evaluation InitiativeDFID Department for International Development ECD Early Childhood DevelopmentE2P from Evidence to Policy NotesFY Fiscal YearGOS Government of SpainHD Human DevelopmentHDN Human Development NetworkHDNCE Human Development Chief Economist's OfficeHDNSPHuman Development Network Social Protection HRITF Health Results Innovation Trust FundIADB Inter-American Development BankICM Implementation Completion MemorandumIE Impact EvaluationIEG Independent Evaluation GroupMAEC Ministry of Foreign Affairs and CooperationMDBs Multilateral Development BanksMEH Ministry of FinancesMfDR Management for Development ResultsM&E Monitoring and EvaluationODA Official Development AssistanceR ResultSC Steering CommitteeSIEF Spanish Impact Evaluation Fund3iI International Initiative for Impact EvaluationTF Trust FundTOR Terms of ReferenceTTLs Task Team LeadersVP Vice PresidencyWB The World Bank

1

Table of Contents

Executive Summary

1. Introduction

2. Design

3. Processes

4. Results

5. Conclusions and recommendations

Annexes

I Terms of Reference

II List of persons interviewed

III List of impact evaluations funded by SIEF and their products (Excel file attached)

IV SIEF Results-based strategy matrix (Excel file attached)

V TTL’s survey questionnaire

VI Preguntas para las reuniones en España

VI Key Performance Indicators and Financial Summary

2

Executive Summary

The SIEF final report is aimed to be used as an input for potential program continuation and/or up-scaling, as well as for similar programs; for the Implementation Completion Memorandum (ICM) report; and for record-keeping purposes. The goal of the report is to assess the extent to which the SIEF has achieved its strategic goals. Furthermore, this report complies with what is stated in the July 2007 Administration Agreement between the government of Spain and the World Bank (which established the SIEF program): “Within six months of the final disbursement date specified in paragraph 10.1, the Bank shall furnish to the Donor a final report on the Project activities”.

Methodology

The methodology used in the report is a combination of different approaches: desk review of SIEF documentation, interviews to key informants, a survey to task team leaders (TTLs), included as annex V, with a response rate of 75%, a literature review of impact evaluation and a field visit to Argentina. Furthermore, use has been made of generally accepted evaluation criteria. Finally, a set of assessment questions included in the Terms of Reference (TOR) guided the design of the questionnaire as well as the preparation of this report. It is to be noted that this report does not attempt to assess SIEF’s impact. However, evidences of SIEF’s policy influence will be mentioned.

Design

The SIEF was established in July 2007 as the result of negotiations between the Government of Spain (Foreign Affairs & Cooperation and Finance Ministries) and the World Bank (Human Development Network). At that time the Government of Spain (GOS) had a budget surplus of approximately 2% of GDP and a commitment to increase ODA to the UN goal of 0.7% of Gross National Income. In addition, the Government of Spain (GOS) was keen in promoting impact evaluation both in the development world, and also of its own aid portfolio (particularly in the area of human development) and it perceived the World Bank as a leader in this field. Furthermore, the GOS was interested in contributing to the generation of rigorous evidence on human development interventions as a global public good.

At the World Bank (and also at other development agencies) there was an intensification of interest on impact evaluation by 2005, responding to the demands for credible evidence on the results of development interventions”. Thus, the Trust Fund for Impact Evaluation and Results-based Management in Human Development Sectors, or Spanish Impact Evaluation Fund (SIEF), was launched in July 2007 by the World Bank and the Government of Spain (GOS) aiming to support the Impact Evaluation (IE) of innovative programs to improve Human Development (HD) outcomes. Its ultimate goal is to contribute to an increase in the effectiveness of development policies by increasing the evidence base on the impact of actions affecting HD.

3

SIEF was endowed with a grant from the GOS of 10.4 million euros, which were received in a single tranche by the Bank in October 2007 (the amount in dollars corresponded to 14.9 million, given the exchange rate at that time). The amount of the grant was decided on the basis of a number of impact evaluations2. Furthermore; the GOS funded two professionals through the Externally Funded Staffing Program to work at the World Bank on SIEF related activities. GOS’s contribution was later supplemented with an additional grant of USD$ 2.1 million (equivalent at that time to 1, 5 million pounds) provided by DFID, starting with an amount of USD$ 500,000 in March 2008. SIEF has been the largest trust fund ever established in the World Bank focused on impact evaluation (and, indeed, for any type of evaluation). The original period of the grant was from July 2007 until July 2010, and it has been extended until June 2012.

SIEF’s strategic goals were the following: To build capacities for understanding and using impact evaluation techniques; to support the application of impact evaluation techniques to selected development programs to improving human development outcomes; to disseminate SIEF findings, and to promote access to knowledge on the impact of human development programs; to use the SIEF as a platform for promoting management for development results. In addition, Annex 1 of the Trust Fund Administration Agreement also included the strengthening of the Donor’s capacity (as part of the objetive of building capacity for the planning, management and evaluation of development programs).

Processes

The SIEF program has been managed by the Bank’s Human Development Network Office of the Chief Economist. SIEF resources supported: (i) Prospective, rigorous impact evaluations of programs in 11 eligible Human Development and Sustainable Development sectors and 72 eligible developing countries across all regions; (ii) Intensive training programs for government counterparts, Bank staff, and staff of partner development agencies in impact evaluation methods; and (iii) Publication and dissemination of evaluation results through articles, meta-studies, and web-based materials.

By July 2007 SIEF made its first call for proposals from existing HDN Impact Evaluation clusters, trying to identify “Quick Win” IE studies, “in order to get the SIEF off to a very visible, quick start, and to help ensure that the Spanish resources produce some tangible results as early as possible”. These proposals were called “quick wins” as they were expected to capitalize on work on Human Development issues that had already been going, searching for proposals of high quality, prospective impact evaluations that had immediate funding needs. The chosen proposals were submitted to SIEF’s Steering Committee. The “quick wins” allowed SIEF to contribute in the short-term to high quality evaluations, but SIEF did not obtained visibility from this first stage. On the other hand, these allocations of funds to “Quick Wins” (approximately 6% of SIEF’s resources) provided SIEF with an opportunity to leverage other sources of funds. During the following years, the bulk of SIEF’s resources were allocated to the new clusters of IEs.

Considering activities by their basic categories, table 3 shows that 69% of the resources were disbursed for impact evaluation research, and that program management absorbed less than 5% of the resources

2 Annex I of the Trust Fund Administration Agreement indicated each year the Trust Fund will support at least 5 rigorous impact evaluations (with a target of 15 evaluations in the period of 3 years),

4

The calls for proposals required a rigorous specification of the counterfactual, and randomized designs have been the most frequently used technique in the IEs funded by SIEF (as reported in 74% of the answers to the TTLs survey). The rest of SIEF funded IEs used regression discontinuity, propensity score matching and other quasi-experimental or observational approaches.

A peer review process of proposals to be funded by SIEF was put in place as a quality control mechanism. This process combined rigor (with an internal and external revision) and flexibility.

The turnover of persons involved at the leadership and operational levels of GOS’s international cooperation, and the shortage of staff in the evaluation unit of the MAEC (referred to in the 2011 DAC’s peer review of Spain), which had to play a crucial role on the donors’ side, limited the active participation of Spain in SIEF’s follow-up. SIEF’s key monitoring instrument has been a dynamic spreadsheet, which was a real-time monitoring tool, that allowed for an active management of SIEF resources, leading to timely transfer of funds to SIEF activities. In addition, the spreadsheet was used as a basic source of information for the Annual and Mid-Year reports. These instruments were useful for decision making concerning funds allocation and reallocations, which facilitated the implementation of the program.

Results

An important feature of the SIEF is that task team leaders leveraged other sources of funds to achieve results. In some cases funding commitments of local counterparts (e.g., in the case of a CCT in Indonesia, in-country resources provided USD$ 2.8 million of which SIEF contributed USD$ 300,000); but also other funding sources were leveraged, such as the International Initiative for Impact Evaluation (3ie) in the case of the IE of Malaria, in Kenya.

Policy Notes, “From Evidence to Policy (E2P”): The SIEF has supported the elaboration of this series of notes, disseminating the results of IE in the HD area. The first E2P note was published in September 2010, and 11 E2P Notes were produced by February 2012.

Books: Impact evaluation results have been published in the following books “Conditional Cash Transfers: Reducing Present and Future Poverty” (translated to Spanish by SIEF), “Making Schools Work: New Evidence on Accountability Reforms” and “No Small Matter: The Impact of Poverty, Shocks, and Human Capital Investments in Early Childhood Development”. Furthermore, an impact evaluation training handbook “Impact Evaluation in Practice” has been produced, and is available in English, Spanish and French.

Other publications: In addition to the policy notes and books, impact evaluation teams supported by the SIEF have produced 25 research papers and articles.

Presentations: SIEF supported over 47 presentations in international conferences and other discussion fora as a means for knowledge sharing and dissemination;

5

Media coverage: In addition, as indicated in the footnote to table 1, some prestigious and influential publications, such as The Lancet, The Economist, the Journal of Economic Literature, and newspapers such as The Financial Times and The Wall Street Journal made reference to outputs produced with SIEF funds.

Datasets: Impact evaluations funded by SIEF have generated data useful not only for evaluation but also for the design of new interventions. In addition, the datasets are submitted to the micro-data catalogue in order to facilitate access.

Impact evaluation training. The objective of the “Turning Promises to Evidence” workshops, the key means by which the SIEF promoted training and capacity building in impact evaluation, has been to provide participants with an overview of impact evaluation. These workshops are one-week regional courses focusing on the design, implementation and management of impact evaluations, tailored to project teams of government counterparts in client countries, Bank staff, and staff of partner development agencies. The main goal of the workshops has been to increase the effectiveness of human development programs by building impact evaluation and results-based management capacities in client countries and partner agencies. Since January 2008 SIEF supported 17 workshops of this kind with $1,500,000. The workshops were conducted in all regions of the developing world, and they also had financial support from the United Kingdom’s Department for International Development (DfID), which provided $475,000, and the Bank-Netherlands Partnership Program (BNPP). The number of participants from client countries and partner agencies trained in impact evaluation techniques in these workshops has been 24063, corresponding to 251 project teams. On average, in each workshop there were participants of 9 countries. The results of the tracer study suggest that the “Turning Promises to Evidence” training program has been effective in building impact evaluation capacities in client countries, and in promoting communities of practice among impact evaluation and human development practitioners and experts.

The following table provides a summary of outputs produced by SIEF:

Types of Products NumberImpact evaluation studies 52Concept notes 47Baseline surveys 10IE reports (unpublished) 12IE articles or papers (publications) 25Books (or chapters in books) 4Regional Workshops 17Participants at the regional workshops

2 406

Examples of the use of SIEF supported IE and of their policy influence

The last section of chapter 4 includes several examples of the use of IE supported by SIEF and particularly of their policy influence, which are indications of their outcomes.3 The target indicator for participants trained on IE, and the number of regional workshops, in the MfDR SIEF matrix were. respectively, 1893 and 12 (compared to the actuals of 2406 and 17).

6

Conclusions and Recommendations

The Spanish Impact Evaluation Fund made an important contribution to the development of credible evidence on the impact of human development interventions in different thematic areas such as health contracting/pay for performance reforms, conditional cash transfers, malaria control, active labor market programs/youth employment, basic education accountability, HIV/AIDS prevention and early childhood development. Furthermore, SIEF made a significant contribution to the development and consolidation of impact evaluation practice at the World Bank and in developing countries. Although this work was done in the context of developing and transition countries, it may also be of potential interest for Spain not only for its international cooperation projects and programs but also as an input for the design of Spain’s own domestic policies (for example, the results on active labor market programs and youth employment4), and to strengthen its evaluation capacity. On the other hand, SIEF has done less work on systematic reviews of evidence generated by impact evaluations, apart from a set of books and the organization of cluster conferences.

A peer review process of proposals to be funded by SIEF was put in place as a quality control mechanism. This process combined rigor and flexibility with transparency.

By working in partnership with other funding sources, SIEF was able to leverage its resources. However, the blending of funds makes practically impossible to disentangle SIEF’s contribution from that of other donors.

The SIEF cluster model proved to be a useful approach that can be replicated 5. Allocating resources to thematic clusters of IEs rather than to individual IEs facilitated the consolidation of IE communities. SIEF contributed to strengthen IE communities of practice in different HD areas, through on the job training, training workshops and dissemination events. This strengthening has taken place to a greater extent in some clusters, such as Health Contracting / Pay For Performance Reforms and Active Labor Market / Youth Employment. Furthermore, there was less knowledge sharing across clusters, which mainly took place through the regional IE workshops and six monthly (initially quarterly) meetings, convened by the SIEF program management team.

The approach to training introduced an innovative two-tracks format, which matched well the different demands from policy makers, evaluation managers and evaluation specialists. This approach strengthened both the demand and the supply for IE at country level and also at the World Bank Human Development Network, where it played a catalytic role. On the other hand, SIEF has not strengthened evaluation capacities of the Spanish donor (which was one of the objectives included in the agreement establishing the SIEF), except for the IE workshop held in 4 An example of an output generated by SIEF that can be useful for Spain along the lines indicated in this paragraph is http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1239047988859/5995659-1239048041095/cf4_zMisc_ALM_SummaryNotes_SIEF-IE-on-ALMP_Jan5_PA_110121.pdf5 The cluster approach for the evaluation of grants was introduced by the Kellogg Foundation in the 90’s, particularly for small grants. Some lessons learned from its experience could be useful in SIEF II. See http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/evaluation-in-the-21st-century/cluster-evaluation http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx

7

Madrid and the technical advice provided to project teams funded by Spanish cooperation which were interested in designing IE.The “Impact Evaluation in Practice” book, produced by SIEF, and translated into Spanish and French, is a very valuable resource that can be used in the development of impact evaluation capacity in developing countries, the World Bank and in Spain.

Applying a six-point scale ratings for outcomes (Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory) to the implementation of the four SIEF goals, combining evidence from different sources (survey to TTLs, documentation and website review, interviews) the ratings are the following (chapter 4 provides ratings for each agreed result corresponding to the different goals, and the rationale as well as the evidence for the ratings): goal 1) to build capacities for understanding and using Impact Evaluation (IE) techniques, Highly Satisfactory; goal 2) to support the application of impact evaluation techniques to selected development programs to improving Human Development (HD) outcomes: Highly Satisfactory; goal 3) to disseminate SIEF findings, and to promote access to knowledge on the impact of HD programs, Satisfactory; and goal 4) to use the SIEF as a platform for promoting Management for Development Results (MfDR), Satisfactory.

An indication of the success of SIEF is that, as it comes to an end, a new multi-donor trust fund for impact evaluations has been established at the World Bank, with an initial endowment provided by the British government’s Department for International Development (a secondary source of funding for SIEF I), with an amount which triples the funding of SIEF. In addition, it is significant that the new fund retains the acronym SIEF (with the “S” now standing for “Strategic”), which is an acknowledgement of the quality of the operations and visibility achieved by the Spanish Impact Evaluation Fund. As stated in the presentation of the new SIEF, it “will build on the knowledge and research generated by the Spanish Trust Fund for Impact Evaluation”.

Recommendations

A new phase of SIEF could focus on systematic reviews of IE results, combining the evidence generated by SIEF funded IEs with evidence from other IE and/or studies, which may allow to go beyond “what works” (a phrase frequently used in SIEF’s publications) to “what works in which circumstances and why”, strengthening the external validity and usefulness of IE findings. There is still a significant proportion of IEs funded by SIEF which are in the process of delivering final results. As their results become available, it will be possible and convenient to produce systematic reviews of IE results.

The more that impact evaluations are designed and implemented for the same type of interventions in different contexts, the greater the extent to which it would be possible to make statements concerning the external validity (or generalizability) of the impact evaluation results.

It would be worthwhile to support also innovative evaluations (as distinct from evaluations of innovative interventions), such as, e.g.,"selective trials”.

In order to increase the flexibility of an impact evaluation fund, it could be worthwhile to complement the periodic calls for proposals with an open and continuous window to support with small contributions the design of evaluations (a sort of seed capital), which could facilitate the finalization of the design of IE proposals sketched at the IE workshops.

8

In a new fund it would be convenient to promote a wider competition for the funds, facilitating the inclusion of non Bank staff in the teams participating in the call for proposals. This could be complemented with the participation of non Bank academics and/or policy makers as external advisors to the technical committee, so as to ensure a perception of full transparency and to draw on a wider pool of expertise.

Sharing experience and knowledge gained across thematic clusters through periodic meetings , or a forum, may support the development of a community of practice of impact evaluators and could lead to economies of scope. An additional means for sharing knowledge would be an appropriate website (as indicated in the next recommendation).

The development and maintenance of an updated, comprehensive and user-friendly website with information about the program of impact evaluations and results achieved should be one of the key means to share knowledge generated by the program. In order to ensure that the website fulfils its role it would be worthwhile to conduct periodic revisions of the website and to budget adequate resources for the website design, maintenance and revision. Furthermore, it would be convenient to explore the possibility of developing an independent website (e.g., www.sief.org), not subject to the constraints that should be followed when it is a World Bank (www.worldbank.org) website, taking into account that it is a partnership program funded by a donor. The program’s website could have an intranet area for internal monitoring of the program and to facilitate information sharing about processes to be followed and intermediate products.

A proactive approach to dissemination could include during the implementation of the impact evaluations a set of activities (such as videos and photographs) that with a very low marginal cost could enhance significantly the quality of dissemination and, as a by-product, and in a cost-efficient way, it would make available audiovisual material for training purposes.

On capacity building for impact evaluation, it would be worthwhile to complement the very successful impact evaluation training workshops designed and implemented by SIEF, with the support to the development of institutional capacities for independent impact evaluation at the country level

Finally, it would be worthwhile to promote a broader engagement with evaluation communities beyond the World Bank, such as the American Evaluation Association, the European Evaluation Society, the International Development Evaluation Association, the United Nations Evaluation Group and the MDBs Evaluation Cooperation Group. IEs and systematic reviews could be disseminated to them, and presentations could be made in their conferences and/or meetings, seeking their feedback.

9

1. Introduction

Purpose of the assessment

The SIEF final report is aimed to be used as an input for potential program continuation and/or up-scaling, as well as for similar programs; for the Implementation Completion Memorandum (ICM) report; and for record-keeping purposes.

The goal of the report is to assess the extent to which the SIEF has achieved its strategic goals. Furthermore, this report complies with what is stated in the July 2007 Administration Agreement between the government of Spain and the World Bank (which established the SIEF program): “Within six months of the final disbursement date specified in paragraph 10.1, the Bank shall furnish to the Donor a final report on the Project activities”.

Methodology

The methodology used in the report is a combination of different approaches: desk review of SIEF documentation, interviews to key informants, a survey to task team leaders (TTLs), with a response rate of 75%, a literature review of impact evaluation and a field visit to Argentina. Furthermore, use has been made of generally accepted evaluation criteria. Finally, a set of assessment questions included in the TOR (see Annex 1) guided the design of the questionnaire as well as the preparation of this report.

It is to be noted that this report does not attempt to assess SIEF’s impact.6 However, some evidences of SIEF’s policy influence will be mentioned.

2. Design

The SIEF was established in July 2007 as the result of negotiations between the Government of Spain (Foreign Affairs & Cooperation and Finance Ministries) and the World Bank (Human Development Network). At that time the Government of Spain (GOS) had a budget surplus of approximately 2% of GDP and a commitment to increase ODA to the UN goal of 0.7% of Gross National Income. In addition, the GOS was keen in promoting impact evaluation both in the development world, and also of its own aid portfolio (particularly in the area of human development) and it perceived the World Bank as a leader in this field. Furthermore, the GOS was interested in contributing to the generation of rigorous evidence on human development interventions as a global public good.

At the World Bank (and also at other development agencies) there was an intensification of interest on impact evaluation by 2005, responding to the demands for credible evidence on the results of development interventions, and in 2006 impact evaluations were established as a “new product line”. 7

6 For some of the difficulties in carrying out an impact evaluation of impact evaluations seehttp://siteresources.worldbank.org/INTISPMA/Resources/383704-1146752240884/Doing_ie_series_14.pdf and for the more general issue of evaluating the use of evaluations, see Feinstein, O.N. (2002) “Use of evaluations and the evaluation of their use”.7 For a brief presentation on the evolution of impact evaluation at the World Bank, highlighting the importance of SIEF, see the presentation “Assessing the Use and Influence of Impact Evaluations:

10

Thus, the Trust Fund for Impact Evaluation and Results-based Management in Human Development Sectors, or Spanish Impact Evaluation Fund (SIEF), was launched in July 2007 by the World Bank and the Government of Spain (GOS) aiming to support the Impact Evaluation (IE) of innovative programs to improve Human Development (HD) outcomes. Its ultimate goal was to contribute to an increase in the effectiveness of development policies by increasing the evidence base on the impact of actions affecting HD. The following diagram describes briefly the “theory of change” embodied in SIEF

Diagram 1: SIEF’s “Theory of Change”

SIEF´S RESOURCES

ACTIVITIES RESULTS

EvaluationsKnowledge Enhanced

DevelopmentEffectiveness

Workshops

Dissemination Capacities

SIEF was endowed with a grant from the GOS of 10.4 million Euros, which were received in a single tranche by the Bank in October 2007 (the amount in dollars corresponded to 14.9 million, given the exchange rate at that time). The amount of the grant was decided on the basis of a number of fully funded impact evaluations Furthermore, the GOS funded two professionals through the Externally Funded Staffing Program to work at the Word Bank full-time on SIEF related activities.

GOS’s contribution was later supplemented with an additional grant of $ 2.1 million (equivalent at that time to 1,5 million pounds) provided by DFID, starting with an amount of $ 500,000 in March 2008. SIEF has been the largest trust fund ever established in the World Bank focused on impact evaluation (and, indeed, in any type of evaluation8). The original period of the grant was from July 2007 until July 2010, but it has been extended until June 2012.

SIEF’s strategic goals were the following:

1. To build capacities for understanding and using impact evaluation techniques;

2. To support the application of impact evaluation techniques to selected development programs to improving human development outcomes;

Evidence from the World Bank Group” by Izlem Yenice, World Bank Independent Evaluation Group, at the 2011 Conference of the American Evaluation Association, which includes preliminary findings of IEG’s forthcoming review of IEs.8 Just as a comparison, the Independent Evaluation Group of the World Bank had in FY 12 a total of $1,2 million as trust funds for all types of evaluations, and its budget for that year included only one impact evaluation and one meta impact evaluation. For FY 13 the same number of impact evaluations were planned. See http://ieg.worldbankgroup.org/content/dam/ieg/IEGWorkPgrmAndBudget2011.pdf

11

3. To disseminate SIEF findings, and to promote access to knowledge on the impact of human development programs;

4. To use the SIEF as a platform for promoting management for development results.

The degree of achievement of these goals is rated, at the end of section 5 (see also annex iii), on the basis of the agreed results for each of them, as per the SIEF results-based strategy (which also includes indicators for the different results).

The following sections of this chapter address the design questions included in the TOR.

Consistency of SIEF design with its goals

– To what extent the SIEF design and resources were consistent with its aimed goals?

There was high consistency between SIEF goals and its design and resources.

A partial answer to this question can be provided at the micro-level, through the answers to the grant completion surveys, which included the following question: “Were the resources allocated to your task(s) consistent with its aimed goals?”. 33 out of 34 answers were positive (the negative answer corresponded to one case in Senegal in which the task team leader had the option to reallocate the resources to another IE in Kenya).

Furthermore, for a more global consideration of this question, it is useful to take into account 3ie’s 5 years budget of US$100 million, allowing 50-60 new studies each year (combination of quick wins, large studies and evaluation design) 9. Though the number of annual studies exceed those of SIEF (it corresponds to the number of studies funded by SIEF during its full period, as reported below in table 2), 3ie annual budget would be of US$ 20 million, without including capacity building activities, an annual budget that is well above SIEF’s resources for its original three year period (which with the extension until June 2012 became a five years period). Strategy and M&E Plan

– To what extent did the SIEF have a useful and reliable Strategy and M&E Plan that contributed to set priorities and track progress towards the achievement of its goals?

After its first 2 years SIEF developed a useful and reliable Strategy and M&E Plan.

The SIEF Results-based strategy matrix (see annex iv), developed after its first two years (when SIEF’s core team was strengthened), provided SIEF with a useful, operational strategy and framework for M&E, which was complemented with a dynamic spreadsheet used as a real-time monitoring tool. The matrix linked in an explicit way the goals with the expected outcomes and the corresponding indicators, outputs, activities and resources for the Fund’s original timeframe (it was not updated to cover the period of SIEF’s extension).

9 See http://www.google.es/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&ved=0CHwQFjAJ&url=http%3A%2F%2Fwww.oecd.org%2Fdataoecd%2F4%2F54%2F41681801.ppt&ei=rztrT_y2M-mp0QWB29ndBg&usg=AFQjCNEP8Qui0B9FYL_-CzGmgxJraomDdA&sig2=qx2M1PA64k-riHWE0yWEaw

12

Results Dissemination and Knowledge Sharing Plan

– To what extent did the SIEF have a useful and reliable Results Dissemination and Knowledge Sharing Plan?

The Plan was useful and reliable but it did not considered explicitly the GOS.

The Results Dissemination and Knowledge Sharing Plan was approved by the SC and it identified well the audiences (with one exception), as shown in diagram 2. The audience that was not identified in the chart is the GOS (the exception referred to above). It would have been worthwhile to identify Spanish organizations (both those involved in cooperation and also those dealing with human development issues) among SIEF’s audiences.

A more general way of framing this point is that in a trust funded program it is convenient to include donors/ partners as one of the audiences for the dissemination of the program’s results.

The implementation of the SIEF dissemination plan was decentralized. Cluster dissemination proposals were submitted, with an initial ceiling of USD$100,000 (cluster leaders were allowed to submit proposals up to USD$185,000 for cluster dissemination and coordination activities). These proposals indicated the target audience, timeline (with a June 2012 deadline) and the proposed budget allocation, for each cluster dissemination activity, indicating the corresponding TTL responsible for the activity. The types of activities included the production, editing and translation of executive summaries of IEs financed by the clusters, organization of (and participation in) conferences and workshops, short notes, and website maintenance. The objectives of the workshops were to present key results, to discuss their implications for policy as well as their external validity, bringing together policy makers, researchers and operational staff, in some cases sponsored by the SIEF and other sources. These workshops complemented, and in some cases were partially based on, the regional SIEF workshops (which are discussed below in section 4).

13

DFID IE TF

Diagram 2 SIEF types of audience as presented in SIEF’s Dissemination Plan

3. Processes

This chapter addresses the TOR’s questions concerning processes:

Efficiency of SIEF Program Management Model

– To what extent has the SIEF program management model (i.e. instruments; economic, human and technical resources; organizational structure; information flows; decision-making in management) been efficient in comparison to the results attained?

The SIEF program management model was efficient.

The SIEF program has been managed by the Bank’s Human Development Network Office of the Chief Economist.SIEF resources supported:

14

WBG evaluation & knowledge

management units: IEG,

PREM, DEC

WBG IE community:

DIME

HD-related sectors

and regionsOther

WBGUnits

Human Development

Network: HDNCE, HDNSP

Client countries

Universities and research centers

International IE networks

Other international organizations

SIEF

(i) Prospective, rigorous impact evaluations of programs in 11 eligible Human Development and Sustainable Development sectors and 72 eligible developing countries across all regions;

(ii) Intensive training programs for government counterparts, Bank staff, and staff of partner development agencies in impact evaluation methods; and

(iii) Publication and dissemination of evaluation results through articles, meta-studies, and web-based materials.

94% of the answers to the survey indicated that the management model was efficient or highly efficient.10 What has been particularly appreciated by TTLs was management’s flexibility in allowing for adaptations to changes in the context.

The leverage of SIEF resources also contributed to SIEF’s efficiency, as it allowed SIEF to contribute to the achievement of greater results with the same resources 11. An important feature of SIEF is that task team leaders leveraged other sources of funds to achieve results. In some cases funding commitments of local counterparts (e.g., in the case of a CCT in Indonesia, in-country resources provided USD$ 2.8 million of which SIEF contributed USD$ 300,000); but also other funding sources were leveraged, such as the International Initiative for Impact Evaluation, 3ie, in the case of the IE of Malaria, in Kenya.

Cluster leaders contribution

– To what extent have cluster leaders contributed to ensure the successful implementation of the SIEF program, and to promoting impact evaluation communities of practice in the program key thematic areas?

The extent of the contribution of cluster leaders was high in some cluster teams, where the initial conditions were favorable and in which cluster leaders played a more supportive role.

Concerning the extent to which cluster leaders contributed to ensure the successful implementation of the SIEF program, and to promoting impact evaluation communities of practice in the program key thematic areas, the survey sheds some light (though the number of answers was too small, given that some of the IE were not under a specific cluster, e.g., they were under the “innovation fund”, and, in some other cases, the person that answered the survey was a cluster leader recusing from answering). However, based on the answers: to the survey and on the interviews to TTLs, it can be stated that task team leaders appreciated cluster leaders’ support of dissemination activities, particularly in some clusters such as Active Labor Markets/Youth Employment, Education and Early Childhood Development. This may have been due to the previous practices in the different thematic/cluster areas (in particular, their experience with impact evaluations) and on the extent to which cluster leaders were supportive.

The distribution of Impact Evaluation Studies (IEs) by themes/clusters/modality is presented in the following table

10 A six-point scale was used in the survey. The answers reported in this paragraph (and in the following ones in this section) are those placed at the two upper levels of the scale.11 Following the OECD glossary definition of “efficiency”, i.e., “A measure of how economicallyresources/inputs (funds, expertise, time,etc.) are converted to results”, http://www.oecd.org/dataoecd/29/21/2754804.pdf.

15

Table 1 Number of Impact Evaluation Studies (IEs) funded by SIEF

Types of Impact Evaluation Studies Nº1.1. Quick Win IE Studies 5 *1.2. Innovation Fund IE Studies

1.2.1. Human Development 4 1.2.2. Sustainable Development 7 1.3. Cluster IE Studies 1.3.1 - Health Contracting / Pay For Performance Reforms 6 1.3.2 - Conditional Cash Transfers 5 1.3.3 - Malaria Control 5 1.3.4 - Active Labor Market Programs / Youth Employment 8 1.3.5 - Basic Education Accountability 5 1.3.6 - HIV/AIDS Prevention 3 1.3.7 - Early Childhood Development 4 TOTAL NUMBER OF IEs 52* to avoid double counting, 5 “Quick Win studies” that received support from the cluster funds were not included in the total number of Quick Win studies. For more details on the 52 studies, see Annex VII, Table 9

Governance of SIEF

– To what extent the governance of the fund at trustee level contributed to efficiency and effectiveness of the SIEF? To what extent was it useful for building an effective partnership between the Bank and donors? Did the Steering Committee enable the SIEF management for an effective delivery of outputs and results?

The governance of the fund contributed to SIEF’s efficiency and effectiveness, as well as to buiidng an effective partnership between the Bank and donors, whereas the SC provided an enabling environment for SIEF management (although it did not met after 2009).

With respect to the governance of the SIEF, the arrangements made were the following: A Trust Fund Steering Committee would meet annually to confirm the overall allocation of resources to the proposed work program, review ongoing and completed activities, and provide strategic guidance to the program management team on the priority topics for future evaluations. The Committee would have 7 members and would be co-chaired by the HDN Vice President and the Secretary of State for Spanish International Cooperation, including two additional Spanish representatives and 3 additional senior Bank members, named by the HDN Vice President: the HD Chief Economist; a DEC representative (either the DIME adviser or the DECRG manager for HD policy); and a regional HD director. Furthermore, a Technical Committee of 7 highly experienced Bank staff with widely recognized expertise in HD and SD research and impact evaluation would be appointed by the HDN VP to support the Program Management team in the vetting and evaluation of proposals for impact evaluation. It was expected that the Technical Committee would be chaired by the HD Chief Economist.Some members of the Technical Committee would also serve on the Steering Committee.

In practice, only one of the Steering Committees (SC) was actually co-chaired by the HDN Vice President and the Secretary of State for Spanish International Cooperation, who delegated the chairmanship of the other meetings to the Director General and to HDN’s Chief Economist.

16

There were six Steering Committee meetings, the last one being held on April 25, 2012. In the next to last SCM, at the end of 2009, it was approved the extension of SIEF until June 2012 and the creation of an Early Childhood Development cluster. Since then, and until the April 2012 SCM, communications between GOS and SIEF management were informal, with two important exceptions:

i. on November 2010 HD’s VP met with the Secretary of State for International Cooperation, confirming Spain’s interest in continuing supporting SIEF though at a lower budgetary level, and it was agreed that a final conference would be held in Spain on SIEF’s results, and

ii. on March 2011, as a follow-up to the November meeting, HD’s Chief Economist met with the Director General of Planning and Development Policies Evaluation of MAEC, and with representatives of the MEH. One of the results of this meeting was the decision to include an acknowledgement of SIEF funding on all products on whose elaboration SIEF funds were used (see below the reference made to this in the case of the E2P Notes), as well as an agreement to send to GOS consolidated monthly reports on SIEF outputs and their dissemination. The following table shows the information about SIEF’s outputs that included in these monthly reports.

Table 2 Information about SIEF’s outputs included in SIEF’s monthly reports

OUTPUTS NUMBER EVENTS 23 PUBLICATIONS* 25 NOTES** 6

Source: SIEF’s monthly reports, March 2011/March 2012

* Including IE reports, papers and dissemination notes.** Notes in prestigious publications (such as The Lancet, The Economist and the Journal of Economic Literature) on outputs produced with SIEF funds (such as evaluation reports and books). These notes are proxies of influence and of acknowledgements of the outputs’ quality.

The interviews held in Madrid, particularly those with the MEH, revealed a communication gap with SIEF, with a strong demand from GOS representatives for information on products and on their cost. However, contrary to the overall perception of GOS officials concerning the contents of the monthly reports as reports on events, table 1 shows that these reports have included a higher number of publications items than of events12. The turnover of persons involved at the leadership and operational levels of GOS’s international cooperation, and the shortage of staff in the evaluation unit of the MAEC (referred to in the 2011 DAC’s peer review of Spain), which had to play a crucial role on the donors’ side, limited the active participation of Spain in SIEF.

12 In this context it is worthwhile to quote the observation made in the DAC 2011 Peer Review of Spain that there is a “need to ensure that /Spain’s/ development agency has sufficient capacity to direct and digest a growing volume of evaluations of projects and programmes”. (p.62). http://www.oecd.org/dataoecd/23/45/49356882.pdf.

17

As pointed out by J.A.Alonso, “en (…) aspectos relacionados con la evolución reciente de la cooperación española se ha evidenciado el desequilibrio existente entre el dinamismo de los compromisos suscritos y las limitadas capacidades humanas, técnicas e institucionales”13

The amounts disbursed by the different types of SIEF activities are presented in table 3. By July 2007 SIEF made its first call for proposals from existing HDN Impact Evaluation clusters, trying to identify “Quick Win” IE studies, “in order to get the SIEF off to a very visible, quick start, and to help ensure that the Spanish resources produce some tangible results as early as possible” (as framed by SIEF’s first program manager). The proposals were called “quick winss” as they were expected to capitalize on work on Human Development issues that had already been going, searching for proposals of high quality, prospective impact evaluations that had immediate funding needs. The chosen proposals were submitted to SIEF’s Steering Committee.

Although the “quick wins” allowed SIEF to contribute in the short-term to high quality evaluations, SIEF did not obtained visibility from this first stage. On the other hand, these allocations of funds to “Quick Wins” (approximately 6% of SIEF’s resources, as shown in the preceding table) provided SIEF with an opportunity to leverage other sources of funds. During the following years, as shown below in the table 3 , the bulk of SIEF’s resources were allocated to the new clusters of IEs.

Considering activities by basic categories, table 3 shows that 69% of the resources were disbursed for impact evaluation research, and that program management absorbed less than 5% of the resources.

Table 3 Amounts allocated and disbursed by type of activity

Activities Allocate

d

% of Total

Allocated

Disbursed

% of Total

Disbursed

1. Impact Evaluation Studies (IEs) 9.817.968 66% 9.817.9

68 69%

1.1. Quick Win IE Studies 879.851 6% 879.851 6%

1.2. Innovation Fund IE Studies 2.009.223 13% 2.009.2

23 14%

1.3. Cluster IE Studies 6.928.894 46% 6.928.8

94 49% 1.3.1 - Health Contracting / Pay For Performance Reforms

1.250.539 8% 1.250.53

9 9%

1.3.2 - Conditional Cash Transfers 895.086 6% 895.086 6%

1.3.3 - Malaria Control 762.790 5% 762.790 5%

1.3.4 - Active Labor Market / Youth Employment

1.276.350 9% 1.276.35

0 9%

1.3.5 - Basic Education Accountability

1.022.601 7% 1.022.60

1 7%

1.3.6 - HIV/AIDS Prevention 981.561 7% 981.561 7%

1.3.7 - Early Childhood Development 739.966 5% 739.966 5%

13 Alonso, J.A. (2012) “España y el sistema multilateral: luces y sombras de un proceso de cambio” en de la Iglesia-Caruncho (coordinador) La eficacia de la ayuda y la cooperación española Madrid: Editorial Complutense, p. 179

18

2. Capacity Building 1.700.000 11% 1.681.2

15 12%

3. Results Dissemination 1.533.000 10% 1.156.9

06 8% 3.1. Program Outreach & Results

Dissemination 600.000 4% 437.519 3%

3.2. Cluster Results Dissemination 933.000 6% 719.387 5%

4. Other Activities 1.859.341 12% 1.504.4

42 11%

4.1. Program Management 781.923 5% 597.201 4%

4.2. Cluster Coordination 510.000 3% 339.822 2%

4.3. Advisory Services 567.418 4% 567.418 4%

5. Total (5=1+2+3+4) 14.910.309 100% 14.160.

530 100%

Transparency in the allocation of funds to IE teams

– To what extent were SIEF resources allocated to impact evaluation teams in a transparent way?

The allocation of funds to IE teams was transparent.

On the extent to which SIEF resources were allocated to impact evaluation teams in a transparent way, 97% of the TTL surveyed answered “highly transparent” or “transparent”. The call for proposals was a transparent process, with explicit “rules of the game”.

Quality control mechanisms and processes

– What type of quality control mechanisms/ processes were established for ensuring the quality of the SIEF impact evaluation portfolio?

A peer review process of proposals with internal and external revision.

A peer review process of proposals submitted for SIEF funding was put in place as a quality control mechanism. This process combined rigor (with an internal and external revision) and flexibility (allowing for a waiver of the process when a rigorous review had already been undertaken by other donor or academic institution or journal); most answers to the TTL survey indicated that it was transparent. The following quote from a TTL compares the SIEF process with that of other funds:

“I have received research support from numerous other funding sources over the years (the World Bank Research Support Budget. the BNPP program, the SFLAC program, DfID EROs, the EFA FTI program preparation fund) and I believe that SIEF sets the bar for having developed a competitive review process that is at the same time substantive and analytically rigorous, while administratively “light”, flexible and efficient. It is not easy to achieve this, but I think SIEF has done so”.

Both, the external and the internal peer reviewers, review the draft concept note and, when needed requests further information and/or suggests changes. The cluster leader communicates

19

reviewers’ comments to TTLs and clears the revised concept note. SIEF Management clears peer review process and authorizes (or not) TTL to submit the proposal for funding

The calls for proposals required a rigorous specification of the counterfactual, and randomized designs have been the most frequently used technique in the IEs funded by SIEF (74% of the answers to the TTLs survey). The rest of SIEF funded IEs used regression discontinuity, propensity score matching and other quasi-experimental or observational approaches. For example in the case of the IE of household connection to the sewerage system in Uruguay and the case of the “Contigo Vamos por Mas” IE in Guanajato, Mexico, which used a quasi-experimental design with a sample of treatment and control localities, as well as propensity score matching and difference-in-differences methods combined with semi-structured interviews to relevant stakeholders.

SIEF monitoring instruments

– To what extent and in what ways did SIEF monitoring instruments (Annual and Mid-Year reports) were used and useful?

SIEF monitoring instruments were usefully used.

The spreadsheet facilitated an active management of SIEF resources, leading to timely transfer of funds to SIEF activities. In addition, the spreadsheet was used as a basic source of information for the Annual and Mid-Year reports. These instruments were useful for decision making concerning funds allocation and reallocations, which facilitated the implementation of the program.

4. Results

The TOR questions concerning SIEF’s results are the following:

– To what extent has the SIEF program achieved its aimed strategic goals?– To what extent have SIEF capacity building activities contributed to a better understanding

and use of impact evaluations?– To what extent SIEF research and dissemination activities have contributed to a better

access to knowledge on the impact of Human Development programs? – When intended results were not achieved as planned, what were the main reasons? – Were there any unintended results?– Are there any lessons learnt on how the SIEF could have achieved or on how similar

programs could achieve better results? – Should the SIEF program be continued / up-scaled, what potential results could be obtained?

What changes could be recommended for it to become more effective?

Rather than addressing each of the questions, this chapter is organized around the SIEF resuts-based matrix, with four sections corresponding to the four SIEF strategic goals. The different goals, and their respective expected results/outcomes will be rated using a six-point scale with the following interpretation (the numbers with the specific ratings are shown below at the right of the SIEF’s MfDR matrix goals and the corresponding results)14:

14 See “ ratings for Outcome” in http://ieg.worldbankgroup.org/content/dam/ieg/ratings/HarmonizeEvalCriteria.pdf

20

Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory.

Capacity building

– Goal 1: To build capacities for understanding and using Impact Evaluation (IE) techniques.

Highly Satisfactory 15 .

Result 1.1 (R11): Evaluation experts and researchers from the WB and its client countries, and from other private and public organizations, use their skills gained at SIEF workshops and the SIEF training materials to design and carry out IE studies.

Highly Satisfactory.

Rationale/evidence: the targets were 1893 trained persons and 12 regional workshops, whereas the actual figures reached were 2403 persons and 17 workshops. The Tracer study provides evidence on the use of skills gained at SIEF workshops; see below Box A.

R12 Managers from the WB and its client countries use the managerial skills gained at SIEF workshops and the SIEF training materials to support the design and implementation of IE studies on their programs.

Highly Satisfactory.

Rationale/evidence: Tracer study and the following paragraphs in this same subsection corresponding to R12:

The objective of the “Turning Promises to Evidence” workshops, the key means by which the SIEF promoted training and capacity building in impact evaluation, has been to provide participants with an overview of impact evaluation. These workshops were one-week regional courses focusing on the design, implementation and management of impact evaluations, tailored to project teams of government counterparts in client countries, Bank staff, and staff of partner development agencies.

The main goal of the workshops has been to increase the effectiveness of human development programs by building impact evaluation and results-based management capacities in client countries and partner agencies. This included cutting edge techniques of impact evaluation, an understanding of its importance for policymaking and program design, a clear understanding of how to construct a results chain, and practical skills in the application of impact evaluation techniques to their own projects. The workshops consisted of a highly interactive mix of small and large group sessions involving the “hands on” application of what is learned throughout the week.

An innovation introduced by these workshops has been to offer participants the possibility to choose between two parallel tracks: one targeted to policy makers, one targeted to researchers. Both tracks were designed to ensure consistency and the diffusion of key messages to both groups. The policy maker track sessions was instrumental in developing

15 The rating for each goal is the mode of the ratings into which the goal is decomposed (the highest frequency rating of the set of ratings associated with the corresponding goal)

21

capacities to manage impact evaluation (and to nurture the demand for them), whereas the technical track sessions developed capacities to conduct impact evaluations.

During the workshops, participating policymakers, technical staff and World Bank task team leaders worked together in group sessions as project teams with expert moderators on the “hands-on” design of a high quality impact evaluations and implementation plans for their projects. On the final day of the workshop, the teams presented their proposed evaluation plans to the whole group and received guidance from each other and the course moderators.

Since January 2008 SIEF supported 17 workshops of this kind with $ 1,500,000. The workshops were conducted in all regions of the developing world, and they also had financial support from the United Kingdom’s Department for International Development (DfID), which provided $475,000, the Bank-Netherlands Partnership Program (BNPP) and other World Bank units and funds (for example, the Rio de Janeiro workshop was also supported by DIME and by the World Bank’s Brazil country office)

The number of participants from client countries and partner agencies trained in impact evaluation techniques in these workshops has been 2406, corresponding to 251 project teams. On average, in each workshop there were participants of 9 countries. Detailed information about these workshops is presented in table 4.

Table 4: Impact evaluation regional workshops

Location Date Countries Attending Participants Project

Teams Quality* Usefulness**

Cairo, Egypt January 13-17, 2008 12 160 17 4,07 4,14

Managua, Nicaragua March 3-7, 2008 11 104 15 4,83 4,84

Madrid, Spain June 23-27, 2008 1 196 9 3,94 3,51

Manila, Philippines December 1-5, 2008 6 137 16 - -

Lima, Peru January 26-30, 2009 9 182 18 4,43 4,54

Amman, Jordan March 8-12, 2009 9 206 17 3,93 3,88

Beijing, China July 20-24, 2009 1 212 12 4,38 4,10

Sarajevo, Bosnia Sep. 21-25, 2009 17 115 12 4,37 4,35

Cape Town, South Africa December 7-11, 2009 10 108 11 4,29 4,16

Kathmandu, Nepal February 22-26, 2010 7 118 15 4,33 4,22

Rio de Janeiro, Brazil April 26-30, 2010 3 224 22 4,37 4,38

Accra, Ghana May 24-28, 2010 17 143 19 4,37 4,38

Tunis, Tunisia October 18-22, 2010 10 80 10 4,63 4,50

Seoul, Rep. of Korea December 6-10, 2010 10 134 12 4,37 4,42

Seoul, Rep. of Korea Aug. 29 - Sep. 2, 2011 6 60 13 4,71 4,49

Dacca, Bangladesh October 9-13, 2011 6 127 16 3,90 4,06

22

Santiago de Chile January 16-27, 2012 9 100 17 - -

Total 17 Regional workshops 9.11 2.406 251 4,33 4,26

Responses of participants at the end of the workshops to the following questions –on average, as rated at the end of each workshop by participants on a scale of 1(minimum) to 5 (maximum):* How would you rate the overall quality of the activity? ** How would you rate the overall usefulness of the activity?

A tracer study report of these workshops was prepared by the HDN Chief Economist Office, and its main findings are presented in the following box

Box A. Findings of the Tracer Study Report of the SIEF-Sponsored Impact Evaluation Training Program

The results of the tracer study suggest that the “Turning Promises to Evidence” training program has been effective in building impact evaluation capacities in client countries, and in promoting communities of practice among impact evaluation and human development practitioners and experts. More specifically, the results of the survey suggest that:

- Former workshop participants are applying the skills they learned by designing, implementing and using impact evaluations for their work. The majority of respondents state that, since the workshop, they have designed impact evaluations (56%), helped manage impact evaluation studies (71%), and built results chains (74%). The vast majority of participants (93%) report to have read impact evaluations.

- Since the workshop, a significant number of participants have continued involved with the impact evaluation study that they contributed to designing during the group sessions;

- The workshops have contributed to consolidating communities of practice in client countries on impact evaluation in key human development thematic areas;

- Participants strongly support applying impact evaluation techniques and results-based management in their organizations.

The overall conclusion of this study is that the “Turning Promises to Evidence” workshops have proved to be useful in building impact evaluation capacities of government officials, development practitioners and evaluation experts in client countries. The results of the study are consistent with the high quality and useful ratings given by participants at the end of each workshop (4.3 on average in a scale 1 to 5, as shown in Table 4).

Conclusion concerning IE capacity building:

The approach to training introduced an innovative two-tracks format, which matched well the different demands from policy makers, evaluation managers and evaluation specialists. This approach strengthened both the demand and the supply for IE at country level and also at the World Bank Human Development Network, where it played a catalytic role. On the other hand, SIEF has not strengthened evaluation capacities of the Spanish donor (which was one of the objectives included in the agreement establishing the SIEF), except for the IE workshop held in Madrid and the technical advice provided to project teams funded by Spanish cooperation which were interested in designing IE.The “Impact Evaluation in Practice” book, produced by SIEF, and translated into Spanish and French, is a very valuable resource that can be used in the development of impact evaluation capacity in developing countries, the World Bank and in

23

Spain. SIEF’s approach has focused on developing impact evaluation capacities of persons rather than on institutional capacities for IE.

Recommendations on IE capacity building:

A new IE program could focus not only on individual evaluation capacities for impact evaluation but also on IE institutional capacities, e.g., cooperating in setting-up institutional mechanisms for managing IE (as in the cases of Mexico and Chile) in organizations that are not responsible for implementing the programs which are evaluated, thus strengthening the credibility of the evidence generated by IEs.

Concerning IE training, in a future program it would be worthwhile to take into account the following recommendations from the SIEF Tracer Study Report

Box B. Recommendations on training for impact evaluation

i) Mobilize sufficient resources in order to match the increasing demand of impact evaluation training and advisory services from client countries and partner agencies that is expected for the next years. Increase impact evaluation capacities of Regional units and promote partnerships with local universities and research centres to allow their playing an increasing role in meeting this demand.

ii) Monitor progress of the impact evaluation studies that are designed at the workshops to detect training needs and potential gains from advisory services activities that may help reduce the number of impact evaluation studies that do not continue after the workshop. Engage senior management and policy makers in results dissemination events to increase management support and mobilization of resources towards impact evaluations in client countries and partner agencies.

iii) Design tools to provide training to workshop participants on how to convince their managers to invest resources to undertake impact evaluations.

iv) Engage former workshop participants in training and research activities to consolidate the expansion of the impact evaluation community of practice. Encourage the participation of former trainees in regional workshops as faculty or moderators to facilitate their transition from active members to members “at the heart” of the community of practice.

v) Tailor the timing and contents of the workshops to the needs of client countries –for instance, by delivering “second generation workshops” focusing on specific technical or policy issues, or expanding the Stata component when demand for such training is detected.

vi) Carry out follow up training and advisory service activities to consolidate and improve the learning outcomes of the workshops. Mobilize resources to deliver training activities on a more regular basis. vii) Improve the monitoring and evaluation of future impact evaluation workshops by adding to the online registration form some questions to capture the applicant’s age, type of organization, and languages spoken; as well as their secondary email addresses and telephones; at the beginning of the training, collect information on the skills level and involvement with impact evaluation work of participants, so that learning outcomes can be measured in the future; translate the survey instrument to other languages, contingent to the languages spoken by participants and to the availability of resources. Whenever possible, customize survey requests and reminders to the mother tongue of each participant.

24

Support of IEs

– Goal 2. To support the application of impact evaluation techniques to selected development programs to improving Human Development (HD) outcomes

Highly Satisfactory

R21 Experts and researchers from the WB and its client countries are increasingly applying IE methods to measure the impact of selected programs having an effect on HD outcomes. Managers from the WB and its client countries are increasingly participating in the design and conduction of these studies.

Highly Satisfactory

Rationale/evidence: 55 IE studies funded by SIEF (see above, table 2), whereas the target was 5016; and evidence from TTLs answers to survey (see also the examples in the last section of this chapter). Tracer study (see above, Box A)

R22 Directors of selected HD-related areas are increasingly giving support to evaluating the impact of their programs.

Highly Satisfactory

Rationale/evidence: table 3 and interviews.

R23 Policy makers and Directors from donor countries are considering applying IE methods for measuring the impact of some of their HD programs.

Satisfactory

Rationale/evidence: Answers to TTLs survey and interviews.

Conclusion on support to IEs SIEF provided valuable support to the application of IE to selected HD development programs, exceeding its targets.

Recommendation:: Given the yet significant IE gap it would be worthwhile to scale-up the work done by SIEF in order to broaden its reach, drawing on the expanded pool of available IE expertise, contributed by SIEF, as well as on the resource materials developed by SIEF for IE capacity building.

16 In fact, Annex I of the Trust Fund Administration Agreement indicated each year the Trust Fund will support at least 5 rigorous impact evaluations (with a target of 15 evaluations in the period of 3 years)

25

Dissemination of SIEF findings and promoting access to knowledge

– Goal 3. To disseminate SIEF findings, and to promote access to knowledge on the impact of HD programs

Satisfactory

R31. Evaluation experts and researchers on selected HD-related areas use the evidence provided by SIEF publications for their research work. They share knowledge through the SIEF communities of practice to design and/or to make progress in their research, and use SIEF-funded resources (publications, instruments, website ) to design and/or make progress in their research.

Satisfactory

Rationale/evidence: Examples in the last paragraphs of this section. There is additional evidence in the answers by TTLs to the surveys and in the interviews. The last section of this chapter provides some examples (such as the cases of Mexico, Burkina Faso and the Nigeria Malaria Control Booster Project).

R32. Managers from the WB and its client countries take into consideration the evidence shown by SIEF publications to explore possible changes in types of management process that could increase the impact of their programs. They share experiences through the SIEF communities of practice, and use some of the SIEF-funded resources to increase the impact of their programs.

Satisfactory

Rationale/evidence: The last section of this chapter provides some examples (cases of Brazil, Malawi and Tanzania HIV AIDS). There is additional evidence in the answers by TTLs to the surveys and in the interviews.

R33. Directors from WB offices and from client countries are using the evidence provided by the SIEF in order to change the focus of their programs towards actions having a higher developmental impact.

Satisfactory

Rationale/evidence Answers by TTLs to the surveys and in the interviews.

R34. Policy makers are aware of the existence of a body of evidence for better decision making in areas having an effect on HD based on the impact of programs.

Moderately Satisfactory

Rationale/evidence: The E2P are useful instruments to make policy makers aware of the existence of relevant evidence, but at the time of this assessment report, they are not showing in SIEF’s website and there is no evidence that they are actually used by policy makers. On the other hand, the following paragraphs of this sub-section make reference to SIEF products that have been used (and could be further utilized) for disseminating IE results.

Policy Notes, “From Evidence to Policy (E2P”): The SIEF has supported the elaboration of this series of notes, disseminating the results of IE in the Human

26

Development area. The first E2P note was published in September 2010, and 11 E2P Notes were produced by February 2012 (Annex iv includes all the titles). The last 7 (i.e., those published since March 2011) make an explicit acknowledgement to SIEF’s support. This is a valuable dissemination instrument, which facilitates the process of translating research and impact evaluation findings into inputs for policy makers. They are presented in a reader friendly way. A possible limitation of some of the Notes is that though their messages frequently correspond to single case studies, they are framed in ways that readers may not realize that their external validity is limited.

Books: Impact evaluation results have been published in the book “Conditional Cash Transfers: Reducing Present and Future Poverty” (SIEF supported its translation to Spanish), “Making Schools Work: New Evidence on Accountability Reforms” and “No Small Matter: The Impact of Poverty, Shocks, and Human Capital Investments in Early Childhood Development”.

Furthermore, an impact evaluation training handbook and toolkit Impact Evaluation in Practice” has been produced, and is available in English, Spanish and French. This book has been reviewed very positively in the American Economic Association’s Journal of Economic Literature (September 2011). The handbook can be downloaded without charge, and accessed through Google or any other search engine, or directly at www.worldbank.org/ieinpractice A complete training toolkit, available at the same website, includes in addition to the the handbook in its English, Spanish and French version, a set of PowerPoint presentations and videos with the core content. This material can be used by people who could not attend to the regional workshops. This book, as indicated by the Director of the Centre for the Evaluation of Development Policies of the UK Institute for Fiscal Studies, provides “an accessible, comprehensive and clear guide to impact evaluation”.

Other publications: In addition to the policy notes and books, impact evaluation teams supported by the SIEF have produced 25 research papers and articles.

Presentations: SIEF supported over 47 presentations in international conferences and other discussion fora as a means for knowledge sharing and dissemination.

Media coverage: In addition, as indicated in the footnote to table 1, some prestigious and influential publications, such as The Lancet, The Economist, the Journal of Economic Literature, and newspapers such as The Financial Times and The Wall Street Journal made reference to outputs produced with SIEF funds.

27

Table 5 Type of products delivered by Impact Evaluations funded by SIEF

Types of Products Number

Concept notes 47

Baseline surveys 10

IE reports (unpublished) 12

IE articles or papers (publications) 25

Books (or chapters in books) 4

More detailed information on the products delivered by IE funded by SIEF can be found in Annex iii. It is worthwhile to mention that in addition to the products included in table 5, impact evaluations funded by SIEF have generated data useful not only for evaluation but also for the design of new interventions. In addition, the datasets are submitted to the micro-data catalogue in order to facilitate access.

Website : the website is outdated (e.g., the last workshops listed are those that were held in 2010, whereas, as shown in table 4, there were 3 additional workshops since then), it does not include detailed information on the actual IE that were implemented (practically none of the products included in table 5 show up in the website) As it is, it only conveys some general information about SIEF but it is of very limited value as a portal for knowledge sharing on IE funded by SIEF. Though it is worthwhile that the website contents has been translated into Spanish, the website in Spanish presents an additional problem of a blurred layout in its first page. Finally, its Uniform Resource Locator (URL) is rather cumbersome:http://web.worldbank.org/WBSITE/EXTERNAL/EXTABOUTUS/ORGANIZATION/EXTHDNETWORK/EXTHDOFFICE/0,,contentMDK:23150708~menuPK:6508083~pagePK:64168445~piPK:64168309~theSitePK:5485727,00.html

Conclusion on dissemination and on promoting access to knowledge: SIEF produced valuable material that has been disseminated mainly by several different non-web means The SIEF website is neither updated nor comprehensive and it has not played a useful role in the dissemination of knowledge generated by SIEF. There is scope for improvement in this area.

Recommendations

The development and maintenance of an updated, comprehensive and user-friendly website with information about the program of impact evaluations and results achieved should be one of the key means to share knowledge generated by the program. In order to ensure that the website fulfils its role it would be worthwhile to conduct periodic revisions of the website and to budget adequate resources for its design, maintenance and revision. Furthermore, it would be convenient to explore the possibility of developing an independent website (e.g., www.sief.org), not subject to the constraints that should be followed when it is a World Bank (www.worldbank.org) website, taking into account that it is a partnership program funded by donor(s). The program’s

28

website could have an intranet area for internal monitoring of the program and to facilitate information sharing about processes to be followed and intermediate products.

A proactive approach to dissemination (rather than the more frequent elaboration of dissemination plans after the evaluations were conducted) could include during the implementation of the evaluation a set of activities (such as videos and photographs) that with a very low marginal cost could enhance significantly the quality of dissemination and, as a by-product, it would make available audiovisual material for training purposes.

Use of SIEF as a platform for promoting MfDR

– Goal 4. To use the SIEF as a platform for promoting Management for Development Results (MfDR)

Satisfactory

R41a. Evaluation experts and researchers from WB client countries lead the design and conduction of some IE studies.

Satisfactory

Rationale/evidence: TTLs surveys provide evidence, complemented by information included in IE reports.

R41b. Managers from WB client countries lead the management of some IE studies, and identify programs with potential for being evaluated in terms of impact.

Moderately Satisfactory

Rationale/evidence: The modus operandi of TF at the WB limit the extent to which managers from WB clients could lead the management of IE studies funded by TFs. However, there are cases in which they have identified programs with potential for IE, as revealed in the interviews.

R41c. Directors and policy makers from WB client countries suggest programs to be evaluated by the SIEF, according to their development strategic priorities; and consider the possibility of integrating impact evaluations in their organizations on a regular basis.

Moderately Satisfactory

Rationale/evidence: same as R41b.

R42a. WB evaluation experts and researchers share resources to elaborate IE studies and to disseminate their results. IE experts from the WB and its client countries, as well as other international organizations and countries share their experiences and develop informal harmonization schemes.

Satisfactory

Rationale/evidence: the workshops provided a forum to share resources to elaborate IE studies, and cluster conferences (or conferences in which cluster leaders participated) were used among other things to disseminate IE results and share

29

experiences. There are also some cases, such as Paraguay’s IE on water and sanitation and the IE on the Youth Development Program in the Dominican Repubic, where IE teams supported by SIEF coordinated with teams from other international organizations

R42b. WB managers develop coordination mechanisms to avoid the duplication of efforts among the IE studies they manage. Managers from the World Bank and its client countries, along with managers from other international organizations and countries, consider the possibility to design joint evaluations.

Satisfactory

Rationale/evidence: same as R42a and DIME progress report

R42c. WB directors and policy makers keep their support to internal coordination initiatives on IE and/or results-based management. Directors and policy makers from the WB and its client countries, as well as from other international organizations and countries, promote harmonization schemes on IE and/or results-based management.

Satisfactory

Rationale/evidence: Participation in joint events with other organizations focused on impact evaluation like the 3ie Conference in Cuernavaca and at NONIE meetings. DIME progress report.

R43a. Evaluation experts and researchers from WB client countries contribute, through the SIEF communities of practice, to the design and implementation of results-based management plans.

Satisfactory

Rationale/evidence: Evaluation experts and researchers from WB client countries participated in SIEF impact evaluation training (see table 4). Building a results chain was one of the key components of SIEF impact evaluation training. The SIEF workshop tracer study shows that 74% of respondents stated that they built results chains for their projects since the workshop, and 97% agree that management for development results can improve the degree of attainment of their organization goals

R43b. Managers from international organizations and countries, and especially from the WB and its client countries contribute, through the SIEF communities of practice, to the design and implementation of results-based management plans.

Satisfactory

Rationale/evidence: Managers from interntional organization and from countries participated in SIEF impact evaluation training (see table 4).Building a results chain was one of the key components of SIEF impact evaluation training. The SIEF workshop tracer study shows that 74% of respondents stated that they built results chains for their projects since the workshop, and 97% agree that management for development results can improve the degree of attainment of their organization goals;

R43c. Directors and policy makers from international organizations and developing countries and especially from the WB and its client countries consider the possibility to focus their areas of responsibility towards the achievement of results. WB and

30

donor country members of the SIEF Steering Committee analyze the information shown at the SIEF monitoring and evaluation reports to consider the possibility of drafting a SIEF extension proposal based on the results-based management framework.

Satisfactory

Rationale/evidence: The recently launched new multi-donor SIEF provides an appropriate framework for an extension proposal of the original SIEF based on a results-based management framework.

Conclusion: Through different initiatives SIEF has been used to promote MfDR

Recommendation: In order to enhance the use of SIEF as a platform for promoting MfDR it would be worthwhile to leverage the regional communities of practice on MfDR, http://www.mfdr.org/CoP/index.html, in addition to the sector and IE communities of practice.

Examples of the use of SIEF supported IE and of their policy influence

Even though a full assessment of policy influence would require case studies, it is useful to provide some examples of use of IE supported by SIEF and, particularly, of their policy influence, based mainly on the survey to TTLs, with the caveat that there may have been other factors influencing policy.

Paying for performance in health:

Thus, in the case of the IE of Plan Nacer, in Argentina, supported by SIEF, the preliminary results of the impact evaluation have helped to inform the next phase of the program and encouraged the expansion of the program to new population and types of health services. Some innovations introduced by the IE of this intervention, as shown in box C, may be useful in other countries.

Box C. Some contributions of Argentina’s Plan Nacer’s IE supported by SIEF

Substantial involvement of policy makersProvincial (sub national) health ministers participated in the process of discussing an “evaluation agenda” (including evaluation questions) and in the presentation of evaluation results

Baseline surveys as an important addition to the stock of reliable statistics in the health sectorRepresentative samples at the provincial level exploited new sources of data (establishments, providers), that can also be used (and are being used) as an input for decision making during implementation. Plan Nacer`s evaluation activities have been increasing the quality and quantity

31

of health data at the national and provincial level17, which can lead to the design and implementation of more effective interventions in the health sector.

Ensuring reliable information through data quality auditsOutcome based payment systems may create an incentive to misreport results, thus “gaming” the system. The introduction of external data quality audits Plan Nacer’s impact evaluation can deter misreporting and alert if and when it becomes a problem.

Development of evaluation capacities at the provincial (sub-national) levelInvolving provincial counterparts in the evaluation team provided opportunities for “learning by doing” at the provincial level, in addition to nurturing the demand for evaluation of high-level policy makers.Source: Interviews with Plan Nacer staff involved in its impact evaluation.

Conditional cash transfers

In Mexico, the interim results of the IE helped develop a strategy of targeted support to more disadvantaged schools in Mexico The results justified a policy change to support schools in a more targeted way.

Burkina Faso: The IE results of a pilot project of conditional and unconditional cash transfers are used for the definition of safety nets programs in Burkina Faso and other countries in the region.

Malaria control

Box D The case of an IE of a Malaria Control Booster Project

Under TF095606, SIEF allocated a total of USD 183,936.06 to the impact evaluation (IE) of the Nigeria Malaria Control Booster Project (MCBP). This is part of a series of impact evaluations coordinated by the Malaria Impact Evaluation Initiative (MIEP)/Development Impact Evaluation Initiative (DIME) with the overall objective of helping malaria-burdened countries develop evidence-based programs to improve health and socioeconomic outcomes, to build their capacity in evidence-based policy design and program management, and to contribute to the global body of knowledge on effective malaria control interventions and policies.

SIEF resources were fundamental in furthering progress on this important evaluation, and have been allocated in a ways fully consistent with the objective of the impact evaluation. SIEF resources contributed to:

i)a census of all public primary health facilities, PMVs, community laboratories (important for understanding the current availability of diagnostic services at the community level),

17 This is an important aspect to take into account when assessing the costs and benefits of impact evaluations. Another example is the case of the IE on malaria, in Nigeria, where what began as an IE activity contributed to improving information on the availability of health service at state levels.

32

communities, and kindred groups in all seven project states (three of which have been participating in IE activities);

ii) preliminary data analysis to inform randomization;

iii)capacity-building for IE states;

iv)design modifications

v)leveraging approximately $1 million additional funds to support IE activities from the Japan Social Development Fund and project budget; and

vi) ongoing engagement at the federal level with the National Malaria Control Program (Federal Ministry of Health) and at the state level with the State Ministry of Health of each IE state, to maximize the likelihood that results are chaneled into policy formulation.

Box E Policy changes triggered by the Zambia Malaria IE

Results from both the evaluation of the public sector intervention and that of the private sector intervention have triggered policy changes. Acknowledging the impressive impact of the new public supply method, Zambia decided to scale up the direct-order system nationwide, which is expected to result in 16,000 fewer malaria-related deaths in children under five by 2015.

On the other hand, results from the private sector intervention demonstrated that there is a vital role to be played by private providers in reducing the malaria burden, but the widespread implementation of such an approach necessitates further regulatory reform. The Pharmaceutical Regulatory Authority (PRA) has recognized the limitations of the current regulatory framework, and an amendment to the Pharmaceutical Act is to be presented to the next session of Parliament as a ‘Pharmaceutical Bill 2011’ to register and certify drug stores.

Active labor market programs / youth employment

In the Dominican Repubic, where youth unemployment is a major issue, SIEF co-financed, in collaboration with IADB, an impact evaluation in 2008 that is going to allow to measure the impact of the program in labor markets outcomes, as well as in soft skills and risky behaviors, distinguishing the relevance of the technical training and the vocational training. Although the final data collection will be carried out in the second half of 2012, preliminary findings from longitudinal surveys and follow-up data collected indicates that the training has a positive impact on women's employability and earnings. The life skills training is at least as important as the technical skills training, and can be more cost-effective than the technical skills training. The impact evaluation led the government of the DR to redesign the program doubling the number of time devoted to life skills training and increasing the focus of apprenticeships in more competitive activities such as trading services, restaurants, and hotels. These improvements have been incorporated to the new design of the program approved for the IADB's $20 million loan extension in July 2011.

33

Uganda YOP Impact Evaluation: descriptive baseline analysis informed the dialogue with program management and government officials, particularly in regard to gender composition of program participants and some governance outcomes. The finding that 1/3 of group members were female prompted a debate around the appeal of different types of vocational training programs to mixed gender groups, and opened a discussion around possible alterations to operational rules concerning gender composition for new vocational skills development programs under preparation.

Education Accountability

In the state of Pernambuco, Brazil, the SIEF-funded IE research demonstrating the positive impact of the teacher bonus program has been a factor in persuading the Pernambuco secretariat to retain the program, and in inspiring several other Brazilian states and municipalities to launch similar teacher bonus programs. The SIEF-funded classroom observation work influenced the government of Pernambuco’s decision to continue carrying out annual observations through a panel of schools as a way of monitoring the ongoing impact of the bonus program on teachers’ effort and classroom effectiveness.

In the state of Andhra Pradesh, India, SIEF contributed to the testing of three methods for encouraging more effective teaching: giving teacher bonuses based on student performance, or giving schools money to buy more supplies or money to hire an extra teacher. The study included 500 rural primary schools run by the government. Paying teachers extra if their students did well on  turned out to be highly effective: Their students did better in assessment tests than the other students.

HIV/AIDS Prevention

Malawi and Tanzania: the experimental impact evaluation programs not only advanced knowledge on the role of conditions in conditional cash transfer programs, but also was informative to policymakers by advancing knowledge on key policy parameters for cash transfer programs, such as transfer size and the identity of the recipient in the household. It informed policymakers on the potential of cash transfer programs for adolescent girls in delaying pregnancy and marriage. The experiment also served as "proof of concept" that social protection programs can reduce the risk of HIV infection among young women in Sub-Saharan Africa. The governments in Malawi and Tanzania are designing secondary school conditional cash transfer/bursary schemes for school-age girls based partially on the findings from this study.

Other thematic areas

Tanzania: The baseline data collection funded by SIEF for the Tanzania’s Social Action Fund (TASAF), a community driven development project being implemented throughout Tanzania, was used not only for the evaluation but also to prepare a targeting paper that has influenced how the next phase of TASAF (TASAF III) is being designed between the World Bank, TASAF, and the government of Tanzania. Furthermore, preliminary impact findings are influencing the

34

design of the specific components of TASAF III and other related government programs for poverty reduction.

South Africa. The Department of Human Settlements has been building houses since 1994 for low-income communities. Up until 2008 the primary focus has been on measuring outputs (the number of houses completed) with a focus on full package subsidies (house, tenure, services, etc.) rather than incremental upgrading. With the introduction of the SIEF funded impact evaluations, performance indicators have moved from output to outcome and impact and these indicators are being built into the current M&E framework at the local level. Thus, the ongoing IE support has contributed to the improvement of performance measures to incorporate a more holistic approach to development. The Government is also considering increasing its focus on incremental upgrading (rather than block subsidies) where the IE has offered some insight into how to make this work. The IE results have served to provide evidence on effective methodologies for upgrading of human settlements and on the consequences of different models of upgrading. Results have opened debate around financial sustainability of projects and the economic consequences on targeted communities. As such, a strengthening of financial literacy components to the standard upgrading is being considered to safeguard communities against household financial mismanagement. Although policy changes have not yet been enforced, the results are now feeding into the policy discussions, with a Ministerial request to provide suggestions on how to operationalize some of the recommendations noted in the IE report.

5. Conclusions and Recommendations

The preceding chapter included a set of conclusions and recommendations after the presentation of each strategic goal and their ratings. This final chapter consists of more general conclusions and recommendations

Conclusions

The Spanish Impact Evaluation Fund made an important contribution to the development of credible evidence on the impact of human development interventions in different thematic areas such as health contracting/pay for performance reforms, conditional cash transfers, malaria control, active labor market programs/youth employment, basic education accountability, HIV/AIDS prevention and early childhood development. Furthermore, SIEF made a significant contribution to the development and consolidation of impact evaluation practice at the World Bank and in developing countries. Although this work was done in the context of developing and transition countries, it may also be of potential interest for Spain not only for its international cooperation projects and programs but also as an input for the design of Spain’s own domestic policies (for example, the results on active labor market programs and youth employment18), and to strengthen its evaluation capacity. On the other hand, SIEF has done less work on systematic

18 An example of an output generated by SIEF that can be useful for Spain along the lines indicated in this paragraph is http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1239047988859/5995659-1239048041095/cf4_zMisc_ALM_SummaryNotes_SIEF-IE-on-ALMP_Jan5_PA_110121.pdf

35

reviews of evidence generated by impact evaluations, apart from a set of books and the organization of cluster conferences.

A peer review process of proposals to be funded by SIEF was put in place as a quality control mechanism. This process combined rigor and flexibility with transparency.

By working in partnership with other funding sources, SIEF was able to leverage its resources. However, the blending of funds makes practically impossible to disentangle SIEF’s contribution from that of other donors.

The SIEF cluster model proved to be a useful approach that can be replicated19. Allocating resources to thematic clusters of IEs rather than to individual IEs facilitated the consolidation of IE communities. SIEF contributed to strengthen IE communities of practice in different HD areas, through on the job training, training workshops and dissemination events. This strengthening has taken place to a greater extent in some clusters, such as Health Contracting / Pay For Performance Reforms and Active Labor Market / Youth Employment. Furthermore, there was less knowledge sharing across clusters, which mainly took place through the regional IE workshops and six monthly (initially quarterly) meetings, convened by the SIEF program management team.

The approach to training introduced an innovative two-tracks format, which matched well the different demands from policy makers, evaluation managers and evaluation specialists. This approach strengthened both the demand and the supply for IE at country level and also at the World Bank Human Development Network, where it played a catalytic role. On the other hand, SIEF has not strengthened evaluation capacities of the Spanish donor (which was one of the objectives included in the agreement establishing the SIEF), except for the IE workshop held in Madrid and the technical advice provided to project teams funded by Spanish cooperation which were interested in designing IE..The “Impact Evaluation in Practice” book, produced by SIEF, and translated into Spanish and French, is a very valuable resource that can be used in the development of impact evaluation capacity in developing countries, the World Bank and in Spain.

Applying a six-point scale ratings for outcomes (Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory) to the implementation of the four SIEF goals, combining evidence from different sources (survey to TTLs, documentation and website review, interviews), the ratings are the following (chapter 4 provides ratings for each agreed result corresponding to the different goals, and the rationale as well as the evidence for the ratings): goal 1) to build capacities for understanding and using Impact Evaluation (IE) techniques, Highly Satisfactory; goal 2) to support the application of impact evaluation techniques to selected development programs to improving Human Development (HD) outcomes: Highly Satisfactory; goal 3) to disseminate SIEF findings, and to promote access to knowledge on the impact of HD programs, Satisfactory; and goal 4) to use the SIEF as a platform for promoting Management for Development Results (MfDR), Satisfactory.

19 The cluster approach for the evaluation of grants was introduced by the Kellogg Foundation in the 90’s, particularly for small grants. Some lessons learned from its experience could be useful in SIEF II. See http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/evaluation-in-the-21st-century/cluster-evaluation http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx

36

An indication of the success of SIEF is that, as it comes to an end, a new multi-donor trust fund for impact evaluations has been established at the World Bank, with an initial endowment provided by the British government’s Department for International Development (a secondary source of funding for SIEF I), with an amount which triples the funding of SIEF. In addition, it is significant that the new fund retains the acronym SIEF (with the “S” now standing for “Strategic”), which is an acknowledgement of the quality of the operations and visibility achieved by the Spanish Impact Evaluation Fund. As stated in the presentation of the new SIEF, it “will build on the knowledge and research generated by the Spanish Trust Fund for Impact Evaluation”.

Recommendations

A new phase of SIEF could focus on systematic reviews of IE results, combining the evidence generated by SIEF funded IEs with evidence from other IE and/or studies, which may allow to go beyond “what works” (a phrase frequently used in SIEF’s publications) to “what works in which circumstances and why”, strengthening the external validity and usefulness of IE findings. There is still a significant proportion of IEs funded by SIEF which are in the process of delivering final results. As their results become available, it will be possible and convenient to produce systematic reviews of IE results.

The more that impact evaluations are designed and implemented for the same type of interventions in different contexts, the greater the extent to which it would be possible to make statements concerning the external validity (or generalizability) of the impact evaluation results.

It would be worthwhile to support also innovative evaluations (as distinct from evaluations of innovative interventions), such as, e.g.,"selective trials”.

In order to increase the flexibility of an impact evaluation fund, it could be worthwhile to complement the periodic calls for proposals with an open and continuous window to support with small contributions the design of evaluations (a sort of seed capital), which could nurture the pipeline of proposals.

In a new fund it would be convenient to promote a wider competition for the funds, facilitating the inclusion of non Bank staff in the teams participating in the call for proposals. This could be complemented with the participation of non Bank academics and/or policy makers as external advisors to the technical committee, so as to ensure a perception of full transparency and to draw on a wider pool of expertise.

Sharing experience and knowledge gained across thematic clusters through periodic meetings , or a forum, may support the development of a community of practice of impact evaluators and could lead to economies of scope. An additional means for sharing knowledge would be an appropriate website (as indicated in the next recommendation).

The development and maintenance of an updated, comprehensive and user-friendly website with information about the program of impact evaluations and results achieved should be one of the key means to share knowledge generated by the program. In order to ensure that the website fulfils its role it would be worthwhile to conduct periodic revisions of the website and to budget adequate resources for the website design, maintenance and revision. Furthermore, it would be

37

convenient to explore the possibility of developing an independent website (e.g., www.sief.org), not subject to the constraints that should be followed when it is a World Bank (www.worldbank.org) website, taking into account that it is a partnership program funded by a donor. The program’s website could have an intranet area for internal monitoring of the program and to facilitate information sharing about processes to be followed and intermediate products.

A proactive approach to dissemination could include during the implementation of the impact evaluations a set of activities (such as videos and photographs) that with a very low marginal cost could enhance significantly the quality of dissemination and, as a by-product, it would make available audiovisual material for training purposes.

On capacity building for impact evaluation, it would be worthwhile to complement the very successful impact evaluation training workshops designed and implemented by SIEF, with the support to the development of institutional capacities for independent impact evaluation at the country level

Furthermore, it would be worthwhile to promote a broader engagement with evaluation communities beyond the World Bank, such as the American Evaluation Association, the European Evaluation Society, the International Development Evaluation Association, the United Nations Evaluation Group and the MDBs Evaluation Cooperation Group. IEs and systematic reviews could be disseminated to them, and presentations could be made in their conferences and/or meetings, seeking their feedback.

Finally, the knowledge generated by SIEF may be used by the Government of Spain to enrich the policy dialogue, particularly in the area of labor markets and youth employment, and to promote the development of impact evaluation capacities.

38

ANNEXES

Annex I

Terms of Reference

Spanish Impact Evaluation Fund Final Report

November 22, 2011

Background

The Trust Fund for Impact Evaluation and Results-based Management in Human Development Sectors, or Spanish Impact Evaluation Fund (SIEF), was established in 2007 by the World Bank and the Government of Spain aiming to support the Impact Evaluation (IE) of innovative programs to improve Human Development (HD) outcomes. Its ultimate goal is to contribute to an increase in the effectiveness of development policies by increasing the evidence base on the impact of actions affecting HD.

The SIEF program is managed by the Bank’s Human Development Network Office of the Chief Economist. SIEF resources support: (i) Prospective, rigorous impact evaluations of programs in 11 eligible Human Development and Sustainable Development sectors and 72 eligible developing countries across all regions; (ii) Intensive training programs for government counterparts, Bank staff, and staff of partner development agencies in impact evaluation methods; and (iii) Publication and dissemination of evaluation results through articles, meta-studies, and web-based materials.

The July 27, 2007 Administration Agreement between the government of Spain and the World Bank, by which the SIEF program was established, specified that “Within six (6) months of the final disbursement date specified in paragraph 10.1, the Bank shall furnish to the Donor a final report on the Project activities”; and that “The Bank will implement a monitoring and evaluation framework with the aim to evaluate the Trust Fund, activities, products and results throughout its implementation”.

On December 3, 2009, the SIEF Steering Committee endorsed a results-based strategy specifying the specific goals, resources, activities, products, and expected outcomes of the program. Following this meeting, the SIEF management team developed a Monitoring & Evaluation Plan that included a final program assessment report –to be produced at the end of the program.

Overall goal of the SIEF final program assessment report (PAR)

The goal of the report is to assess the extent to which the SIEF has achieved its strategic goals:6. To build capacities for understanding and using impact evaluation techniques;7. To support the application of impact evaluation techniques to selected development

programs to improving human development outcomes;8. To disseminate SIEF findings, and to promote access to knowledge on the impact of

human development programs;

39

9. To use the SIEF as a platform for promoting management for development results.

In addition to provide information to the Donor, the SIEF final PAR will be used as an input for potential program continuation and/or up-scaling, as well as for similar programs; for the Implementation Completion Memorandum (ICM) report; and for record-keeping purposes.

Assessment questions:

The assessment will address the following questions on the SIEF design, process and results achieved:

Design level:- To what extent the SIEF design and resources were consistent with its aimed goals?- To what extent did the SIEF have a useful and reliable Strategy and M&E Plan that

contributed to set priorities and track progress towards the achievement of its goals?- To what extend did the SIEF have a useful and reliable Results Dissemination and

Knowledge Sharing Plan?

Process level:- To what extent has the SIEF program management model (i.e. instruments; economic, human

and technical resources; organizational structure; information flows; decision-making in management) been efficient in comparison to the results attained?

- To what extent have cluster leaders contributed to ensure the successful implementation of the SIEF program, and to promoting impact evaluation communities of practice in the program key thematic areas?

- To what extent the governance of the fund at trustee level contributed to efficiency and effectiveness of the SIEF? To what extent was it useful for building an effective partnership between the Bank and donors? Did the Steering Committee enable the SIEF management for an effective delivery of outputs and results?

- To what extent were SIEF resources allocated to impact evaluation teams in a transparent way?

- To what extent and in what ways did the SIEF program management increase or reduce efficiency in delivering outputs and attaining outcomes?

- What type of quality control mechanisms and types of process were established for ensuring the quality of the SIEF impact evaluation portfolio?

- What type of (administrative, financial and managerial) obstacles did the SIEF face and to what extent have this affected its efficiency?

- To what extent and in what ways did SIEF monitoring instruments (Annual and Mid-Year reports) were used and useful?

Results level:- To what extent has the SIEF program achieved its aimed strategic goals?- To what extent have SIEF capacity building activities contributed to a better understanding

and use of impact evaluations?- To what extent SIEF research and dissemination activities have contributed to a better access

to knowledge on the impact of Human Development programs? - When intended results were not achieved as planned, what were the main reasons? - Were there any unintended results?

40

- Are there any lessons learnt on how the SIEF could have achieved or on how similar programs could achieve better results?

- Should the SIEF program be continued / up-scaled, what potential results could be obtained? What changes could be recommended for it to become more effective?

Products and delivery dates:

The consultant is responsible for submitting the following deliverables to the SIEF Program Management:1) Draft report outline and plan and impact evaluation completion template for requesting

evaluation task team leaders the information required for the assessment –to be submitted by December 15, 2011;

2) Interviews made to SIEF team members, cluster leaders, impact evaluation task team leaders and researchers, donors, and targeted government officials, development practitioners and impact evaluation experts from client countries –to be submitted by end of first week of February 2012;

3) Final report. The final report will be 20 to 30 pages in length. It will also contain an executive summary of no more than 5 pages that includes a brief description of the SIEF program, its context and current situation, the purpose of the assessment, its methodology and its major findings, conclusions and recommendations. A draft version of the report will be sent to the SIEF management team by end of February 2012. The final version will be delivered before March 25, 2011, after reception of the SIEF team comments and suggestions;

4) Presentation disseminating the conclusions and recommendations from the draft report among government officials and other development partners in Madrid, Spain –March/April 2012.

41

Budget and work period:

The consultancy includes a total of XXX working days with a daily rate of USD XXX per day, total of USD XXX. The work period is until April 15th, 2012 and it includes one week in Washington DC for interviews.

The source of funds will be IE-P105560-IMPE-TF058114

Supervision:

The Consultant will work under the supervision of the SIEF Monitoring and Evaluation focal point, Carlos Asenjo, reporting directly to the SIEF Program Manager, Laura Rawlings.

42

Annex IIList of persons interviewed

Government of Spain / Currently dealing with SIEF

Bibian Zamora Gimenez, Head Evaluation Unit, MAEC, Government of Spain

David Conde Lozano, MAEC, Government of Spain

Ana De Vicente Lancho, MEH, Government of Spain

Agustín Navarro de Vicente-Gella, MEH, Government of Spain

Leonardo Rodríguez García, MEH, Government of Spain

Maria Jesus Calvo Romerales, MEH, Government of Spain

Government of Spain / Former officers who were involved with SIEF

Juan Lopez Doriga, former Director General for Development Cooperation Policy, MAEC, Government of Spain

Jose Antonio Gonzalez Mancebo, former Under Director General for Development Cooperation Policy, MAEC, Government of Spain

Cecilia Rocha, former Head of the Evaluation Unit, MAEC, Government of Spain

Belen Sanz, former Head of the Evaluation Unit, MAEC, Government of Spain

Carlos Ariza, former staff Evaluation Unit, MAEC, Government of Spain

Ivan Touza, former staff Evaluation Unit, MAEC, Government of Spain

Adan Ruiz, former staff Evaluation Unit, MAEC, Government of Spain

World Bank

Enrique Giménez de Córdoba, Spain ED’s Office, World Bank

Laura Rawlings , current SIEF Program Manager, World Bank

Barbara Bruns, former SIEF Program Manager and Education cluster leader, World Bank

Ariel Fiszbein, HDN Chief Economist, World Bank

Paul Gertler, former HDN Chief Economist

43

Adam Ross, SIEF core team and SIEF MDTF, World Bank

Rita Almeida, ALMP/YE SIEF cluster, World Bank

Mattias Lundberg, ALMP/YE, SIEF cluster, World Bank

Phillippe Leite, CCT SIEF cluster, World Bank

Christel Vermeersch, P4P Health SIEF cluster, World Bank

Damien de Walque, HIV/AIDS SIEF cluster, World Bank

Arianna Legovini, DIME, Malaria Control, SIEF cluster, World Bank

Luis Alberto Andres, SIEF TTL, World Bank

Sebastian Martinez, former SIEF Core Team

Javier Baez, former IEG staff, World Bank

Carlos Asenjo Ruiz, SIEF Core Team,

Paloma Acevedo, SIEF Core Team

Key informants on SIEF Evaluation of Plan Nacer

Martin Sabignoso, National Coordinator, Plan Nacer, Government of Argentina

Humberto Silva, Technical Director,Plan Nacer, Ministerio de Salud, Argentina

Leonardo Zanazzi, Plan Nacer, Ministerio de Salud, Argentina

Luis Orlando Perez, Sr. Public Health Specialist, World Bank, Argentina

Paula Giovagnoli, Plan Nacer, World Bank, Argentina

Annex V

TTL’s survey questionnaire

Spanish Impact Evaluation Fund (SIEF)

Grant Completion Survey

44

As you may recall, the SIEF management team developed a Monitoring & Evaluation Plan that includes annual monitoring reports and a final program assessment report –to be produced at the end of the program. A crucial input for this report is information and insights from TTLs.

This survey aims to collect from TTLs of impact evaluations supported by the SIEF the information required to finalize the GRM Completion reports –which TTLs need to complete within six months after the grant closing date–, as well as to provide inputs for the SIEF Monitoring 2011 and Final Assessment Reports.

Please answer the following questions by January 26.

1. Were the resources allocated to your task(s) consistent with its aimed goals?

2. Please list the key questions addressed by the impact evaluation study

3. Please summarize the method(s) or technique(s) used for the impact evaluation

4. To what extent the goals of your task(s) were achieved? Please select one of the following responses:

( ) Highly Satisfactory( ) Satisfactory( ) Moderately Satisfactory ( ) Moderately Unsatisfactory ( ) Unsatisfactory( ) Highly Unsatisfactory

5. Were there any unintended results from the evaluation?

6. If intended results were not achieved as planned, what were the main reasons?

7. What knowledge results/findings were obtained by the SIEF evaluation in which you participated?

8. Please list the main research products (baseline and follow up / impact evaluation reports) delivered by your task team with SIEF funds

9. How the results achieved by your evaluation task were disseminated? Please list the main results dissemination products (publications / events) delivered by your task team with SIEF funds

10. Were there any knowledge sharing activities using results emerging from the task that you led?

11. To what extent have you found the SIEF program management model (i.e. instruments; economic, human and technical resources; organizational structure; information flows; decision-making in management)

a)efficient (in comparison to the results attained) Please select one of the following responses:

45

( ) Highly Efficient( ) Efficient( ) Moderately Efficient( ) Moderately Inefficient( ) Inefficient( ) Highly Inefficient

b) helpfulPlease select one of the following responses:

( ) Highly Helpful( ) Helpful( ) Moderately Helpful( ) Moderately Unhelpful( ) Unhelpful( ) Highly Unhelpful

12. Have you faced any administrative, financial and/or managerial obstacles and to what extent have this affected the efficiency of your task(s)?

13. To what extent have cluster leaders contributed to ensure the successful implementation of the SIEF task(s) that you led?

Please select one of the following responses:( ) Highly Positive Contribution( ) Positive Contribution( ) Moderately Positive Contribution( ) Moderately Negative Contribution( ) Negative Contribution( ) Highly Negative Contribution

14. Do you consider that cluster leaders promoted impact evaluation communities of practice in the program key thematic areas?

15. To what extent were SIEF resources allocated to your impact evaluation team(s) in a transparent way?

Please select one of the following responses: ( ) Highly Transparent( ) Transparent( ) Moderately Transparent( ) Moderately Opaque( ) Opaque( ) Highly Opaque

16. To what extent and in what ways did the SIEF program management increase or reduce efficiency in delivering outputs and attaining outcomes?

Please select one of the following responses.( ) Highly Increased Efficiency( ) Increased Efficiency( ) Moderately Increased Efficiency( ) Moderately Reduced Efficiency( ) Reduced Efficiency

46

( ) Highly Reduced Efficiency

17. Did the impact evaluation study lead to any changes in the design of the program/project or the way it was implemented? If so, please explain briefly

18. Have the impact evaluation results been released? If so, did they : a) informed any decisions regarding the continuation/expansion of the program? Please explain briefly. b) triggered any policy changes? Please explain briefly.

19. Are there any lessons learnt on how the SIEF could have achieved, or on how similar programs could achieve, better results?

20. Should the SIEF program be continued / up-scaled, what potential results could be obtained? What changes could be recommended for it to become more effective?

21. Would you like to make any other comment(s) or suggestion(s) concerning the SIEF program?

Once completed the survey kindly remember to:

(i) Send it by email to Carlos Asenjo (casenjoruiz@worldbank.org) and Osvaldo Feinstein (ofeinstein@yahoo.com)

ANNEX VI

Preguntas para las reuniones en España

1.. ¿ Hay factores del CONTEXTO que sea conveniente destacar por su influencia durante la implementación del SIEF?

2.¿Qué esperaban en España como resultado del SIEF? ¿ Había alguna expectativa que el SIEF beneficiara de modo directo a España?

3. ¿Concidían las expectativas del MAEC y del MEH? ¿Habían diferencias?

4. ¿Ha generado el SIEF beneficios para España?

5. ¿En qué medida consideras que se alcanzaron los objetivos del SIEF?

6. ¿Hubo efectos no buscados?

47

7. ¿Cuáles fueron los obstáculos o limitaciones que se presentaron en el funcionamiento del SIEF (si los hubo)? Obstáculos que pudieron incidir en una brecha sobre lo que se intentó hacer y y lo que efectivamente se hizo

8. ¿Qué lecciones se pueden derivar de la experiencia del SIEF para futuros proyectos de este tipo?

9 . Sobre la contribución del SIEF: ¿Qué permitió hacer el SIEF que no se hubiera hecho sin SIEF? ¿Qué hubiera pasado sin el SIEF?

10. ¿ Tiene sugerencias sobre personas a contactar y documentos o publicaciones que sería útil tomar en cuenta en la elaboración del informe final del SIEF?

Annex VII

Spanish Impact Evaluation Trust Fund (SIEF)Key Performance Indicators and Financial Summary

As of April 20, 2012

Table 1.- SIEF Key Performance Indicators (June 2008 - April 2012):

Indicators ¹ End FY2008

(Jun 30, 2008)End FY2009

(Jun 30, 2009)End FY2010

(Jun 30, 2010) End FY2011

(Jun 30, 2011) Act. FY2012

(Apr 20, 2012)SIEF Target

Financial performance: - - - - - -

% Disbursed / SIEF initi al tangible as sets ¹ 5% 19% 44% 82% 101% 100% ⁵ ⁶

% Disbursed / SIEF tota l funds ² 9% 22% 43% 77% 93% -

Activity implementation: - - - - - -

% IE teams access ing thei r SIEF a l location 15% 54% 96% 100% 100% 100%

Product delivery: - - - - - -

# IE workshops del ivered 3 6 12 14 17 12 ⁴

# Workshop participants tra ined in IE methods 460 985 1,905 2,035 2,406 1,893 ⁴

# IE basel ine data col lected ² 2 8 13 18 21 -

# IE fol low-up data col lected ² 0 2 8 22 33 -

# IE studies completed ³ 0 2 6 10 27 15 ⁵

# IE impact evaluation s tudies publ is hed 0 2 8 15 19 -

48

¹ % of total SIEF initial tangible assets (SIEF initial contribution once deducted the 5% cost recovery fee).² % of SIEF total funds --including the 5% cost recovery fee and the investment income earned by the trust fund.³ Baseline and follow-up data collections supported by the SIEF only --data collections supported by other sources of funding are not included.⁴ Impact evaluation teams having completed baseline and follow-up analysis --upon which further impact evaluation research may be conducted.⁵ SIEF Strategy Matrix (2009). ⁶ SIEF Administration Agreement (2007)

49

Graph 1.- SIEF Key Financial Indicators (June 2008 - April 2012) ¹ ² ³ :

Table 2.- SIEF Key Financial Indicators (June 2008 - April 2012) ¹ ² ³ : Financial

Flows End FY2008

(Jun 30, 2008)End FY2009

(Jun 30, 2009)End FY2010

(Jun 30, 2010)End FY2011

(Jun 30, 2011)Act. FY2012

(Apr 20, 2012)%

Total Funds 14,658,284 15,139,647 15,271,572 15,320,552 15,324,325 100%

Allocated 13,556,214 15,104,793 15,105,933 15,105,487 15,237,448 99%

Receipts 6,647,125 11,183,960 14,352,270 15,099,644 14,910,309 97%

Committed 2,157,121 4,160,698 8,273,660 12,974,064 14,412,709 94%

Disbursed 708,295 2,711,872 6,200,325 11,562,275 14,208,714 93%

¹ For a description of the indicators used in this table, see Box A.² Data only reflect SIEF support; other sources of financing (World Bank, DFID) are not included.³ % of total funds when the program was established --earned income is not included.

50

Table 3. – SIEF Program: Budget Execution Summary as of April 20, 2012 ¹ ²

Activities Allocated(Apr 20, 2012)

Receipts (Apr 20, 2012)

Committed (Apr 20, 2012)

Disbursed (Apr 20, 2012)

% Disbursed / Receipts (Apr

20, 2012)

% of Total Allocated (Apr

20, 2012)

1. Impact Evaluation Studies (IEs) 10.145.107 9.817.968 9.832.195 9.817.968 97% 67%

1.1. Quick Win IE Studies 879.851 879.851 879.851 879.851 100% 6%

1.2. Innovation Fund IE Studies 2.079.475 2.009.223 2.014.239 2.009.223 97% 14%

1.3. Cluster IE Studies 7.185.781 6.928.894 6.938.105 6.928.894 96% 47%

1.3.1 - Health Contracting / Pay For Performance Reforms

1.324.671 1.250.539 1.257.185 1.250.539 94% 9%

1.3.2 - Conditional Cash Transfers 909.301 895.086 895.086 895.086 98% 6%

1.3.3 - Malaria Control 823.075 762.790 762.790 762.790 93% 5%

1.3.4 - Active Labor Market / Youth Employment

1.333.125 1.276.350 1.276.352 1.276.350 96% 9%

1.3.5 - Basic Education Accountability

1.038.492 1.022.601 1.025.164 1.022.601 98% 7%

1.3.6 - HIV/AIDS Prevention 982.217 981.561 981.561 981.561 100% 6%

1.3.7 - Early Childhood Development

774.900 739.966 739.966 739.966 95% 5%

2. Capacity Building 1.700.000 1.700.000 1.695.495 1.679.868 99% 11%

3. Results Dissemination 1.533.000 1.533.000 1.249.180 1.183.025 77% 10%

3.1. Program Outreach & Results Dissemination

600.000 600.000 489.625 457.019 76% 4%

3.2. Cluster Results Dissemination 933.000 933.000 759.555 726.006 78% 6%

4. Other Activities 1.859.341 1.859.341 1.635.839 1.527.853 82% 12%

4.1. Program Management 781.923 781.923 685.083 620.613 79% 5%

4.2. Cluster Coordination 510.000 510.000 383.338 339.822 67% 3%

4.3. Advisory Services 567.418 567.418 567.418 567.418 100% 4%

5. Total (5=1+2+3+4) 15.237.448 14.910.309 14.412.709 14.208.714 93% 100%

¹ For a description of the indicators used in this table, see Box A.² Data only reflect SIEF support; other sources of financing (World Bank, DFID…) are not included.

51

Table 4.- SIEF Research Program: Budget Execution as of April 20, 2012 ¹ ² ³

Impact Evaluation Studies Country / Num. IE Studies ⁴

Task Team Leader Allocated Receipts Committed Disbursed % Disbursed / Allocated

1.1. Quick Win IE Studies 4 - 608,661 608,661 608,661 608,661 100%Tanzania's Social Action Funds (TASAF II) Tanzania Berk Ozler 169,907 169,907 169,907 169,907 100%Human Development Voucher (Bono de Desarrollo Humano) (Ecuador) Norbert Schady 137,091 137,091 137,091 137,091 100%Long Term Follow-Up of Beneficiaries of a Simulation Program Jamaica Christel Vermeersch 57,065 57,065 57,065 57,065 100%Early Childhood Development (ECD) Brazil Pedro Olinto 119,599 119,599 119,599 119,599 100%Early Childhood Development (ECD) (Mexico) Sebastian Martinez 0 0 0 0 -Extra Teacher Provision (ETP) (Kenya) Barbara Bruns 0 0 0 0 -Interaction of Information and SBM Reforms in Basic Education (Benin) Barbara Bruns 0 0 0 0 -Maharashtra and Orissa Rural Water Supply and Sanitation India Priti Kumar 125,000 125,000 125,000 125,000 100%1.2. Innovation Fund IE Studies 11 - 2,079,475 2,009,223 2,014,239 2,009,223 97%1.2.1. Human Development 4 - 806,075 773,824 773,824 773,824 96%TIVET Vouchers IE Project Kenya Shobhana Sosale 262,000 262,000 262,000 262,000 100%Citizen Report Card Project (Phase II) Uganda Damien de Walque 151,050 151,003 151,003 151,003 100%Evaluation of Poverty Targeting Processes and Mechanisms for Accountability

(Philippines) Xavier Gine 0 0 0 0 -

Ex-Combatant Reintegration and Peacebuilding in Liberia Liberia Mattias Lundberg 333,025 307,276 307,276 307,276 92%Effects of Home Based HIV Counseling and Testing ⁵ Kenya Markus P. Goldstein 60,000 53,545 53,545 53,545 89%1.2.2. Sustainable Development 7 - 1,273,400 1,235,399 1,240,415 1,235,399 97%

Effect of Informal Settlement Upgrading on Health in S. Africa South Africa Arianna Legovini 274,150 258,827 263,843 258,827 94%Domestic Use of Irrigation Water Ethiopia Arianna Legovini 147,000 145,974 145,974 145,974 99%GPOBA/SIEF Uganda Water IE Uganda Luis Tineo 123,000 120,306 120,306 120,306 98%Innovative Treatment Solutions for Rural Drinking Water Sources Kenya Arianna Legovini 200,000 199,671 199,671 199,671 100%Impact of Water Wells and Drainage on Household Welfare and Child Health in Urban Slums

(Nigeria) Pedro Olinto 0 0 0 0 -

Access to Water Infrastructure and HD Outcomes in Rural Areas Paraguay Miguel Vargas-Ramirez 182,850 171,279 171,279 171,279 94%

Sewerage Access and Health Outcomes for Households in Urban Areas

Uruguay Carlos E. Velez 151,400 147,135 147,135 147,135 97%

Water Rights Formalization for Agricultural Use and HD Outcomes Peru Marie-Laure Lajaunie 195,000 192,207 192,207 192,207 99%

52

Table 4.- SIEF Research Program: Budget Execution as of April 20, 2012 ¹ ² ³ (cont.)

Impact Evaluation Studies Country / Num. IE Studies ⁴

Task Team Leader Allocated Receipts Committed Disbursed % Disbursed / Allocated

1.3. Cluster IE Studies 35 - 7,456,971 7,200,084 7,209,295 7,200,084 97%

1.3.1 - Health Contracting / Pay For Performance Reforms 5 A. Wagstaff, A. Ross, C. Vermeersch

1,324,671 1,250,539 1,257,185 1,250,539 94%

Plan Nacer - Child-Mother Provincial Investment Project Argentina Adam Ross 344,766 339,812 339,812 339,812 99%Performance-Based Contracting for Gen. Health and HIV Services Rwanda Christel Vermeersch 209,800 197,383 197,383 197,383 94%Paying for Performance in China's Health Sector (Health XI Eval. ) China Adam Wagstaff 323,629 272,854 276,896 272,854 84%Pay For Performance in the Health Sector in Turkey (Turkey) Sarbani Chakraborty 50,476 50,476 50,476 50,476 100%Poverty Reduction Support Credits Impact on Maternal Health Benin Christophe Lemiere 296,000 296,000 296,000 296,000 100%Innovative Health Insurance for India's Poor India Robert Palacios 100,000 94,013 96,618 94,013 94%1.3.2 - Conditional Cash Transfers 6 P. Leite, E. Galasso, M.

Grosh1,050,492 1,036,277 1,036,277 1,036,277 99%

Conditional and Unconditional Cash Transfers in Nahouri Burkina Faso Damien de Walque 224,600 224,475 224,475 224,475 100%CCT program in Rural Education in Morocco Morocco Nadine Poupart 442,017 442,017 442,017 442,017 100%Estimating the Dynamic Effects of Supportive Chile (Chile Solidario) Chile Emanuela Galasso 108,074 108,074 108,074 108,074 100%Evaluating and Improving the Indonesian Conditional Cash Transfer ⁵ Indonesia Susan Wong 0 0 0 0 -Community Based CCT Pilot Program Tanzania Anush Bezhanyan 160,000 160,000 160,000 160,000 100%Contigo Vamos por Mas Oportunidades program in Guanajuato Mexico Theresa Jones 115,800 101,710 101,710 101,710 88%1.3.3 - Malaria Control 5 Jed Friedman, Arianna

Legovini823,075 762,790 762,790 762,790 93%

The Role of Malaria Control in Improving Education Kenya Donald Bundy 100,222 100,222 100,222 100,222 100%The Role of Malaria Control in Improving Education Senegal Donald Bundy 42,952 42,952 42,952 42,952 100%Improving Malaria Outcomes through Evidence-Based Prog. Design Zambia Jed Friedman 252,687 193,311 193,311 193,311 77%Improving Malaria Outcomes through Evidence-Based Prog. Design (Congo, DR) Jed Friedman 0 0 0 0 -Improving Malaria Outcomes through Evidence-Based Prog. Design India Jed Friedman 243,007 242,369 242,369 242,369 100%Community Distributors and Patent Medicine Vendors for Malaria Control

Nigeria Arianna Legovini 184,207 183,936 183,936 183,936 100%

53

Table 4.- SIEF Research Program: Budget Execution as of April 20, 2012 ¹ ² ³ (cont.)

Impact Evaluation Studies Country / Num. IE Studies ⁴

Task Team Leader Allocated Receipts Committed Disbursed % Disbursed / Allocated

1.3.4 - Active Labor Market Programs / Youth Employment 7 R. Almeida, M. Lundberg

1,343,124 1,286,350 1,286,351 1,286,350 96%

India NREG Evaluation India Rinku Murgai 270,000 254,327 254,327 254,327 94%Youth Development Program Dom. Rep. Juan Martin Moreno 224,625 223,622 223,624 223,622 100%My First Job (Mi Primer Empleo) (Honduras) Cornelia Tesliuc 9,999 9,999 9,999 9,999 100%NUSAF Youth Opportunities Program Uganda Patrick Premand 349,860 349,860 349,860 349,860 100%Jamaica Steps-To-Work (Jamaica) Cornelia Tesliuc 0 0 0 0 -TVST OVAY Impact Evaluation Malawi S. R. Jobanputra 240,000 239,531 239,531 239,531 100%Economic Empowerment of Adolescent Girls (EPAG) Project (Liberia) Mattias Lundberg 0 0 0 0 -Turning Theses into Enterprises Tunisia Stefanie Brodmann 55,000 54,727 54,727 54,727 100%Effectiveness of Turkish National Employment Agency Trainings Turkey Cristobal Ridao-Cano 118,640 117,785 117,785 117,785 99%A Youth Wage Subsidy Experiment South Africa Elizabeth Ninan 75,000 36,498 36,498 36,498 49%1.3.5 - Basic Education Accountability 5 H. Patrinos, D. Filmer, B.

Bruns1,038,492 1,022,601 1,025,164 1,022,601 98%

Pernambuco and Sao Paulo Perf. Pay Reform for School Teachers Brazil Barbara Bruns 295,873 295,873 298,436 295,873 100%Nepal Community Managed School Nepal Nazmul Chaudhury 150,000 149,931 149,931 149,931 100%AGEs - Support to School Management Parental Empowerment Mexico Harry A. Patrinos 292,739 292,606 292,606 292,606 100%Role of Assessment to Provide Information for Accountability Liberia Nathalie Lahire 199,880 185,805 185,805 185,805 93%Randomized Evaluation of Andhra Pradesh Primary Education India V. Sundararaman 100,000 98,386 98,386 98,386 98%1.3.6 - HIV/AIDS Prevention 3 David Wilson, Damien

De Walque1,102,217 1,101,561 1,101,561 1,101,561 100%

Evaluating CCT to Prevent HIV and Other STIs Tanzania Damien De Walque 371,400 371,131 371,131 371,131 100%Evaluating the Impact of HIV Prevention Campaign for the Youth Lesotho Damien De Walque 384,680 384,680 384,680 384,680 100%CCT, Schooling, and HIV Risk Malawi Berk Ozler 346,137 345,751 345,751 345,751 100%

54

Table 4.- SIEF Research Program: Budget Execution as of April 20, 2012 ¹ ² ³ (cont.)

Impact Evaluation Studies Country / Num. IE Studies ⁴

Task Team Leader Allocated Receipts Committed Disbursed % Disbursed / Allocated

1.3.7 - Early Childhood Development 4 Emanuela Galasso 774,900 739,966 739,966 739,966 95%IE and Cost-Benefit Analysis of 3 types of FTI-funded ECD Interv. Cambodia Sophie Naudeau 211,000 200,948 200,948 200,948 95%Nadie es Perfecto, IE of a Program to Improve Parenting Skills Chile Emanuela Galasso 225,000 223,309 223,309 223,309 99%IE of ECD in Mozambique Mozambique Sophie Naudeau 164,900 141,743 141,743 141,743 86%Atencion a Crisis ECD and CCT Pilots Nicaragua Renos Vakis 174,000 173,966 173,966 173,966 100%Jamaica Evaluation of Parenting Program ⁵ Jamaica Harriet Nannyonjo 0 0 0 0 - Total SIEF Research program 50 - 10,145,107 9,817,968 9,832,195 9,817,968 97%

Table 5.- SIEF Cluster Coordination and Dissemination Activities: Budget Execution as of April 20, 2012

Impact Evaluation Studies Country / Num. IE Studies ⁴

Task Team Leader Allocated Receipts Committed Disbursed % Disbursed / Allocated

SIEF Cluster 1 Coordination and Results Dissemination World Christel Vermeersch 185,000 185,000 154,728 136,883 74% SIEF Cluster 2 Coordination and Results Dissemination World Emanuela Galasso 185,000 185,000 120,950 112,598 61% SIEF Cluster 3 Coordination and Results Dissemination World Jed Friedman 249,000 249,000 199,747 173,811 70% SIEF Cluster 4 Coordination and Results Dissemination World Rita Almeida 250,000 250,000 192,913 192,913 77% SIEF Cluster 5 coordination and Results Dissemination World Harry Patrinos 229,000 229,000 186,022 186,022 81% SIEF Cluster 6 coordination and Results Dissemination World Damien de Walque 185,000 185,000 180,773 162,341 88% SIEF Cluster 7 coordination and Results Dissemination World Emanuela Galasso 160,000 160,000 107,760 101,260 63%

Total Cluster Coordination - - 1,443,000 1,443,000 1,142,893 1,065,828 74%

55

Table 6. – SIEF Program: Budget Execution Summary (Previous Years)

Activities Allocated End_FY2008

Receipts End_FY2008

Committed End_FY2008

Disbursed End_FY2008

Allocated End_FY2009

Receipts End_FY2009

Committed End_FY2009

Disbursed End_FY2009

1. Impact Evaluation Studies (IEs) 7,412,874 983,785 1,454,042 350,172 8,961,453 5,235,620 2,708,440 1,604,570

1.1. Quick Win IE Studies 1,483,955 943,785 350,172 350,172 879,851 922,786 762,101 762,101

1.2. Innovation Fund IE Studies 0 0 376,040 0 2,689,185 576,025 429,336 53,296

1.3. Cluster IE Studies ³ 5,928,919 40,000 727,830 0 5,392,417 3,736,809 1,517,002 789,173

1.3.1 - Health Contracting / Pay For Performance Reforms

1,194,183 40,000 61,438 0 1,047,681 1,099,566 164,891 103,453

1.3.2 - Conditional Cash Transfers

1,085,740 0 15,700 0 1,085,740 411,140 216,730 201,030

1.3.3 - Malaria Control 602,735 0 202,164 0 602,735 493,868 362,127 159,963

1.3.4 - Active Labor Market / Youth Employment

1,538,125 0 162,456 0 1,148,125 641,935 350,692 188,236

1.3.5 - Basic Education Accountability

756,436 0 254,972 0 756,436 566,300 262,534 7,561

1.3.6 - HIV/AIDS Prevention 751,700 0 31,100 0 751,700 524,000 160,029 128,929

1.3.7 - Early Childhood Development

0 0 0 0 0 0 0 0

2. Capacity Building ⁴ 1,500,000 1,500,000 465,278 259,619 1,500,000 1,500,000 856,271 650,612

3. Results Dissemination 1,500,000 1,500,000 56,396 11,021 1,500,000 1,500,000 110,438 65,063

3.1. Program Outreach & Results Dissemination

1,500,000 1,500,000 56,396 11,021 1,500,000 1,500,000 110,438 65,063

3.2. Cluster Results Dissemination 0 0 0 0 0 0 0 0

4. Other Activities 3,143,340 2,663,340 181,405 87,482 3,143,340 2,948,340 485,550 391,627

4.1. Program Management 2,133,340 2,133,340 53,708 15,375 2,045,092 2,045,092 169,073 130,740

4.2. Cluster Coordination 510,000 30,000 55,590 0 510,000 315,000 104,228 48,638

4.3. Advisory Services 500,000 500,000 72,107 72,107 588,248 588,248 212,249 212,249

5. Total (5=1+2+3+4) 13,556,214 6,647,125 2,157,121 708,295 15,104,793 11,183,960 4,160,698 2,711,872

56

Table 6. – SIEF Program: Budget Execution Summary (Previous Years)

Activities Allocated End_FY2010

Receipts End_FY2010

Committed End_FY2010

Disbursed End_FY2010

Allocated End_FY2011

Receipts End_FY2011

Committed End_FY2011

Disbursed End_FY2011

1. Impact Evaluation Studies (IEs) 9,886,593 9,583,279 5,693,005 3,817,452 10,186,147 10,180,303 9,511,033 8,207,793

1.1. Quick Win IE Studies 879,851 879,851 879,851 879,851 879,851 879,851 879,851 879,851

1.2. Innovation Fund IE Studies 2,046,025 2,042,125 1,091,375 789,040 2,083,375 2,079,475 1,940,185 1,607,267

1.3. Cluster IE Studies ³ 6,960,716 6,661,303 3,721,779 2,148,560 7,222,920 7,220,977 6,690,998 5,720,675

1.3.1 - Health Contracting / Pay For Performance Reforms

1,357,681 1,356,042 763,225 336,404 1,326,310 1,324,671 1,223,883 1,018,047

1.3.2 - Conditional Cash Transfers

811,545 811,545 456,590 352,252 911,540 911,494 854,821 623,715

1.3.3 - Malaria Control 752,735 753,075 468,229 288,179 822,735 823,075 710,972 697,945

1.3.4 - Active Labor Market / Youth Employment

1,323,125 1,079,485 863,136 566,704 1,333,125 1,333,125 1,202,887 1,122,786

1.3.5 - Basic Education Accountability

982,913 929,039 651,858 392,959 1,071,493 1,071,494 1,007,593 1,000,975

1.3.6 - HIV/AIDS Prevention 982,717 982,217 500,840 194,160 982,717 982,217 955,526 935,916

1.3.7 - Early Childhood Development

750,000 749,900 17,903 17,903 775,000 774,900 735,316 321,290

2. Capacity Building ⁴ 1,700,000 1,849,650 1,346,385 1,303,402 1,700,000 1,700,000 1,425,358 1,406,715

3. Results Dissemination 1,385,000 785,000 340,011 269,943 1,360,000 1,360,000 615,223 602,639

3.1. Program Outreach & Results Dissemination

600,000 600,000 243,407 168,872 600,000 600,000 383,463 361,140

3.2. Cluster Results Dissemination 785,000 185,000 96,604 101,071 760,000 760,000 231,760 241,499

4. Other Activities 2,134,341 2,134,341 894,259 809,528 1,859,341 1,859,341 1,422,450 1,345,128

4.1. Program Management 1,373,862 1,088,714 172,290 139,353 813,714 781,923 471,695 437,888

4.2. Cluster Coordination 510,000 510,000 186,342 134,548 510,000 510,000 383,338 339,822

4.3. Advisory Services 250,479 535,627 535,627 535,627 535,627 567,418 567,418 567,418

5. Total (5=1+2+3+4) 15,105,933 14,352,270 8,273,660 6,200,325 15,105,487 15,099,644 12,974,064 11,562,275

57

Table 7.- SIEF Research Program: Budget Execution (Previous Years)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2008 Receipts

End_FY2008 Committed End_FY2008

Disbursed End_FY2008

Allocated End_FY2009

Receipts End_FY2009

Committed End_FY2009

Disbursed End_FY2009

1.1. Quick Win IE Studies 4 - 1,145,170 715,000 283,197 283,197 608,661 651,596 608,661 608,661

Tanzania's Social Action Funds (TASAF II)

Tanzania Berk Ozler 170,000 170,000 0 0 169,907 169,907 169,907 169,907

Human Development Voucher (Bono de Desarrollo Humano)

(Ecuador) Norbert Schady 400,000 200,000 58,094 58,094 137,091 137,091 137,091 137,091

Long Term Follow-Up of Beneficiaries of a Simulation Program

Jamaica Christel Vermeersch 100,000 100,000 5,966 5,966 57,065 100,000 57,065 57,065

Early Childhood Development (ECD) Brazil Pedro Olinto 120,000 120,000 94,137 94,137 119,599 119,599 119,599 119,599

Early Childhood Development (ECD) (Mexico) Sebastian Martinez 110,290 0 0 0 0 0 0 0

Extra Teacher Provision (ETP) (Kenya) Barbara Bruns 64,880 0 0 0 0 0 0 0Interaction of Information and SBM Reforms in Basic Education

(Benin) Barbara Bruns 55,000 0 0 0 0 0 0 0

Maharashtra and Orissa Rural Water Supply and Sanitation

India Priti Kumar 125,000 125,000 125,000 125,000 125,000 125,000 125,000 125,000

1.2. Innovation Fund IE Studies 11 - 0 0 376,040 0 2,689,185 576,025 429,336 53,296

1.2.1. Human Development 4 - 0 0 190,862 0 1,144,435 453,025 244,158 53,296

TIVET Vouchers IE Project Kenya Shobhana Sosale 0 0 144,100 0 262,200 0 144,100 0

Citizen Report Card Project (Phase II) Uganda Damien de Walque 0 0 4,802 0 151,050 120,000 43,484 38,682

Evaluation of Poverty Targeting Processes and Mechanisms for Accountability

(Philippines) Xavier Gine 0 0 0 0 398,160 0 0 0

Ex-Combatant Reintegration and Peacebuilding in Liberia

Liberia Mattias Lundberg 0 0 41,960 0 333,025 333,025 56,575 14,615

Effects of Home Based HIV Counseling and Testing ⁵

Kenya Markus P. Goldstein 0 0 0 0 0 0 0 0

58

Table 7.- SIEF Research Program: Budget Execution (Previous Years; cont.)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2010 Receipts

End_FY2010 Committed End_FY2010

Disbursed End_FY2010

Allocated End_FY2011

Receipts End_FY2011

Committed End_FY2011

Disbursed End_FY2011

1.1. Quick Win IE Studies 4 - 608,661 608,661 608,661 608,661 608,661 608,661 608,661 608,661

Tanzania's Social Action Funds (TASAF II)

Tanzania Berk Ozler 169,907 169,907 169,907 169,907 169,907 169,907 169,907 169,907

Human Development Voucher (Bono de Desarrollo Humano)

(Ecuador) Norbert Schady 137,091 137,091 137,091 137,091 137,091 137,091 137,091 137,091

Long Term Follow-Up of Beneficiaries of a Simulation Program

Jamaica Christel Vermeersch 57,065 57,065 57,065 57,065 57,065 57,065 57,065 57,065

Early Childhood Development (ECD) Brazil Pedro Olinto 119,599 119,599 119,599 119,599 119,599 119,599 119,599 119,599

Early Childhood Development (ECD) (Mexico) Sebastian Martinez 0 0 0 0 0 0 0 0

Extra Teacher Provision (ETP) (Kenya) Barbara Bruns 0 0 0 0 0 0 0 0Interaction of Information and SBM Reforms in Basic Education

(Benin) Barbara Bruns 0 0 0 0 0 0 0 0

Maharashtra and Orissa Rural Water Supply and Sanitation

India Priti Kumar 125,000 125,000 125,000 125,000 125,000 125,000 125,000 125,000

1.2. Innovation Fund IE Studies 11 - 2,046,025 2,042,125 1,091,375 789,040 2,083,375 2,079,475 1,940,185 1,607,267

1.2.1. Human Development 4 - 746,275 746,075 681,732 529,340 806,275 806,075 765,065 716,470

TIVET Vouchers IE Project Kenya Shobhana Sosale 262,200 262,000 262,000 222,700 262,200 262,000 262,000 262,000

Citizen Report Card Project (Phase II) Uganda Damien de Walque 151,050 151,050 121,219 117,028 151,050 151,050 149,526 149,526

Evaluation of Poverty Targeting Processes and Mechanisms for Accountability

(Philippines) Xavier Gine 0 0 0 0 0 0 0 0

Ex-Combatant Reintegration and Peacebuilding in Liberia

Liberia Mattias Lundberg 333,025 333,025 298,512 189,612 333,025 333,025 304,944 304,944

Effects of Home Based HIV Counseling and Testing ⁵

Kenya Markus P. Goldstein 0 0 0 0 60,000 60,000 48,595 0

59

Table 7.- SIEF Research Program: Budget Execution (Previous Years; cont.)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2008 Receipts

End_FY2008 Committed End_FY2008

Disbursed End_FY2008

Allocated End_FY2009

Receipts End_FY2009

Committed End_FY2009

Disbursed End_FY2009

1.2. Innovation Fund IE Studies 11 - 0 0 376,040 0 2,689,185 576,025 429,336 53,296

1.2.2. Sustainable Development 7 - 0 0 185,178 0 1,544,750 123,000 185,178 0

Effect of Informal Settlement Upgrading on Health in S. Africa

South Africa Arianna Legovini 0 0 41,470 0 300,000 0 41,470 0

Domestic Use of Irrigation Water Ethiopia Arianna Legovini 0 0 9,620 0 147,500 0 9,620 0

GPOBA/SIEF Uganda Water IE Uganda Luis Tineo 0 0 0 0 123,000 123,000 0 0Innovative Treatment Solutions for Rural Drinking Water Sources

Kenya Arianna Legovini 0 0 134,088 0 200,000 0 134,088 0

Impact of Water Wells and Drainage on Household Welfare and Child Health in Urban Slums

(Nigeria) Pedro Olinto 0 0 0 0 245,000 0 0 0

Access to Water Infrastructure and HD Outcomes in Rural Areas

ParaguayMiguel Vargas-Ramirez

0 0 0 0 182,850 0 0 0

Sewerage Access and Health Outcomes for Households in Urban Areas

Uruguay Carlos E. Velez 0 0 0 0 151,400 0 0 0

Water Rights Formalization for Agricultural Use and HD Outcomes

Peru Marie-Laure Lajaunie 0 0 0 0 195,000 0 0 0

1.3. Cluster IE Studies 35 - 6,267,704 268,785 794,805 66,975 5,663,607 4,007,999 1,670,442 942,613

1.3.1 - Health Contracting / Pay For Performance Reforms

5A. Wagstaff, A. Ross, C. Vermeersch

1,194,183 40,000 61,438 0 1,047,681 1,099,566 164,891 103,453

Plan Nacer - Child-Mother Provincial Investment Project

Argentina Adam Ross 164,766 0 39,762 0 164,766 164,766 54,867 15,104

Performance-Based Contracting for General Health and HIV Services

Rwanda Christel Vermeersch 134,800 40,000 16,380 0 134,800 40,000 53,477 37,097

Paying for Performance in China's Health Sector (Evaluation of Health XI)

China Adam Wagstaff 400,000 0 0 0 400,000 400,000 0 0

Pay For Performance in the Health Sector in Turkey

(Turkey) Sarbani Chakraborty 198,217 0 1,045 0 51,715 198,800 52,297 51,252

Poverty Reduction Support Credits Impact on Maternal Health

Benin Christophe Lemiere 296,400 0 0 0 296,400 296,000 0 0

Innovative Health Insurance for India's Poor

India Robert Palacios 0 0 4,250 0 0 0 4,250 0

60

Table 7.- SIEF Research Program: Budget Execution (Previous Years; cont.)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2010 Receipts

End_FY2010 Committed End_FY2010

Disbursed End_FY2010

Allocated End_FY2011

Receipts End_FY2011

Committed End_FY2011

Disbursed End_FY2011

1.2. Innovation Fund IE Studies 11 - 2,046,025 2,042,125 1,091,375 789,040 2,083,375 2,079,475 1,940,185 1,607,267

1.2.2. Sustainable Development 7 - 1,299,750 1,296,050 409,643 259,700 1,277,100 1,273,400 1,175,119 890,797

Effect of Informal Settlement Upgrading on Health in S. Africa

South Africa Arianna Legovini 300,000 296,800 126,683 126,083 277,350 274,150 239,228 229,575

Domestic Use of Irrigation Water Ethiopia Arianna Legovini 147,500 147,000 96,526 96,526 147,500 147,000 145,913 145,913

GPOBA/SIEF Uganda Water IE Uganda Luis Tineo 123,000 123,000 0 0 123,000 123,000 116,317 114,717Innovative Treatment Solutions for Rural Drinking Water Sources

Kenya Arianna Legovini 200,000 200,000 167,610 33,522 200,000 200,000 167,610 167,610

Impact of Water Wells and Drainage on Household Welfare and Child Health in Urban Slums

(Nigeria) Pedro Olinto 0 0 0 0 0 0 0 0

Access to Water Infrastructure and HD Outcomes in Rural Areas

ParaguayMiguel Vargas-Ramirez

182,850 182,850 0 0 182,850 182,850 181,693 59,331

Sewerage Access and Health Outcomes for Households in Urban Areas

Uruguay Carlos E. Velez 151,400 151,400 18,825 3,569 151,400 151,400 138,698 106,398

Water Rights Formalization for Agricultural Use and HD Outcomes

Peru Marie-Laure Lajaunie 195,000 195,000 0 0 195,000 195,000 185,659 67,252

1.3. Cluster IE Studies 35 - 7,231,906 6,932,493 3,992,969 2,419,751 7,494,110 7,492,167 6,962,188 5,991,865

1.3.1 - Health Contracting / Pay For Performance Reforms

5A. Wagstaff, A. Ross, C. Vermeersch

1,357,681 1,356,042 763,225 336,404 1,326,310 1,324,671 1,223,883 1,018,047

Plan Nacer - Child-Mother Provincial Investment Project

Argentina Adam Ross 314,766 314,766 199,268 123,126 344,766 344,766 321,971 288,956

Performance-Based Contracting for General Health and HIV Services

Rwanda Christel Vermeersch 194,800 194,800 127,319 100,799 209,800 209,800 192,164 181,624

Paying for Performance in China's Health Sector (Evaluation of Health XI)

China Adam Wagstaff 400,000 400,000 20,356 20,356 323,629 323,629 272,886 114,945

Pay For Performance in the Health Sector in Turkey

(Turkey) Sarbani Chakraborty 51,715 50,476 50,476 50,476 51,715 50,476 50,476 50,476

Poverty Reduction Support Credits Impact on Maternal Health

Benin Christophe Lemiere 296,400 296,000 296,000 0 296,400 296,000 296,000 296,000

Innovative Health Insurance for India's Poor

India Robert Palacios 100,000 100,000 69,806 41,647 100,000 100,000 90,385 86,047

61

Table 7.- SIEF Research Program: Budget Execution (Previous Years; cont.)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2008 Receipts

End_FY2008 Committed End_FY2008

Disbursed End_FY2008

Allocated End_FY2009

Receipts End_FY2009

Committed End_FY2009

Disbursed End_FY2009

1.3. Cluster IE Studies 35 - 6,267,704 268,785 794,805 66,975 5,663,607 4,007,999 1,670,442 942,613

1.3.2 - Conditional Cash Transfers 6P. Leite, E. Galasso, M. Grosh

1,184,525 98,785 72,676 56,976 1,226,931 552,331 357,921 342,221

Conditional and Unconditional Cash Transfers in the Nahouri Province

Burkina Faso Damien de Walque 224,600 0 0 0 224,600 0 0 0

CCT program in Rural Education in Morocco

Morocco Nadine Poupart 450,000 50,000 41,316 41,316 494,211 444,211 266,660 266,660

Estimating the Dynamic Effects of Supportive Chile (Chile Solidario)

Chile Emanuela Galasso 109,925 48,785 31,360 15,660 108,120 108,120 91,261 75,561

Evaluating and Improving the Indonesian Conditional Cash Transfer

Indonesia Susan Wong 400,000 0 0 0 400,000 0 0 0

Community Based CCT Pilot Program Tanzania Anush Bezhanyan 0 0 0 0 0 0 0 0Contigo Vamos por Mas Oportunidades program in Guanajuato

Mexico Theresa Jones 0 0 0 0 0 0 0 0

1.3.3 - Malaria Control 5Jed Friedman, Arianna Legovini

602,735 0 202,164 0 602,735 493,868 362,127 159,963

The Role of Malaria Control in Improving Education

Kenya Donald Bundy 36,587 0 71,587 0 36,587 36,587 71,587 0

The Role of Malaria Control in Improving Education

Senegal Donald Bundy 36,247 0 71,587 0 36,247 36,587 71,587 0

Improving Malaria Outcomes through Evidence-Based Program Design

Zambia Jed Friedman 247,687 0 0 0 247,687 247,687 129,236 129,236

Improving Malaria Outcomes through Evidence-Based Program Design

(Congo, DR) Jed Friedman 109,207 0 0 0 109,207 0 0 0

Improving Malaria Outcomes through Evidence-Based Program Design

India Jed Friedman 173,007 0 50,590 0 173,007 173,007 81,317 30,727

Community Distributors and Patent Medicine Vendors for Malaria Control

Nigeria Arianna Legovini 0 0 8,400 0 0 0 8,400 0

62

Table 7.- SIEF Research Program: Budget Execution (Previous Years; cont.)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2010 Receipts

End_FY2010 Committed End_FY2010

Disbursed End_FY2010

Allocated End_FY2011

Receipts End_FY2011

Committed End_FY2011

Disbursed End_FY2011

1.3. Cluster IE Studies 35 - 7,231,906 6,932,493 3,992,969 2,419,751 7,494,110 7,492,167 6,962,188 5,991,865

1.3.2 - Conditional Cash Transfers 6P. Leite, E. Galasso, M. Grosh

952,736 952,736 597,781 493,443 1,052,731 1,052,685 996,011 764,906

Conditional and Unconditional Cash Transfers in the Nahouri Province

Burkina Faso Damien de Walque 224,600 224,600 210,543 110,405 224,600 224,600 224,475 224,475

CCT program in Rural Education in Morocco

Morocco Nadine Poupart 444,211 444,211 279,163 279,163 444,211 444,211 400,414 400,414

Estimating the Dynamic Effects of Supportive Chile (Chile Solidario)

Chile Emanuela Galasso 108,120 108,120 108,074 103,874 108,120 108,074 108,074 108,074

Evaluating and Improving the Indonesian Conditional Cash Transfer

Indonesia Susan Wong 15,805 15,805 0 0 0 0 0 0

Community Based CCT Pilot Program Tanzania Anush Bezhanyan 60,000 60,000 0 0 160,000 160,000 158,682 17,728Contigo Vamos por Mas Oportunidades program in Guanajuato

Mexico Theresa Jones 100,000 100,000 0 0 115,800 115,800 104,366 14,215

1.3.3 - Malaria Control 5Jed Friedman, Arianna Legovini

752,735 753,075 468,229 288,179 822,735 823,075 710,972 697,945

The Role of Malaria Control in Improving Education

Kenya Donald Bundy 71,587 71,587 71,587 0 100,222 100,222 100,222 100,222

The Role of Malaria Control in Improving Education

Senegal Donald Bundy 71,247 71,587 71,587 0 42,612 42,952 42,952 42,952

Improving Malaria Outcomes through Evidence-Based Program Design

Zambia Jed Friedman 272,687 272,687 129,236 129,236 252,687 252,687 172,664 170,538

Improving Malaria Outcomes through Evidence-Based Program Design

(Congo, DR) Jed Friedman 0 0 0 0 0 0 0 0

Improving Malaria Outcomes through Evidence-Based Program Design

India Jed Friedman 173,007 173,007 120,237 90,201 243,007 243,007 224,037 218,316

Community Distributors and Patent Medicine Vendors for Malaria Control

Nigeria Arianna Legovini 164,207 164,207 75,582 68,742 184,207 184,207 171,097 165,917

63

Table 7.- SIEF Research Program: Budget Execution (Previous Years; cont.)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2008 Receipts

End_FY2008 Committed End_FY2008

Disbursed End_FY2008

Allocated End_FY2009

Receipts End_FY2009

Committed End_FY2009

Disbursed End_FY2009

1.3. Cluster IE Studies 35 - 6,267,704 268,785 794,805 66,975 5,663,607 4,007,999 1,670,442 942,613

1.3.4 - Active Labor Market Programs / Youth Employment

7 R. Almeida, M. Lundberg

1,658,125 10,000 172,455 9,999 1,158,124 651,934 360,691 198,235

India NREG Evaluation India Rinku Murgai 390,000 0 46,835 0 290,000 290,000 71,962 25,128Youth Development Program Dom. Rep. Juan Martin Moreno 249,625 0 115,621 0 149,625 173,075 128,507 12,886My First Job (Mi Primer Empleo) (Honduras) Cornelia Tesliuc 308,640 10,000 9,999 9,999 208,639 9,999 9,999 9,999NUSAF Youth Opportunities Program Uganda Patrick Premand 309,860 0 0 0 209,860 178,860 150,222 150,222

Jamaica Steps-To-Work (Jamaica) Cornelia Tesliuc 400,000 0 0 0 300,000 0 0 0

TVST OVAY Impact Evaluation MalawiSangeeta Raja Jobanputra

0 0 0 0 0 0 0 0

Economic Empowerment of Adolescent Girls (EPAG) Project

(Liberia) Mattias Lundberg 0 0 0 0 0 0 0 0

Turning Theses into Enterprises Tunisia Stefanie Brodmann 0 0 0 0 0 0 0 0Effectiveness of Turkish National Employment Agency Trainings

Turkey Cristobal Ridao-Cano 0 0 0 0 0 0 0 0

A Youth Wage Subsidy Experiment South Africa Elizabeth Ninan 0 0 0 0 0 0 0 0

1.3.5 - Basic Education Accountability 5H. Patrinos, D. Filmer, B. Bruns

756,436 0 254,972 0 756,436 566,300 262,534 7,561

Pernambuco and Sao Paulo Performance Pay Reform for School Teachers

Brazil Barbara Bruns 175,000 0 14,438 0 175,000 175,000 19,026 4,587

Nepal Community Managed School Nepal Nazmul Chaudhury 150,000 0 0 0 150,000 150,000 2,974 2,974AGEs - Support to School Management Parental Empowerment

Mexico Harry A. Patrinos 220,136 0 185,744 0 220,136 30,000 185,744 0

Role of Assessment to Provide Information for Accountability

Liberia Nathalie Lahire 211,300 0 11,100 0 211,300 211,300 11,100 0

Randomized Evaluation of Andhra Pradesh Primary Education

India Venkatesh Sundararaman

0 0 43,690 0 0 0 43,690 0

64

Table 7.- SIEF Research Program: Budget Execution (Previous Years; cont.)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2010 Receipts

End_FY2010 Committed End_FY2010

Disbursed End_FY2010

Allocated End_FY2011

Receipts End_FY2011

Committed End_FY2011

Disbursed End_FY2011

1.3. Cluster IE Studies 35 - 7,231,906 6,932,493 3,992,969 2,419,751 7,494,110 7,492,167 6,962,188 5,991,865

1.3.4 - Active Labor Market Programs / Youth Employment

7 R. Almeida, M. Lundberg

1,333,124 1,089,484 873,135 576,703 1,343,124 1,343,124 1,212,887 1,132,786

India NREG Evaluation India Rinku Murgai 290,000 290,000 250,257 182,613 270,000 270,000 214,761 214,761Youth Development Program Dom. Rep. Juan Martin Moreno 199,625 199,625 185,582 102,107 224,625 224,625 199,127 170,333My First Job (Mi Primer Empleo) (Honduras) Cornelia Tesliuc 9,999 9,999 9,999 9,999 9,999 9,999 9,999 9,999NUSAF Youth Opportunities Program Uganda Patrick Premand 334,860 334,860 329,522 209,833 349,860 349,860 349,860 341,351

Jamaica Steps-To-Work (Jamaica) Cornelia Tesliuc 0 0 0 0 0 0 0 0

TVST OVAY Impact Evaluation MalawiSangeeta Raja Jobanputra

200,000 200,000 89,551 63,927 240,000 240,000 234,442 224,477

Economic Empowerment of Adolescent Girls (EPAG) Project

(Liberia) Mattias Lundberg 70,000 0 0 0 0 0 0 0

Turning Theses into Enterprises Tunisia Stefanie Brodmann 55,000 55,000 8,223 8,223 55,000 55,000 54,790 53,457Effectiveness of Turkish National Employment Agency Trainings

Turkey Cristobal Ridao-Cano 98,640 0 0 0 118,640 118,640 117,978 117,978

A Youth Wage Subsidy Experiment South Africa Elizabeth Ninan 75,000 0 0 0 75,000 75,000 31,930 430

1.3.5 - Basic Education Accountability 5H. Patrinos, D. Filmer, B. Bruns

982,913 929,039 651,858 392,959 1,071,493 1,071,494 1,007,593 1,000,975

Pernambuco and Sao Paulo Performance Pay Reform for School Teachers

Brazil Barbara Bruns 228,874 175,000 78,439 78,439 328,874 328,875 280,864 274,247

Nepal Community Managed School Nepal Nazmul Chaudhury 150,000 150,000 92,015 92,015 150,000 150,000 149,931 149,931AGEs - Support to School Management Parental Empowerment

Mexico Harry A. Patrinos 292,739 292,739 243,667 177,709 292,739 292,739 292,607 292,606

Role of Assessment to Provide Information for Accountability

Liberia Nathalie Lahire 211,300 211,300 189,180 10,200 199,880 199,880 185,805 185,805

Randomized Evaluation of Andhra Pradesh Primary Education

India Venkatesh Sundararaman

100,000 100,000 48,556 34,596 100,000 100,000 98,386 98,386

65

Table 7.- SIEF Research Program: Budget Execution (Previous Years; cont.)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2008 Receipts

End_FY2008 Committed End_FY2008

Disbursed End_FY2008

Allocated End_FY2009

Receipts End_FY2009

Committed End_FY2009

Disbursed End_FY2009

1.3. Cluster IE Studies 35 - 6,267,704 268,785 794,805 66,975 5,663,607 4,007,999 1,670,442 942,613

1.3.6 - HIV/AIDS Prevention 3 David Wilson, Damien De Walque

871,700 120,000 31,100 0 871,700 644,000 162,279 131,179

Evaluating CCT to Prevent HIV and Other STIs

Tanzania Damien De Walque 290,400 0 8,000 0 290,400 250,000 93,000 85,000

Evaluating the Impact of HIV Prevention Campaign for the Youth

Lesotho Damien De Walque 306,800 120,000 0 0 306,800 120,000 2,250 2,250

CCT, Schooling, and HIV Risk Malawi Berk Ozler 274,500 0 23,100 0 274,500 274,000 67,029 43,929

1.3.7 - Early Childhood Development 4 Emanuela Galasso 0 0 0 0 0 0 0 0

Impact Evaluation and Cost-Benefit Analysis of 3 types of FTI-funded ECD

Cambodia Sophie Naudeau 0 0 0 0 0 0 0 0

Nadie es Perfecto, IE of a Program to Improve Parenting Skills

Chile Emanuela Galasso 0 0 0 0 0 0 0 0

IE of ECD in Mozambique Mozambique Sophie Naudeau 0 0 0 0 0 0 0 0Atencion a Crisis Early Childhood Development & Conditional Cash

Nicaragua Renos Vakis 0 0 0 0 0 0 0 0

Jamaica Evaluation of Parenting Program

Jamaica Harriet Nannyonjo 0 0 0 0 0 0 0 0

Total SIEF Research program 50 - 7,412,874 983,785 1,454,042 350,172 8,961,453 5,235,620 2,708,440 1,604,570

66

Table 7.- SIEF Research Program: Budget Execution (Previous Years; cont.)

Impact Evaluation Studies Country/# IEs ⁴ Task Team Leader Allocated

End_FY2010 Receipts

End_FY2010 Committed End_FY2010

Disbursed End_FY2010

Allocated End_FY2011

Receipts End_FY2011

Committed End_FY2011

Disbursed End_FY2011

1.3. Cluster IE Studies 35 - 7,231,906 6,932,493 3,992,969 2,419,751 7,494,110 7,492,167 6,962,188 5,991,865

1.3.6 - HIV/AIDS Prevention 3 David Wilson, Damien De Walque

1,102,717 1,102,217 620,840 314,160 1,102,717 1,102,217 1,075,526 1,055,916

Evaluating CCT to Prevent HIV and Other STIs

Tanzania Damien De Walque 371,400 371,400 272,655 85,000 371,400 371,400 365,686 347,259

Evaluating the Impact of HIV Prevention Campaign for the Youth

Lesotho Damien De Walque 384,680 384,680 128,325 120,000 384,680 384,680 384,680 384,680

CCT, Schooling, and HIV Risk Malawi Berk Ozler 346,637 346,137 219,860 109,160 346,637 346,137 325,160 323,977

1.3.7 - Early Childhood Development 4 Emanuela Galasso 750,000 749,900 17,903 17,903 775,000 774,900 735,316 321,290

Impact Evaluation and Cost-Benefit Analysis of 3 types of FTI-funded ECD

Cambodia Sophie Naudeau 211,000 211,000 0 0 211,000 211,000 200,948 58,618

Nadie es Perfecto, IE of a Program to Improve Parenting Skills

Chile Emanuela Galasso 200,000 200,000 0 0 225,000 225,000 223,309 26,279

IE of ECD in Mozambique Mozambique Sophie Naudeau 165,000 164,900 17,903 17,903 165,000 164,900 141,318 136,118Atencion a Crisis Early Childhood Development & Conditional Cash

Nicaragua Renos Vakis 174,000 174,000 0 0 174,000 174,000 169,741 100,275

Jamaica Evaluation of Parenting Program

Jamaica Harriet Nannyonjo 0 0 0 0 0 0 0 0

Total SIEF Research program 50 - 9,886,593 9,583,279 5,693,005 3,817,452 10,186,147 10,180,303 9,511,033 8,207,793

67

Table 8.- SIEF Cluster Coordination and Dissemination Budget Execution (Previous Years)

Impact Evaluation Studies Country / Num. IE Studies ⁴

Task Team Leader Allocated End_FY2008

Receipts End_FY2008

Committed End_FY2008

Disbursed End_FY2008

Allocated End_FY2009

Receipts End_FY2009

Committed End_FY2009

Disbursed End_FY2009

SIEF Cluster 1 Coordination World Christel Vermeersch 85,000 0 0 0 85,000 85,000 9,450 9,450 SIEF Cluster 2 Coordination World Emanuela Galasso 85,000 0 0 0 85,000 0 0 0 SIEF Cluster 3 Coordination World Jed Friedman 85,000 0 21,540 0 85,000 85,000 42,343 20,803 SIEF Cluster 4 Coordination World Rita Almeida 85,000 30,000 34,050 0 85,000 85,000 40,133 6,083 SIEF Cluster 5 coordination World Harry Patrinos 85,000 0 0 0 85,000 30,000 5,681 5,681 SIEF Cluster 6 coordination World Damien de Walque 85,000 0 0 0 85,000 30,000 6,621 6,621 SIEF Cluster 7 coordination World Emanuela Galasso 0 0 0 0 0 0 0 0

Total Cluster Coordination - - 510,000 30,000 55,590 0 510,000 315,000 104,228 48,638

Table 8.- SIEF Cluster Coordination and Dissemination Budget Execution (Previous Years; cont.

Impact Evaluation Studies Country / Num. IE Studies ⁴

Task Team Leader Allocated End_FY2010

Receipts End_FY2010

Committed End_FY2010

Disbursed End_FY2010

Allocated End_FY2011

Receipts End_FY2011

Committed End_FY2011

Disbursed End_FY2011

SIEF Cluster 1 Coordination World Christel Vermeersch 185,000 85,000 17,785 17,785 185,000 185,000 62,178 62,178 SIEF Cluster 2 Coordination World Emanuela Galasso 185,000 85,000 20,080 20,080 185,000 185,000 33,059 33,059 SIEF Cluster 3 Coordination World Jed Friedman 185,000 85,000 77,460 64,460 185,000 185,000 74,623 74,623 SIEF Cluster 4 Coordination World Rita Almeida 185,000 85,000 78,913 50,713 185,000 185,000 116,428 95,628 SIEF Cluster 5 coordination World Harry Patrinos 185,000 85,000 49,341 45,714 185,000 185,000 183,906 180,279 SIEF Cluster 6 coordination World Damien de Walque 185,000 185,000 28,864 28,864 185,000 185,000 66,164 66,164 SIEF Cluster 7 coordination World Emanuela Galasso 185,000 85,000 10,503 8,003 160,000 160,000 78,740 69,390

Total Cluster Coordination - - 1,295,000 695,000 282,946 235,619 1,270,000 1,270,000 615,098 581,321

68

Table 9. – SIEF Impact Evaluation Program: Total Disbursements by Source of Funding (2007 – April 2012)

Project No. Activities Country Task Team Leader / Coordinator SIEF Disbursed

Total disbursed as of YTD March

FY12($ 000's)

Of which:TFs

Of which:WB Budget

% SIEF / Total Disbursed

- 1.1. Quick Win IE Studies - - 879.851 2.020.415 1.718.654 301.762 44%P108101 Tanzania's Social Action Funds (TASAF II) Tanzania Berk Ozler 169.907 986.022 862.480 123.542 17%P103413 Human Development Voucher (Bono de Desarrollo Humano) Ecuador Norbert Schady 137.091 287.429 258.071 29.358 48%

P106250 Long Term Follow-Up of Beneficiaries of a Simulation ProgramJamaica Christel Vermeersch 57.065 107.065 107.065 0 53%

P109590 Early Childhood Development (ECD) Brazil Pedro Olinto 119.599 415.663 266.802 148.861 29%P113404 Early Childhood Development (ECD) Mexico Sebastian Martinez 0 99.236 99.236 0 0%- Extra Teacher Provision (ETP) Kenya Barbara Bruns 0 0 0 0 -- Interaction of Information and SBM Reforms in Basic Education Benin Barbara Bruns 0 0 0 0 -P109258 Maharashtra and Orissa Rural Water Supply and Sanitation India Priti Kumar 125.000 125.000 125.000 0 100% - 1.2. Innovation Fund IE Studies - - 2.009.223 5.525.130 3.686.595 1.838.535 36%- 1.2.1. Human Development - - 773.824 2.516.337 2.365.040 151.298 31%P112248 TIVET Vouchers IE Project Kenya Shobhana Sosale 262.000 417.327 397.000 20.327 63%P089386 Citizen Report Card Project (Phase II) Uganda Damien de Walque 151.003 700.664 642.576 58.088 22%- Evaluation of Poverty Targeting Processes and Mechanisms for AccountabilityPhilippines Xavier Gine 0 0 0 -P114924 Ex-Combatant Reintegration and Peacebuilding in Liberia Liberia Mattias Lundberg 307.276 886.770 862.110 24.659 35%P115705 Effects of Home Based HIV Counseling and Testing Kenya Markus P. Goldstein 53.545 511.577 463.354 48.223 10%- 1.2.2. Sustainable Development - - 1.235.399 3.008.793 1.321.556 1.687.237 41%P117545 Building a Brighter Future: The Effect of Informal Settlement Upgrading on Health in South AfricaSouth Africa Arianna Legovini 258.827 258.827 258.827 0 100%P117733 Domestic Use of Irrigation Water Ethiopia Arianna Legovini 145.974 152.939 145.974 6.965 95%P110442 GPOBA/SIEF Uganda Water IE Uganda Luis Tineo 120.306 245.463 206.463 39.000 49%P117474 Innovative Treatment Solutions for Rural Drinking Water SourcesKenya Arianna Legovini 199.671 199.671 199.671 0 100%- Impact of Water Wells and Drainage on Household Welfare and Child Health in Urban SlumsNigeria Pedro Olinto 0 0 0 -

P095235 IE -Water Access to Rural Communities in Paraguay Paraguay Miguel Vargas-Ramirez 171.279 978.580 171.279 807.302 18%

P101432 IE - Sewarage access in Uruguay Uruguay Carlos E. Velez 147.135 348.874 147.135 201.739 42%

P104760 Water Rights in Peru Peru Marie-Laure Lajaunie 192.207 824.438 192.207 632.232 23%

69

Table 9. – SIEF Impact Evaluation Program: Total Disbursements by Source of Funding (2007 – April 2012; cont.)

Project No. Activities Country Task Team Leader / Coordinator SIEF Disbursed

Total disbursed as of YTD March

FY12($ 000's)

Of which:TFs

Of which:WB Budget

% SIEF / Total Disbursed

- 1.3. Cluster IE Studies - - 6.928.894 19.185.267 14.372.112 4.813.154 36%- 1.3.1 - Health Contracting / Pay For Performance Reforms - Christel Vermeersch, 1.250.539 2.407.861 2.098.192 309.669 52%P105804 Plan Nacer - Child-Mother Provincial Investment Project Argentina Adam Ross 339.812 418.504 414.502 4.002 81%

P098456 Performance-Based Contracting for General Health and HIV/AIDS ServicesRwanda Christel Vermeersch 197.383 448.817 427.299 21.518 44%

P116722 Paying for Performance in China's Health Sector (Evaluation of Health XI)China Adam Wagstaff 272.854 273.006 272.854 152 100%

P111055 Pay For Performance in the Health Sector in Turkey Turkey Sarbani Chakraborty 50.476 56.749 50.476 6.272 89%

P113202 Poverty Reduction Support Credits Impact on Maternal Health Benin Christophe Lemiere 296.000 1.068.758 839.048 229.709 28%P118734 Health Insurance Implementation Evaluation India Robert Palacios 94.013 142.028 94.013 48.015 66%- 1.3.2 - Conditional Cash Transfers - Margaret Grosh,

Phillippe Leite, 895.086 2.888.349 1.849.109 1.039.240 31%

P099430 Evaluation of the Impact of a Pilot Project of Conditional and Unconditional Cash Transfers in the Nahouri ProvinceBurkina Faso Damien de Walque 224.475 758.400 599.625 158.776 30%

P109306 CCT program in Rural Education in Morocco Morocco Nadine Poupart 347.806 730.069 447.538 282.531 48%P110350 Estimating the Dynamic Effects of Supportive Chile (Chile Solidario)Chile Emanuela Galasso 61.094 105.059 105.059 0 58%P119655 Evaluating and Improving the Indonesian Conditional Cash TransferIndonesia Susan Wong 0 210 210 0 0%P102681 Community Based CCT Pilot Program Tanzania Anush Bezhanyan 160.000 970.818 594.967 375.851 16%P115067 Contigo Vamos por Mas Oportunidades program in the State of GuanajuatoMexico Theresa Jones 101.710 323.793 101.710 222.082 31%

- 1.3.3 - Malaria Control - Jed Friedman, Arianna Legovini 762.790 1.271.699 1.139.227 132.471 60%

P113951 School Health and Nutrition: the Role of Malaria Control in Improving EducationKenya Donald Bundy 100.222 100.222 100.222 0 100%P113954 School Health and Nutrition: the Role of Malaria Control in Improving EducationSenegal Donald Bundy 42.952 42.952 42.952 0 100%P113223 Improving Malaria Outcomes through Evidence-Based Program DesignZambia Jed Friedman 193.311 243.311 243.311 0 79%- Improving Malaria Outcomes through Evidence-Based Program DesignCongo, DR Jed Friedman 0 0 0 -P106233 Improving Malaria Outcomes through Evidence-Based Program DesignIndia Jed Friedman 242.369 327.916 306.172 21.743 74%P105846 IE of the Training of Community Directed Distributors and Patent Medicine Vendors in the Prevention and Treatment of Malaria in Anamba StateNigeria Arianna Legovini 183.936 557.298 446.570 110.728 33%

70

Table 9. – SIEF Impact Evaluation Program: Total Disbursements by Source of Funding (2007 – April 2012; cont.)

Project No. Activities Country Task Team Leader /

Coordinator SIEF Disbursed

Total disbursed as of YTD March

FY12($ 000's)

Of which:TFs

Of which:WB Budget

% SIEF / Total Disbursed

- 1.3. Cluster IE Studies - - 6.928.894 19.185.267 14.372.112 4.813.154 36%

- 1.3.4 - Active Labor Market Programs / Youth Employment - Rita Almeida, Mattias Lundberg 1.276.350 3.989.433 1.937.434 2.051.999 32%

P104857 India National Rural Employment Guarantee Evaluation India Rinku Murgai 254.327 780.834 474.324 306.510 33%

P112569 IE - YDP, Youth Development Program (PJE, Programa Juventud y Empleo)Dominican

RepublicJuan Martin Moreno 223.622 223.665 223.665 0 100%

- My First Job (Mi Primer Empleo) Honduras Cornelia Tesliuc 9.999 9.999 9.999 0 100%P105730 NUSAF Youth Opportunities Program (YOP) Uganda Patrick Premand 349.860 642.201 642.201 0 54%- Jamaica Steps-To-Work Jamaica Cornelia Tesliuc 0 0 0 -

P121256 SIEF-Cluster 4-Malawi / TVST OVAY Impact Evaluation MalawiSangeeta Raja Jobanputra

239.531 367.191 278.536 88.654 65%

P108322 Economic Empowerment of Adolescent Girls (EPAG) Project Liberia Mattias Lundberg 0 72.082 0 72.082 0%P111813 Turning Theses into Enterprises Tunisia Stefanie Brodmann 54.727 273.593 54.727 218.865 20%

P120514 SIEF-Cluster 4-Turkey Turkey Cristobal Ridao-Cano 117.785 518.575 217.484 301.092 23%

P112927 A Youth Wage Subsidy Experiment for South Africa South Africa Elizabeth Ninan 36.498 1.101.293 36.498 1.064.795 3%

- 1.3.5 - Basic Education Accountability - Harry Patrinos, Deon Filmer, Barbara Bruns 1.022.601 4.406.753 3.937.931 468.822 23%

P109100 Evaluating the Effects of a Performance Pay Reform for School Teachers in Pernambuco and Sao PauloBrazil Barbara Bruns 295.873 348.547 336.543 12.004 85%P099792 Nepal Community Managed School project Nepal Nazmul Chaudhury 149.931 899.606 785.540 114.066 17%P111606 AGEs - Support to School Management Parental Empowerment (Apoyo a la Gestion Escolar)Mexico Harry A. Patrinos 292.606 327.802 327.802 0 89%P103454 What Is the Role of Assessment in Providing Information for Accountability?Liberia Nathalie Lahire 185.805 1.377.786 1.248.352 129.434 13%

P099794 Randomized Evaluation of Andhra Pradesh Primary Education IndiaVenkatesh Sundararaman

98.386 1.453.012 1.239.694 213.318 7%

- 1.3.6 - HIV/AIDS Prevention - David Wilson, Damien De Walque

981.561 1.914.493 1.711.886 202.608 51%

P113775 Evaluating CCT to Prevent HIV and Other Sexually Transmitted Infections (STIs)Tanzania Damien De Walque 371.131 767.866 678.297 89.569 48%P110611 Evaluating the Impact of HIV Prevention Campaign for the Youth: Abstinence, Fidelity and Safe SexLesotho Damien De Walque 264.680 725.838 687.838 38.000 36%P111939 CCT, Schooling, and HIV Risk Malawi Berk Ozler 345.751 420.790 345.751 75.039 82%- 1.3.7 - Early Childhood Development - Emanuela Galasso 739.966 2.306.679 1.698.333 608.345 32%P113999 Impact Evaluation and Cost-Benefit Analysis of 3 types of FTI-funded ECD InterventionsCambodia Sophie Naudeau 200.948 426.070 349.875 76.195 47%P122231 Nadie es Perfecto, IE of a Program to Improve Parenting Skills Chile Emanuela Galasso 223.309 373.629 277.417 96.212 60%P121439 IE of ECD in Mozambique Mozambique Sophie Naudeau 141.743 141.743 141.743 0 100%P097266 Atencion a Crisis Early Childhood Development & Conditional Cash Transfer PilotsNicaragua Renos Vakis 173.966 1.025.243 914.527 110.715 17%P095673 Jamaica Evaluation of Parenting Program Jamaica Harriet Nannyonjo 0 339.995 14.771 325.223 0%

- Total SIEF Impact Evaluations - - 9.817.968 26.730.812 19.777.361 6.953.451 37%

71

Box A. Financial flows and budget execution indicators

In this report four types of financial flows have been taken into consideration:

Allocations are the amounts that have been addressed by the SIEF Steering and Technical Committees or by the SIEF Program Manager to specific activities or projects. -Strictu senso, these amounts are not flows, but formal decisions regarding what receipts transfers, commitments and disbursements will be made.

Receipts are the amounts that are transferred from the SIEF main account to specific grant accounts so that impact evaluation teams (or SIEF core team members in case of activities delivered directly by them, such workshops or advisory services) can have access to the amounts that have been allocated to the projects or activities for which they are responsible.

Commitments are the payments to third parties -such as firms or consultants- that impact evaluation teams (or SIEF core team members in cases as explained above) have formally agreed to make conditioned to their delivering of a specific product or service.

Disbursements are the amounts that are paid to such third parties upon receipt of the agreed deliverables.

Three budget execution indicators are used in this report:

% Receipts / allocated reflects the extend until which SIEF allocations have been channeled towards the individuals (SIEF core team members, cluster leaders and impact evaluation task team leaders) that are directly accountable for delivering specific products or services.

% Committed / allocated reflects the extend until which SIEF allocations are being used to hire firms and consultants, or to address staff time in order to ensure the delivery of products and/or services under a specific planned timeline.

% Disbursed / allocated reflects the extend until which SIEF allocations have been used to pay for the products and/or services delivered.

72