Booz I Allensiteresources.worldbank.org/IDA/Resources/IDA13_Results_Inception... · Booz I Allen 1 Hamilton 6002 Allen Hamilton Inc. 8283 Greensboro Drive McLean, VA 22102 Tel 1-703-902-5000

Booz I Allen 1 Hamilton

6002 Allen Hamilton Inc. 8283 Greensboro Drive McLean, VA 22102

Tel 1-703-902-5000 Fax 1-703-902-3333

www.boozallen.com

June 2,2004

Mr. James W. Adams, Vice President Operations, Policy and Country Services The World Bank Group 1818 H Street, NW Washington, DC 20043

Subject: IDA13 Resul ts Measurement System: S p r i n g 2004 Upda te

Dear Mr. Adams:

Booz A l l en H a m i l t o n is pleased to submit, our Inception Report for the above- referenced project.

This repor t i s based on further analysis, clarifications a n d incorporates comments Bank staff comments on the Dra f t Inception Report, w h i c h was submitted on M a y 17,2004. W e w i s h to acknowledge with appreciation the guidance a n d suppor t p rov ided by the management and staff of the World Bank Group. Booz A l l e n H a m i l t o n has devoted i ts best professional efforts to this assignment. Al l analysis, f indings a n d conclusions represent our best judgment based on the in format ion p rov ided to us.

Sincerely, c------”~\

Samuel Mintz, Ph.D. Program Manager

BOOZ ALLEN HAMILTON INC.

THE WORLD BANK GROUP

INTERNATIONAL DEVELOPMENT ASSOCIATION

IDA RESULTS M€ASUR€M€NTSYST€M SPRING 2004 UPDATE

INCEPTION REPORT

JUNE 02,2004

Booz I Allen I Hamilton

ABBREVIATIONS and ACRONYMS

CAS

CFAA

CPAR

GDP

GNI

GNlpc

HNP

I CA

IDA

LCU

PCR

PEM

PER

PRSP

RMS

UIS

UNESCO

UNICEF

WHO

Country Assistance Strategy

Country Financial Accountability Assessment

Country Procurement Assessment Reviews

Gross Domestic Product

Gross National Income

Gross National Income per Capita

Health, Nutrition, and Population Department (World Bank)

Investment Climate Assessment

International Development Association

Local Currency Unit

Primary Completion Rate

Public Expenditure Management

Public Expenditure Review

Poverty Reduction Strategy Paper

Results Management System

UNESCO Institute for Statistics

United Nations Educational, Scientific, and Cultural Organization

United Nations Children’s Fund

World Health Organization

Booz I Allen I Hamilton i

TABLE OF CONTENTS

EXECUTIVE SUMMARY ........................................................................................ ES-1 I . INTRODUCTION ..................................................................................................... 1

1.1 OBJECTIVES AND SCOPE OF WORK ...................................................................... 1 1.2 SCOPE OFENGAGEMENT ...................................................................................... 2 1.3 PROCEDURES ....................................................................................................... 2

INPUT TARGETS ................................................................................................ 3 IDENTIFICATION OF WUT TARGETS .................................................................... 3 VERIFICATION OF ACHIEVED TARGETS ................................................................ 3

OUTCOME TARGETS ....................................................................................... 4 111.1 PRIVATE SECTOR DEVELOPMENT ........................................................................ 4 111.1.1 REVIEW OF DATA PRODUCTION ....................................................................... 4 111.1.2 REVIEW OF OUTCOME PROCEDURES ................................................................ 6 111.2 EDUCA~ON .......................................................................................................... 7 111.3 HEALTH ............................................................................................................. 11

I V . RESULTS ............................................................................................................ 12 IV. l PRIVATE SECTOR DEVELOPMENT ...................................................................... 12 IV.2 EDUCATION ........................................................................................................ 14 IV.3 HEALTH ............................................................................................................. 14

SUMMARY OF FINDINGS .................................................................................. 15

I1 . II.1 11.2

I11 .

V . ANNEXES ........................................................................................ A-1

. Booz I Alien 1 Hamilton

.. 11

List of Tables

Table 1 : Verification of Input Targets

Table 2: Outcome Targets and Reported Results: Education

Table 3. Verified Outcome Results: Private Sector Development

Table 4: Verified Outcome Results: Education

Table 5. Verified Outcome Results: Health

List of Annexes

Annex A: Annex B: Annex C: Annex D: Education Data Annex E: Health Data Annex F: Input Indicator Data Annex G: References and Bibliography

IDA Spring 2004 Evaluation and Measurement System Targets World Bank Staff Interviewed Private Sector Development Data

JAllen I Hamilton I P

... Ill

IDA- 13 Results Measurement System: Progress and Proposals Inception Report

EXECUTIVE SUMMARY

This Inception Report presents the initial findings of a review conducted by Booz Allen Hamilton (Booz Allen) of the procedures used by the International Development Association (IDA) to assemble relevant data for its measurement of results in connection with the Thirteenth Replenishment of resources process (IDAl3). Booz Allen was asked to examine whether the IDA1 3 Spring 2004 targets were met and whether the procedures used to make this assessment were reliable.

Background. The IDA1 3 replenishment program is a comprehensive, three-year process in which IDA and its donor nations discuss assistance objectives for the world’s poorest countries, and which culminates in donor commitments to replenish IDA funds. In 2002, pursuant to a series of resolutions by its Executive Directors, IDA introduced a Results Measurement System (RMS) to the policy framework for IDA1 3 that measures results prior and during the replenishment period. The RMS is designed to strengthen IDA’S focus on development outcomes and promote greater transparency by informing donors about the effectiveness of assistance.

The RMS in place since July 2002 tracks the effectiveness of assistance on two levels: At the institutional level, input indicators based on the delivery of selected analytic and diagnostic assessments are used to determine whether IDA has attained the targeted results. The assessments measure IDA’S performance in key aspects of development effectiveness: financial accountability, procurement assessment, public expenditures management, investment climate, and poverty reduction.’ These are the key areas that underpin IDA’S dialogue with governments about the proper use of public resources and other aspects of development effectiveness.

0 At the country level, outcome indicators measure the progress of all development partners, including IDA and country governments, in achieving desired development goals in the areas that are critical to growth and poverty reduction: education (primary completion rate), health (measles immunization rate) and private sector development (time and cost of business start-up).

0

As part of its RMS, IDA established and reviewed targets for monitoring and evaluating both the input indicators and outcome indicators.2 Additional donor contributions for replenishment are linked to the achievement of these targets. IDA Management reported on progress against its Spring 2004 input targets as well as progress on the outcome targets.

These assessment reports are: Country Financial Accountability Assessments (CFAA), Country Procurement Assessment Reviews (CPAR), Public Expenditure Reviews (PER), Country Assistance Strategies (CAS) and Investment Climate Assessments (ICA).

See Schedule B to Attachment I1 of Additions to IDA Resources: Thirteenth Replenishment: Supporting Poverty Reduction Strategies (lDA/SecM2002-0488), September 17, 2002 (spring 2004 input and outcome indicator targets) (appended hereto as Annex A).

1

2

P - 600z 1 Allen I Hamilton

ES- 1

IDA- 13 Results Measurement System: Progress and Proposals Inception Report - P

The Studv. In May 2004, Booz Allen was engaged by the World Bank to provide an independent review of IDA’S progress towards meeting its IDA13 Spring 2004 results measurement targets. Specifically, Booz Allen was asked to:

0 Review the input indicator targets, and determine whether IDA has met the requirements for analytical work as set forth in the lDAl 3 replenishment resolution; Review the data supporting the outcome indicator targets, examine the procedures used by IDA staff in assembling and assessing the relevant data (which may have been collected by outside entities), and determine whether these procedures reliably will enable IDA to determine that the lDAl 3 Spring 2004 targets have been met; and

0

0 Based on the review, determine whether the IDA1 3 Spring 2004 targets have been met.

A Booz Allen team (the “Team”) consisting of 6 people began work at the IDA off ices on Monday, May 17. This Team included experts with backgrounds in economics, business and statistics.

The Team received full access to the IDA13 database of basic analytical, assessment and other material supporting the input and outcome indicators, as well as related documentation concerning the lDAl 3 replenishment program. In addition, the Team conducted interviews with IDA managers and staff responsible for the results measurement program. The Team also reviewed the procedures used to produce, collect, and evaluate the data necessary to generate the outcome results.

Summarv of Findings. Based on its review of the data set provided by IDA and the interviews conducted with World Bank Group managers and staff, the Team reached four principal findings:

First, on an overall basis, we find that IDA met its target objectives with respect to both the input indicator and outcome indicator targets as set forth in the IDA1 3 replenishment resolution.

reported the number of identified analytical work reports set forth in the IDA1 3 replenishment resolution.

used to produce, collect and evaluate the underlying data that support the outcome results are calculated within the limits of the system to lead to an accurate measurement of outcome results, and that those processes were followed by IDA during the IDA1 3 Spring 2004 review.

the outcome results met and in some cases exceeded the target objectives.

Second, with respect to input indicator targets, we find that IDA produced and

Third, with respect to outcome indicator targets, we find that the IDA processes

Fourth, again specifically with respect to outcome indicator targets, we find that

L - Booz 1 Allen I Hamilton

ES-2

IDA- 13 Results Measurement System: Progress and Proposals InceDtion Report

1. INTRODUCTION

1.1 The International Development Association (IDA) of the World Bank Group engaged Booz Allen Hamilton to produce an inception report to review the procedures used by the IDA in assembling the relevant data and ascertain whether these procedures are reliable in determining that the IDA 13 Spring 2004 targets have been met. This Inception Report presents the findings and conclusions resulting from this review.

The lDAl 3 replenishment program is a comprehensive, three-year process in which IDA and its donor nations discuss development assistance objectives for the world’s poorest countries, and which culminates in donor commitments to replenish IDA funds. In 2002, pursuant to a series of resolutions by its Executive Directors, IDA introduced a Results Measurement System to the policy framework for lDAl 3 that measures results prior and during the replenishment period. The RMS is designed to strengthen IDA’s focus on development outcomes and promote greater transparency by informing donors about the effectiveness of assistance.

Objectives and Scope of Work

The RMS, in place since July 2002, tracks the effectiveness of assistance on two levels, 0 At the institutional level, input indicators, based on the delivery of selected

analytic and diagnostic assessments, are used to determine whether IDA has attained the targeted results. The assessments measure IDA’s performance in the areas that underpin IDA’S dialogue with governments about proper use of public resources and other key aspects of development effectiveness: financial accountability, procurement assessment, public expenditures management, investment climate, and poverty reduction. These are the key areas that underpin IDA’s dialogue with governments about the proper use of public resources and other aspects of development effectiveness. At the country level, outcome indicators are used to measure the progress of all development partners, including IDA and country governments, in achieving desired development goals in the areas that are critical to growth and poverty reduction: education (primary completion rate), health (measles immunization rate) and private sector development (time and cost of business start-up).

0

For input targets, the Team determined whether or not IDA had satisfied the requirements for analytical work as set forth in the IDA13 replenishment resolution. The analytical work included production of the following reports:

Country Financial Accountability Assessments; Country Procurement Assessment Reviews; Public Expenditure Reviews; Investment Climate Assessments; Country Assistance Strategies; and Poverty Analyses.

Booz I Allen I Hamilton 1

IDA- 13 Results Measurement System: Progress and Proposals Inception Report - - For the outcome targets, the Team examined the procedures used by the IDA staff in assembling the relevant data to verify whether these procedures were reliable in determining that the IDA1 3 Spring 2004 targets were met in private sector development, education and health.

The targets set forth for the private sector development were Reduce time required for business start-up (in numbers of business days) by 7 percent from end-2001 to end-2003; and Reduce formal cost of business start-up (in percent of GDP per capita) by 7 percent from end-2001 to end-2003.

0

0

The targets set forth for education were 0 Increase the population-weighted average primary completion rate to 69

percent with a substantial number of countries reaching a higher rate; and Increase the number of countries with positive growth rates in primary completion rates to 38 countries.

0

The targets set forth for health were 0 increase in population-weighted coverage rate of measles immunization

to 60 percent with a substantial number of countries reaching a higher rate; and Increase in number of countries with 80 percent coverage of measles vaccination to 29 countries.

0

1.2 Scope of Engagement

The scope of this engagement entailed a review of the input and outcome targets set forth in the IDA13 replenishment resolution and the provision of an inception report. Specifically Booz Allen was asked to:

0

0

0

1.3

Review the input indicator targets, and determine whether IDA has met the requirements for analytical work as set forth in the IDAl 3 replenishment resolution; Review the data supporting the outcome indicator targets, examine the procedures used by IDA staff in assembling and assessing the relevant data (which may have been collected by outside entities), and determine whether these procedures reliably will enable IDA to determine that the IDA1 3 Spring 2004 targets have been met; and

Based on the review, determine whether the IDAl 3 Spring 2004 targets have been met.

Procedures

To determine whether IDA has met the input indicator targets, the Team first identified all the analytical reports produced under the IDA13 framework and counted and observed the completeness of the reports to verify that the targets were achieved.

Booz I Allen 1 Hamilton 2

IDA-1 3 Results Measurement System: Progress and Proposals Inception Report 7 - - For the outcome targets, the Team first identified all of the indicators associated with the private sector development, health, and education. Next, we examined the procedures by interviewing respective IDA staff and collected relevant documentddata. Next, the methods used by IDA to calculate the results were explained and observed. Finally, we verified that the procedures were reliable in determining that targets had been met.

If. INPUT TARGETS

11.1 Identification of Input Targets

In reviewing the input targets, the Team identified and verified four analytical work reports: (i) Country Financial Accountability Assessments (CFAA); (ii) Country Procurement Assessment Reviews (CPAR); (iii) Public Expenditure Reviews (PER); and, (iv) Investment Climate Assessments (ICA). Moreover, we identified and verified the underpinning of every Country Assistance Strategy (CAS) with current poverty analysis.

11.2 Verification of Achieved Targets

To determine whether IDA had met the requirements for analytical work as set forth in the IDA1 3 replenishment resolution for the period between fiscal year 2001 and Spring 2004, the Team verified that all of the analytical work that IDA had produced was present at IDA, either in electronic or hard copy form. Based on our examination, we verified the existence of:

0 16 ICA reports; 0 42 CPAR reports; 0 51 CFAA reports; and

42 PER reports.

The Team also verified that of 27 CAS reports discussed by the Board since July 1, 2002, 21 were underpinned by current poverty analysis. The remaining 6 CAS, which identified gaps in poverty data also included a plan for addressing the issue. To establish that at least 50% of the CFAA, PER and CPAR reports addressed countries in Africa, we verified the number of reports produced in African countries . For CPAR, 19 were from Africa; for PER, 20 were from Africa, and for CFAA, 21 were from Africa.

Table 1 presents a summary of the analytical work verified.

__i


TARGET REPORTED

Analytical Total Africa Total Africa

CFAA 40 20 51 21

CPAR 38 19 42 19

PER 40 20 42 20

ICA 14 - 16

Work

111. OUTCOME TARGETS

VERIFIED

Total Africa

51 21

42 19

42 20

16

111.1 Private Sector Development

In order to verify the outcome targets set forth in the IDA1 3 target requirements, the Team examined the procedures used to produce the data necessary to estimate the time and formal cost required to set up a business. The Team then reviewed the procedures used to calculate the outcome results. Baseline data were available for a sample of 39 of the 81 IDA countries.

111.1.1 Review of Data Production

The World Bank staff produces the data for private sector indicators. Specifically, the data collection is a product of the "Doing Business" Project managed in the Private Sector Vice Presidency of the Bank. (This is not the case, as discussed further below, for the data collection process pertaining to education and the health sector indicators.)

The procedures for data collection in the Doing Business Project begin with the preparation of a desk study of the laws and regulations in the target country concerning business registration and other relevant business start-up issues. Desk study procedures, to ensure comparability across countries, contain clearly defined criteria. Such criteria include: the type of company, the type of city where the registration takes place, size of capital and number of workers.

See Appendix F for more details on the input targets. 3


IDA- 13 Results Measurement System: Progress and Proposals Inception Report P P

The product of the desk study is a country procedural guide that can be used by future entrepreneurs and others wishing to establish a business. The procedures described in the guide are based on existing legal and regulatory instruments and an assessment of the corresponding costs and time for completing each procedure. The content of this draft is then submitted to an iterative validation process with several local incorporated lawyers and other experts with the requisite expertise in business registration under the particular national law. They then confirm or invalidate each component of the draft procedures. The process of validation is completed when consensus has been reached as to the exact nature, sequence, cost and timeframe of the procedures. At that time, a baseline for the indicators is developed. Updates of the results from regular consultations with the local lawyers allow the timely update of the estimates for each indicator and permit changes in the time and cost to be tracked. IDA then adds these data to its RMS database.

It is possible that the conclusions that can be drawn from these data, which are based in large measure on the de jure framework for business registration in the particular country, could be somewhat out of phase with the de facto reality of the business registration process. For example, the cost and time estimates that this method yields, based initially as it is on the de jure requirements of the applicable legal instrument, may not reflect the actual time spent and costs incurred in setting up a business in a given country by a specific business owner. Additional costs resulting from dysfunctions in the public administration (e.g., individual misfeasance, capabilities of staff, etc.) may not be captured fully in the analysis. The estimates are likely to be at the low end of the spectrum of possible actual costs incurred and time spent.

The limitations of the selected indicators have been presented to donors and other users of Private Sector Development data. Despite these acknowledged limitations, the team concludes that while the “Doing Business” Project uses an idealized model, the estimates that it produces are useful indicators of the climate towards business and a sound base for policy dialogue.

-- - Boor I Allen 1 Hamilton

5


P p___

111.1.2 Review of Outcome Procedures

As noted above, the two aggregate indicators are time and cost. Based on a review of the IDA 13 Replenishment Report (Additions to IDA Resources: Thirteenth Replenishment - Supporting Poverty Reductions Strategies, September 17, 2002), and the IDA-1 3 Results Measurement System: Spring 2004 Update dated May 10,2004, slight differences were found between the two reports regarding how the targets for the two indicators were articulated. The differences were:

The lDAl 3 Replenishment Report sets its target based on reduction of time (in number of business days) and formal cost required for business start-up by 7 percent whereas the lDAl 3 Results Measurement System: Spring 2004 Update, May 2004 report sets its target as a reduction in the population-weighted average of 7 percent. During meetings with IDA and World Bank technical staff, it was indicated that the understanding with Deputies in all technical discussions was that all outcome indicators would be population weighted4. The IDA1 3 Replenishment Report indicates reduction in formal cost of business start-up (in percent of GDP per capita) whereas the May 2004 report expresses it in Gross National Income (GNI) per capita. During meetings with IDA and World Bank technical staff, it was indicated that from the initial discussion of IDA targets and for consistency purpose, the indicators would be expressed in GNI per capita as shown in Annex C.

0

0

The time indicators are computed as follows5:

Step 1.

Step 2.

Step 3.

Step 4.

Step 5.

Report the population by country in 2001 and 2003, from other broadly used and accepted databases. Compute each country population weight in 2001, calculated by dividing the population of one country in 2001 by the total population of 39 IDA countries. The same method is used for computing country population weight in 2003. Report the number of days necessary to set up a business at end-2001 (baseline) and end-2003. Compute the population weighted average days spent in setting up a business by multiplying the number of days required to set up a business by the population weight for each of the 39 IDA countries and sum the results. Compute the percentage change in number of days between the two years’ weighted average.

A simple arithmetic averaging of the non-population-weighted number of days and the rate of change between the resulting average for each year were also computed to

The omission of the reference to “population weighted in the IDA13 Replenishment Report is 4

an error.

Illustrations of these steps are detailed in Annex C of this report. 5

- - Booz 1 Allen 1 Hamilton

6

IDA- 13 Results Measurement System: Progress and Proposals Inception Report P - - assess whether the weighting process influenced the achievement of results toward the target.

The computation of the IDA formal cost indicator was necessarily more involved for two reasons. First, the use of local currency units precludes the use of simple arithmetic averaging. Second, for comparability purposes, the costs are expressed in terms of a reference point, in this case the Gross National Income per capita (GNlpc). The cost indicator is computed as follows6:

Step 1. Step 2.

Step 3. Step 4.

Step 5.

Step 6.

Step 7.

Report the population by country in 2001 and 2003. Compute the population weight in the sample for 2001 and 2003, as computed for the time indicator. Report the GNlpc in 2001 and 2003 for each country’. Report the cost to set up a business at end-2001 and end-2003 in each of the 38 countries of the sample. Compute the 2002 and 2004 cost in terms of GNlpc for each country of the sample; this is calculated by dividing the 2002 cost by 2001 GNlpc and repeats the procedure for 2004. Compute the population-weighted cost in terms of GNlpc for 2002 and 2004 for each country of the sample and sum to obtain the population- weighted average for the 38 countries. Compute percent change in the average for each of the two years.

The team also conducted a straightforward arithmetic averaging, Le. non-population weighted, to assess whether the weighting process influenced the achievement of results toward the target.

111.2 Education

The Team conducted a comprehensive review of the procedures applied by the IDA staff in collecting the relevant education data by first reviewing the report IDA 13 Results - Measurement System: Spring 2004 Update. Following this review, we interviewed World Bank staff responsible for preparing the reports. The interviews reviewed the

Illustrations of these steps are available in Annex C of this report.,

The GNI for Zimbabwe was not available. Therefore, the sample for the cost indicator was, for all practical

6

7

purpose, reduced to 38 countries. - i - Booz 1 Allen 1 Hamilton

7

IDA- 13 Results Measurement System: Progress and Proposals

indicator 1: Increase the PCR from the baseline

sequence of procedures applied to determine changes in the Primary Completion Rate (PCR).

The PCR rate is calculated as the ratio of the total number of students enrolled (grade- specific enrollment) in the last year of primary school in a specific year to the total number of children of the Official Graduation Age in that year. It was determined that the data containing the total number of students enrolled in the last year of primary school was obtained from the UNESCO Institute of Statistics (US) and was supplemented by completion or enrollment data collected directly from national education systems by World Bank task teams. The age-specific populations projections maintained by the World Bank Health, Nutrition, and Population (HNP) department are used to calculate the total number of children in the Official Graduation Age population. The data sources identified collectively provide the most reliable and consistent sources of enrollment and population information available.

Two procedures were reviewed: (1 ) PCR Original Methodology and Updated Data; and (2) PCR Most-Recent-Trend Methodology and Updated Data. The specific results reviewed by the Team are shown in table 2 below.

Table 2: Outcome Targets and Reported Results - Education

Target (%) 69 69

Result (%) 70 73

INDICATORS

I

indicator 2: increase the number of countries with

Target (number of countries)

I MRT PROCEDURE

38 38

positive PCR growth rate Result (number of countries)

45 43

IDA used two procedures to calculate imputed values of PCR for years where data was missing. The first, labeled as the Original Methodology, imputed PCR values for years where no data were available for a specific year on the premise that the most recent observation from a previous period is the best approximation of the missing year and is therefore brought forward for the missing year. The second, labeled as the Most Recent Trend Methodology imputed PCR values for a missing year on the premise that the trend of an indicator remains the same unless a new observation indicates otherwise.

We reviewed two data sets, one for each procedure, containing PCR data for the years 1990, 2000, 2001, and 2002. Each data set was derived from the raw data included in the PCR tables shown in the IDA 73 Results - Measurement System: Spring 2004 Update Appendix B. The IDA analyzes the raw data using two somewhat different approaches for each of these indicators. These are: PCR Original Approach and Updated Data for Indicator Number One; PCR Most-Recent-Trend Approach and Updated Data for Indicator Number One; PCR Original Approach and Updated Data for

=r - Booz I Allen Hamilton

8

IDA- 13 Results Measurement System: Progress and Proposals Inception Report --- Indicator Number Two; and PCR Most-Recent-Trend Approach and Updated Data for Indicator Number Two'.

PCR Original Approach and Updated Data for Indicator Number One

To verify that the target of 69 percent was met, and that the actual result was 70 percent, we examined the following procedures and, where appropriate, calculated values as described in the PCR original approach methodology:

Step 1. Enrollment and repetition data in the year 2002 were obtained from data published by UIS supplemented by completion or enrollment data collected directly from national education systems by the World Bank task teams. For each country included in the analysis, we calculated the share (percentage) their population represents for the aggregate official graduation age population. The calculated share (percentage) for each country was multiplied by the reported Primary Completion Rate to obtain a weighted share.

The weighted share that each country comprised of the aggregate population was multiplied by the Primary Completion rate (percentage) for the year 2002.

The calculations obtained for each country in Step 2, were summed to obtain a weighted average aggregate percentage for all countries included in the analysis. The resulting percentage (rounded to the nearest percentage point) for the year 2002 was 70 percent. This was the same value as reported in Table 2 (page 6 of the IDA 13 Results - Measurement System: Spring 2004 Update report).

Step 2.

Step 3.

PCR Most-Recent-Trend Approach and Updated Data for Indicator Number One

To verify that the target of 69 percent was met, and that the actual result was 73 percent, we examined the following procedures and, where appropriate, calculated values as described in the PCR Most-Recent-Trend Methodology.

Step 1. For each country included in the analysis, we calculated the share (percentage) their population represents for the aggregate official graduation age population. The calculated share (percentage) for each country was multiplied by the reported Primary Completion Rate to obtain a weighted share.

Step 2. The weighted share that each country comprised of the aggregate population was multiplied by the Primary Completion rate (percentage) for the year 2002.

The calculation obtained for each country, as described in Step 2 was summed to obtain a weighted average percentage for all countries included in the analysis. The resulting percentage (rounded to the

Step 3.

~

See Appendix D for more details on education outcome procedures 8

Booz 1 Allen 1 Hamilton 9


d

nearest percentage) for year 2002 was 73 percent. This was the same value as reported in Table 3 (page 7 of the IDA 73 Results - Measurement System: Spring 2004 Update report).

PCR Oriqinal Approach and Updated Data for Indicator Number Two

To verify that the number of countries that had positive PCR Growth relative to 1990 was at least 38 and that the actual result in the year 2002 was 45, the following procedures were examined:

Step 1. For each country included in the analysis, we compared the 1990 PCR values to the 2002 PCR values, and counted those countries that had a positive change from 1990 to 2002. Countries that did not have PCR values for 1990 were excluded from the total count.

The calculation from Step 1 for each country was summed to obtain the total number of countries that had a positive PCR growth rate compared to 1990. The result for year 2002 was 45 countries. This was the same value as reported in Table 2 (page 6 of the IDA 73 Results - Measurement System: Spring 2004 Update report).

PCR Most-Recent-Trend Approach and Updated Data for Indicator Number Two

To verify that the number of countries that had positive PCR Growth relative to 1990 was at least 38 and that the actual result was 43, we examined the following procedures:

Step 2.

Step 1.

Step 2.

For each country included in the analysis, we compared the 1990 PCR values to the 2002 PCR values, and counted those countries that had a positive change from 1990 to 2002. Countries that did not have PCR values for 1990 were excluded from the total count. In addition, countries with 1990 PCR values in excess of one hundred were also excluded from the calculation.

The calculation from Step 1 for each country was summed to obtain the total number of countries that had a positive PCR growth rate compared to 1990. The result for year 2002 was 43 countries. This was the same value as reported in Table 3 (page 7 of the IDA73 Results - Measurement System: Spring 2004 Update report).

As shown in Table 2, applying the second (Most Recent Trend) methodology increases the population weighed PCR average from 70 percent to 73 percent, but reduces the number of countries with positive growth rates from 45 to 43. From a technical perspective, each methodology has strengths and limitations. For this reason, we agree that the results applying both approaches should be shown. Increasing completion rates is a long-term process. Therefore, one should not expect extensive changes during the time frame for which data was collected. Nonetheless, the results, using either the first or the second methodology, indicate progress toward the objective of increasing the proportion of the population completing primary education.

- P



We duplicated, for each of the two methodologies the procedures applied by IDA to derive the results and found that IDA complied with the procedures described in the May 10,2004 IDA1 3 Results report.

111.3 Health

IDA used measles immunization rate to measure progress in the health sector because of the relationship of measles to child mortality and poverty. Because of the statistical methods used by the World Health Organization (WHO) and The United Nations Children’s Fund (UNICEF), estimates of the measles immunization rate are generally available annually. Additionally, measles immunization reports tend to have a high reporting frequency.

To review the procedures used by the IDA staff in assembling the relevant data, the Team conducted an extensive interview with the IDA staff member who performed the analysis and a thorough review of the relevant report (IDA1 3 Results-Measurement System: Spring 2004 Update). We determined, that two sets of data were used because the time lag for the collection of measles data has been shortened from three to two years. The first data set was used to produce the results that relate to the intended time frame upon which the IDA13 targets were set (1 999 - 2001). The second data set was used to relate to the year 2002, expanding the set of relevant data to a three-year period (1 999-2002). The data used for the measles immunization coverage rate are compiled by WHO and UNICEF from surveys conducted by national and international agencies and administrative reports of national health programs.

The Team examined the procedures employed to produce the 2001 and 2002 results for targets set forth for the two indicatorsg. To determine whether or not the targets set forth for the first indicator were met, the Team reviewed the procedure by first, assessing the data. Based on the data assessment, the number of observations used was for 70” countries. Moreover, the results were calculated as follows:

See Appendix E for more details on health outcome procedures lo Dominica was not included in this observation as shown in Annex E - - -

Booz 1 Allen 1 Hamilton 11

IDA- 13 Results Measurement System: Progress and Proposals Inception Report - - -

Step 1. Take the percentage rate of measles immunization for children under 12 months for each of these 70 countries and multiply it by the estimated birth for each of these respective countries. This provided the estimated number of children who were immunized for each of these 70 countries.

Step 2. Take the estimated number of children who were immunized and sum these up for all the countries.

Step 3.

Step 4.

Sum estimated birth for all countries.

Take the results from Step 2 and divide it by the results from Step 3 to get the results.

The approach used to examine the 80 percent coverage of measles vaccination to 29 countries was to extract and sum all countries with an 80 percent or greater coverage rate. As reported in the IDA13 Results Measurement System: Spring 2004 Update, the results were based on 71 countries. We used the same approach to assess whether the outcome target was met. The Team found that the procedures applied by IDA to derive the target 1 and target 2 results were reliable because we were able to duplicate and reproduce these results as shown in IDA1 3 Results-Measurement System: Spring 2004 Update report.

IV. RESULTS

IV.l Private Sector Development

Based on our review of the data described above, the Team found that the IDA1 3 targets have been met: the number of days required to start a business has fallen by 12 percent over the two-year period, and the cost to start a business has declined by 19 percent. Table 3 presents a summary of our findings:

Booz ] Allen I Hamilton 12

IDA- 13 Results Measurement System: Progress and Proposals Inception Report - P

_a P - P



P -

Reduce Time Required for Business Start-up (in numbers of business days)

Reduce Formal Cost of Business Start up (in Percent of GDP per capita)

Table 3: Verified Outcome Results - Private Sector Development

7

7

IN Dl CATO RS

Indicator One: Increase the PCR from the baseline

TARGET (Yo)

Target (%) 69 69

Result (%) 70 73

Indicator Two: Increase the number of countries with

IV.2 Education

38 I 38 I Target (Number of countries)

END 2001 TO END 2003

Result (Number of countries)

Reported (%) I Verified (%)

45 43

12

Verified (Number of countries)

12

45 43

19 19

94

Based on our review of the data described above, Booz Allen finds that the results in all instances met the PCR targets set by IDA as part of IDA1 3. Table 4 presents a summary of our findings:

Table 4: Verified Outcome Results - Education

MRT PROCEDURE 1 P:ZL?E$E I IN DI CATO RS

I Verified (“7’0) I 70 I 73

positive PCR growth rate

IV.3 Health

For the first indicator, Booz Allen verified that there was a 61 percent coverage rate for year 2001 and a 65 percent coverage rate for year 2002. The most recent data provided by IDA were used to compute these numbers.

To verify that there were 27 countries with 80 percent coverage rate for year 2001 and 29 countries with 80 percent coverage rate for year 2002, we extracted from the data all the countries that had 80 percent or greater coverage of measles vaccination. The Team - P -

Booz ] Allen Hamilton 14

IDA-1 3 Results Measurement System: Progress and Proposals Inception Report - - - .__

found that there were 27 countries in year 2001 that had 80 percent or greater coverage rate and for year 2002, there were 29 countries with 80 percent or greater coverage rate.

i INDICATORS i I

Table 5 presents a summary of our findings.

TARGETS 2001 2002

Reported Verified Reported Verified (”/I

Table 5. Verified Outcome Results: Health

Number of Countries with 80% Coverage

29 27 27 29 29

~

Increase in Population-Weighted Coverage Rate of Measles

60 61 61 65 ~

65

V. SUMMARY OF FINDINGS

The Team thoroughly reviewed the procedures used to measure the input and the outcome targets as set forth in the IDA1 3. Our overall findings are:

first, on an overall basis, we find that the IDA met its target objectives with respect to both the input indicator and outcome indicator targets as set forth in the IDA1 3 replenishment resolution.

Second, specifically with respect to input indicator targets, we find that IDA produced and reported the number of identified analytical work reports set forth in the IDA1 3 replenishment resolution.

processes used to produce, collect and evaluate the underlying data that support the outcome results are calculated within the limits of the system to lead to an accurate measurement of outcome results, and that those processes were followed by IDA during the IDA1 3 Spring 2004 review.

the outcome results met and in some cases exceeded the target objective.

Third, specifically with respect to outcome indicatortargets, we find that the IDA

Fourth, again specifically with respect to outcome indicator targets, we find that



Annexes

- P

ANNEX A

IDA SPRING 2004 EVALUATION AND MEASUREMENT SYSTEM TARGETS

1.

0

0

0

0

0

2.

Input Indicators

A total o f 40 Country Financial Accountability Assessments completed, of which at least 50 percent for African countries; A total of 38 Country Procurement Assessment Reviews completed, of which at least 50 percent for African countries; A total of 40 Public Expenditure Reviews completed, of which at least 50 percent for African countries; All Country Assistance Strategies prepared since July 2002 underpinned by . current poverty analysis; and A total of 14 Investment Climate Assessments completed.

Outcome Indicators

Education: 0 Increase population-weighted average primary completion rate to 69 percent with

a substantial number of countries reaching a higher rate; and Increase number of countries with positive growth rates in primary completion rates to 38 countries.

Health: 0

0

Increase overall coverage rate (population-weighted) of measles immunization to 60 percent, with a substantial number of countries reaching a higher rate; and Increase number of countries with 80% coverage of measles vaccination to 29 countries.

Private Sector Development: 0

o

Reduce time required for business start-up (in number of business days) by 7 percent from end-2001 ; and Reduce formal cost of business start-up (in percent of GDP per capita) by 7 percent from end-2001.

- - _.__ - Booz 1 Allen I Hamilton

A - 1

IDA-13 Results Measurement System: Progress and Proposals Annexes

ANNEX B

INTERVIEWS WITH WORLD BANK STAFF

Task Manager:

Rui Coutinho Off ice of the Vice President Operations, Policy and Country Services

rcoutinho @worldbank.orq 202-473-2254

Private Sector Development:

Axel Peuker investment Climate, IFC

apeuker @ worldbank.orq 202-473-8676

Caralee McLiesh Monitoring, Analysis and Policy Unit, World Bank

[email protected] 202-473-8676

Education

Eric Valdeman Swanson Program manager Development Economics Development Data Group

eswanson @worldbank.orq 202-473-3832

Changqing Sun Economist Development Economics Data Group

csunl @worldbank.orq 202-458-8226

Health

Eric Valdeman Swanson Program Manager Development Economics Data Group

eswanson @ worldbank.orq 202-473-3832

Changqing Sun Economist Development Economics Data Group 202-458-8226

B - 1

IDA-1 3 Results Measurement System: Progress and Proposals Inception Report

Annexes

- P

csunl @worldbank.orq

IDA Results Measurement System

Pilar Maisterra Resource Mobilization Department

pmaisterra @ worldbank.orq 202-473-9629

Akihiko Nishio Phone: 202-458-2552 anishio @ worldbank.orq

Economic and Sector WorWOperations Policy and Country Services:

James Stephens Delivery Management

Sstephens @worldbank.orq 202-473-1 869

Olga Jonas Country Services

[email protected] 202-473-4655

Booz I Allen I Hamilton B - 2

0 X u Z z a

e E

8

art;

m Q

4-.

$ 2 3

- C

C

- L C E C

- r C C C

- C C

C C

- C C C 0 r T

- (5: (5:

- m m

5 3 7

5 n

- r d

-

IDA- 13 Results Measurement System: Progress and Proposal Inception Report

Annexes

~ -

Included Included Total number of Total number Total number No. Of in 71 in 79 Total Crude measles of measles of measles

vaccination vaccination vaccination Country Name 2001 2002 countries observed country country Birth list list 1999 200 1 2002

ANNEX E

HEALTH DATA

No, of countries w,

IDA- I 3 Results Measurement System: Progress and Proposal Inception Report

Annexes

Total Countries t 70

- P

Booz 1 Allen I Hamilton E - 2

IDA- 13 Results Measurement System: Progress and Proposal Annexes Inception Report I

E____

Total population-weight coverage rate for 2001

P - 1_p

Booz I Allen 1 Hamilton E - 3

60.73%


Annexes

ANNEX F

INPUT INDICATORS DATA

Region Country Verification Report Number

I Country Financial Accountability Assessment Date of Date Delivered to Report IDA Client

Booz 1 Allen I Hamilton F - 1

I D A - 13 Results Measurement System: Progress and Proposal Inception Report

Annexes

- - P

_._ - Booz I Allen I Hamilton

F - 2

IDA-1 3 Results Measurement System: Progress and Proposal Inception Report

Annexes

- - P

- - - Booz I Allen I Hamilton

F - 3

IDA-13 Results Measurement System: Progress and Proposal Inception Report

Annexes

MNA Djibouti 511 7/04 Feb-04 Dec-03 Yemen, Republic of 511 7/04 Dec-00 Dec-00

LCR IBolivia 511 7/04 Nov-001 May-01

P P P P

Booz I Allen I Hamilton F - 4

Dominica Nicaragua

511 7/04 Jun-03 Jun-03 511 7/04 Report No. 25918-Nl Jan-04 Jun-03


Annexes

P

Public Exoenditures Reviews

Report No. Nigeria 5/17/04 2571 0-UNI States Finances Study

Public Expediture Review: lmprovinc Budget Management for Poverty

Public Expenditure Performance in Africa Region Rwanda: Evidence from a Public Working Paper Expenditure Tracking Study in the

Rwanda 5/17/04 Series No. 45 Health and Education Sectors The United Rewblic of ReDOrt No.

Report No. Chad 5/17/04 2461 2 Reduction

ITanzania 1 5/17/04 123812-TA /Public Expenditure Review /Public Expenditure Review FY02:

Dec-03 Oct-03

Dec-01 Dec-01

Sep-03 Jun-03

Apr-03 Jul-02

Jun-02 Jul-02

Mar-03 Jul-02

Oct-01 Ap r-0 1

Ma -02 A r-02 -t- Jun-03 1 Jun-03

Jun-03 1 May-03 I

P

* Booz 1 Allen I Hamilton -

F-5


Annexes

c_-- 4

511 7/04

511 7/04

511 7/04

511 7/04

5/17/04

The Republic

Zambia

Public Expenditure Review 2003 Report No. Supporting Budget Reforms at the 271 35-UG Central and Local Government Levels Sep-03

Public Expenditure Review: Public Report No. Expenditure, Growth and Poverty: A 22543-214 Synthesis Dec-01

Report No. Public Expenditure Management and 261 62-ZA Financial Accountability Review Nov-03

Report No. 25437-IND Selected Fiscal Issues in a New Era Feb-03

Report No. 261 91 -IND Report Jun-03

Decentralizing Indonesia: A Regional Public Expenditure Review Overview

Enhancing Service Delivery through

EAP Indonesia +

511 7/04

5/17/04

5/17/04

511 7/04

Indonesia

~

Report No. 24434-AM Public Expenditure Review Apr-03 Report No. 25233-AZ Public Expenditure Review Apr-03

From Aid Dependency to Fiscal Seif- Report Reliance: A Public Expenditure and No.24297-BIH Institutional Review October, 2002 Report No. 2291 3-GE Public Expenditure Review NOV-02

Mongolia

Papua New Guinea

SAR Bangladesh

Maldives

Pakistan

Pakistan

Armenia ---I--- Bosnia and

Improved Resource Allocation and Institutional Reform: Integrated Fiduciary Assessment and Public

Public Expenditure Review: Country Financial Accountability Assessment: Joint Report of World Bank, International Monetary Fund and

Public Expenditure and Financial

Report No. 5/17/04 2561 1-KH Expenditure Review

Report No. 5/17/04 24443-LA Asian Development Bank

Sep-03

Jun-02

May-03

Sep-01

Jun-03

Dec-02

Mar-03

May-03

Feb-02

Jun-02

Jun-03

Jun-02

May-02

Sep-00

May-03

Feb-01

Jun-02

Dec-02

Jun-02

Jun-02

- - - =

Booz 1 Allen I Hamilton F - 6

IDA-13 Results Measurement System: Progress and Proposal Inception Report

Annexes

P - - I_p_ - - Public Expenditure Review: (In Two Volumes) Volume I: Main Report Fiscal Policies for Growth and Poverty Reduction Public Economic Management Review Public Exuenditure and Institutional

I I I

Mar-04 Mar-03

Mar-03 Jun-02

Review (in Three Volumes) Volume 1 : Executive Summary Feb-03 JuI-02

Jublic Expenditure Review public Expenditure Management for 'overty Reduction and Fiscal Sustainability Jublic Expenditure Review: mproving the Poverty Focus of Public

3ECS Fiscal Issues: Dominica. 'olicies to Achieve Fiscal Sustainability and Improve Efficiency md Equity of Public Expenditures

3 pending

- - Booz 1 Allen I Hamilton

F - 7

August 20,2002 NOV-02

Jun-01 Apr-01

Dec-01 NOV-01

Feb-04 Feb-04

IDA-1 3 Results Measurement System: Progress and Proposal Inception Report

Annexes

__.__

Country Assistance Strategies Reviews

All Country Assistance Strategies Prepared since July 2002 Underpinned by Current Poverty Analysis I I I

PRSP I Poverty Other poverty CAS Presented to the Board after 07/01/02 I Assessment analysis I I

Booz 1 Allen 1 Hamilton F - 8


Annexes

ANNEX G

REFERENCES AND BIBLIOGRAPHY

Publications :

Performance Management in IDA, International Development Association, IDA 13, April 2002

IDA 73 Results-Measurement System: Spring 2004 Update, International Development Association, IDA1 4, May1 0, 2004

IDA Results Measurement System: Progress and Proposals, April 3, 2003

IDA Results Measurement System: Update Note, Operations Policy and Country Services, October 29,2003

Doing Business Project, Entry Regulations Survey, World Bank, 2004

Country Financial Accountability Assessments, World Bank, 2001 -2004

Country Procurement Assessment Reviews, World Bank, 2001 -2004

Public Expenditure Reviews, World Bank, 2001 -2004

Investment Climate Assessments, World Bank, 2001 -2004

Poverty Analyses and Country Assistance Strategies, 2001 -2004

In addition to the above publications, the below web site links and tables were reviewed:

Web Site Links:

Links to Public Expenditure Reviews Delivered to IDNBlend Countries under IDA1 3

Links to Investment CIimate Assessments Delivered to IDNBlend Countries under IDA1 3

Tables:

Private Sector Development: IDA PSD Trigger Updates 2, Monitoring, Analysis and Policy Unit, World Bank, May 2004

Education: WDl Download, World Bank Estimates, Development Economics Development Data Group, World Bank, April 1, 2004

Health: WDI Download, World Bank Estimates, Development Economics Development Data Group, World Bank, April 1, 2004

Booz Allen I Hamilton G - 1

THE WORLD BANWIFWM.1.G.A.

OFFICE MEMORANDUM DATE:

TO:

FROM:

EXTENSION:

SUBJECT:

June 3,2004

Mr. Chander M

Messrs. Jeffrey

8400 1

sudev, Chairman, COD

stein (MDF) and She

IDA13 Results Targets: Expert Panel Review - Inception Report Comments from Bank Manapement

Following the arrangements for an Expert Panel Review o f achievements made towards IDAl 3 Results Targets which were approved by the Executive Directors on April 29,2004 (“Proposal for an Expert Panel Review o f Achievements Made Towards IDA13 Results Targets,” IDA/R2004-0083, April 16,2004), Bank Management engaged the consulting firm o f Booz, Allen, Hamilton (“the Consultants”) to provide a team o f experts to carry out the f irst step under the Terms o f Reference for the Expert Panel Review (“the TOR”). This note presents comments from Bank Management on the inception report prepared by the Consultants (“IDA Results Measurement System: Spring 2004 Update”).

Background

As recorded in the Report to the Board by the Committee on Development Effectiveness (CODE), CODE2004-0027, April 16,2004, the origin o f the review i s a request from the U.S. chair, prompted by U.S. legislation approved in February 2004. Management was advised by the US. authorities that this legislation anticipated that the Bank would provide for an independent entity, such as a private auditing firm, to conduct and make publicly available an extemal performance audit verifying whether IDAl 3 spring 2004 results targets have been met. The results o f the review would then be made available to determine whether a US$200 mi l l ion tranche o f supplementary financing could be released for IDA.

The approved arrangements include not only the verification exercise motivated by the U.S. request, but also a review o f longer-term technical issues related to results measurement. The review is being carried out by a panel o f experts, consisting o f experts provided by a private consulting firm and two or three experts in social statistics or related fields. The first step has just been completed by the Consultants and consists o f an inception report which verifies whether the targets have been met, and whether the procedures used by IDA staff in assembling relevant data are reliable. The second step will be carried out by individual experts, involving preparation o f a final report which builds on the inception report and makes technical recommendations towards improving the measurement and tracking o f the indicators under the results measurement framework for IDA14. The recommendations from the second step could provide useful inputs into the MDBs’ harmonization agenda.

-2- June 3,2004

Preparation of the Inception Report

The Consultants were chosen under the Bank procedure called “Selection Based on Consultants’ Qualifications.’’ A multi-disciplinary team o f six experts was mobilized by the Consultants. In accordance with the TOR, the Consultants carried out the work on the basis o f a desk review and interviews with IDA staff. The Consultant team and IDA staff discussed a wide range o f relevant issues, such as the detailed procedures needed to compute primary completion rates. The Consultants started their work on May 17 and completed the report on June 2. Details on staffing, procedures and conclusions can be found in the inception report.

The Inception Report

The key conclusions o f the inception report are the following:

0 the target objectives were met with respect to both input and outcome indicators; and

0 the procedures used by IDA to produce, collect and evaluate data, and those used to report progress on indicators, were found to be sound and accurate.

Bank Management agrees with the overall conclusions o f the inception report. In accordance with the TOR, Bank Management had an opportunity to review the report at a draft stage, and provided a limited number o f technical corrections, which have been incorporated in the attached version o f the report.

The Consultants also provided some comments on possible alternative targets for outcome indicators concerning private sector development, including use o f simple (rather than population-weighted) averages and modification o f the set o f countries to reduce the impact o f populous countries on the result. Bank Management fel t that this analysis extended beyond the TOR for the f i rs t step, but will be provided as valuable inputs to the individual experts for the second step o f the review.

Next Steps

In accordance with the TOR, the next step would be for CODE to consider the attached report, and report to the Board on i ts discussion. After CODE consideration, the report will be published, along with a summary o f CODE’S deliberations.

Documents

Booz I Allensiteresources.worldbank.org/IDA/Resources/IDA13_Results_Inception... · Booz I Allen 1 Hamilton 6002 Allen Hamilton Inc. 8283 Greensboro Drive McLean, VA 22102 Tel 1-703-902-5000