Upload
hector-cross
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
USING EVIDENCE TO INFORM POLICY:IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS
Overseas Development Institute
May 4, 2010
ARIEL FISZBEINCHIEF ECONOMIST, HUMAN DEVELOPMENTWORLD BANK
Motivation
‘Since it is difficult to distinguish the good from the bad prophet, we must be suspicious of all prophets: it is better to avoid revealed truths, even if we feel exalted by their simplicity and splendor, even if we find them comfortable because they come at no cost. It is better to be content with more modest and less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated'’ Primo Levi
2
How can we learn from experience?
Traditional approach: identify best practice cases and inductively derive conclusions.
But the strength of those conclusions is directly related to the internal and external validity of the underlying evaluation.
We could do better if we plan well and take a more systematic approach to learning from experience.
3
The evaluation problem
Impact: difference in outcome with and without program. Cannot observe simultaneously. with and without comparisons deceptive if
participation depends on individual attributes that influence outcomes
before/after comparisons can be misleading if other things happened during the period
Need to estimate counterfactual (control or comparison) Treated & control have same characteristics on
average, Only reason for difference in outcomes is the
intervention
4
Multiple design options
Experimental (often using phase in) Quasi-experiments/non-experimental
Regression Discontinuity (RD) Difference in difference – panel data Other (Instrumental Variables,
matching, etc) In all cases, these will involve
knowing the rule for assigning treatment
Advantages of prospective design
5
Evolution of IE in the World Bank6
...Completed before 2004
…Completed from 2004
…Ongoing in 2010
0
25
50
75
100
125
150
175
200
18
58
175
Number of Ongoing IEs at the World Bank…
Strong focus on HD areas7
Social Protec-tion20%
Health17%
Education13%ECD & Nutr.
7%
Other9%
Gov .4%
Urban Upgr.6%
Agric.6%
Fin. & Priv. Sec.8%
Local Dev.10%
The World Bank Impact Evaluation Program: Thematic Areas
HD thematic ar-eas: 58% of total World Bank IEs
Strategic themes8
Education, Other
Pay For Performance in Health
Malaria
HIV
Early Childhood Development
Active Labor Market Programs
School Accountability
Conditional Cash Transfers
0 5 10 15 20 25
6
9
9
13
14
16
19
20
The World Bank Impact Evaluation program on Human Development
Number of OngoingImpact Evaluations
Thematic areas
… and IEs of non-HD projects often measure impact on HD outcomes:
Agricultural Development Indicators: anthropometrics
Housing Indicators: morbidity, anthropometrics
Rural Electrification Indicators: study time, respiratory illnesses
Rural Roads Indicators: school attendance
Urban Upgrading Indicators: employment
Water Supply and Sanitation Indicators: morbidity, anthropometrics
9
Building a global evidence base on the impact of HD programs
10
Europe & Central Asia
Middle East & North Africa
East Asia & the Pacific
Latin America & Caribbean
South Asia
Africa
0 10 20 30 40 50 60
1
3
10
17
21
54
The World Bank Impact Evaluation program on Human Development
Number of OngoingImpact Evaluations
Regions
Priority thematic areas
1. Conditional Cash Transfers2. Paying for Performance in Health (P4P)3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth
Employment6. HIV/AIDS Prevention7. Early Childhood Development
11
12
CCTs have become very popular….
13
CCTs have become very popular….
First generation evaluation questions14
Impact on consumption and poverty Side effects?
Impacts on service utilization
Impacts on human development outcomes
Impacts on consumption and poverty
Mexico(1999)
Nicaragua(2002)
Colombia(2006)
Honduras(2002)
Average transfer (% of per capita consumption)
20% 30% 13% 11%
Impact on per capita consumption (%)
8.3%** 20.6%** 10%** 7%*
Impact: headcount index (% points)
1.3** 5.3** 2.9* --
Impact: poverty gap (% points)
3.0** 9.0** 7.0** 2.0*
Impact: sqd. poverty gap (% points)
3.4** 8.6** 2.2** 2.0*
15
CCT impacts on food consumption16
Impacts on school enrollment
Age range Baseline enrollment
Impact (% points)
Size of transfer
Colombia 8-1314-17
91.7%63.2%
2.1**5.6***
17%
Chile 6-15 60.7% 7.5*** 3-7%
Ecuador 6-17 75.2% 10.3*** 10%
Mexico Grade 0-5Grade 6
Grade 7-9
94.0%45.0%42.5%
1.98.7***
0.620%
Nicaragua 7-13 72.0% 12.8*** 30%
Cambodia Grade 7-9 65.0% 31.3*** 2%
Bangladesh 11-18 44.1% 12.0** 1%
Pakistan 10-14 29.0% 11.1*** 3%
17
Impacts on health service utilization
Age range
Baseline level
Impact (% points)
Size of transfer
Colombia <24 months24-48
months
n.a.
n.a.
22.8**
33.2***
17%
Chile 0-6 years 17.6% 2.4 7%
Ecuador 3-7 years n.a. 2.7 10%
Honduras 0-3 years 44.0% 20.2*** 9%
Mexico 0-3 years 69.8% 8.4 20%
Nicaragua 0-3 55.4% 13.1* 27%
18
Impacts on education and health outcomes
Only mixed success in terms of improving final outcomes in education and health:
Education: Increases in school enrollment and years of completed schooling have not come hand-in-hand with improved learning outcomes Health: Some programs, but my no means all, have improved child nutrition (as measured by height-for-age, hemoglobin status)
19
20
Second generation evaluation questions
Role of conditions: Testing CCTs vs. UCTs (Burkina Faso, Malawi, Morocco and Yemen)
Does it matter who receives the cash? Mothers vs. fathers (Burkina Faso, Morocco and Yemen); Girls vs. parents in Malawi
Design of payments: Cambodia; Colombia
Priority thematic areas
1. Conditional Cash Transfers2. Paying For Performance in Health3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth
Employment6. HIV/AIDS Prevention7. Early Childhood Development
21
Results-based financing (RBF) ≈ Pay-for-performance (P4P)
Provision of payment for the
attainment of well-defined results
Transfer of money or material goods conditional on taking a measureable action or achieving a predetermined
performance target (CGD, 2009)
DonorCentral governmentLocal governmentPrivate insurer
$Service providersFacilities / NGOsCentral governmentLocal governments
RBF takes many forms…
Payers Payees
22
Impact evaluation of P4P in Health
Effects on non-contracted services provided?
Impact on equity of services?
Impact on out-of-pocket expenditures?
Impact Evaluations:
Will the introduction of pay for performance for patient services impact the quality and performance of hospitals/health centers?
23
24
P4P in Rwanda
National P4P scheme to supplement input-based budgets with bonus payments based on the quantity and quality of maternal and child healthcare (14 output indicators)
25
“Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010)
Impact on proportion of institutional deliveries
36.3
49.7
34.9
55.6
30.0
40.0
50.0
60.0
Baseline (2006) Follow up (2008)
Prop
ortio
n of o
f ins
tituti
onal
deliv
erie
s
Control facilities Treatment (PBF facilities)
7.3 % increasedue to PBF
26
“Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010)
Impact on quality of prenatal care
-0.10
0
-0.13
0.15
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
0.20
Baseline (2006) Follow up (2008)
Stan
dard
ized
Pren
atal
effo
rt sc
ore
Control facilities Treatment (PBF facilities)
15 % Standard deviation increase due to PBF
27
P4P in Rwanda: Lessons
No effect on the number of prenatal care visits or on immunization rates. Need demand side incentives?
Greatest effect on services that had the highest payment rates and are under the provider control.
Financial performance incentives can improve both use of and quality of health services.
An equal amount of financial resources without the incentives would not have achieved the same gain in outcomes.
28
P4P experiment in UgandaLundberg, Marek and Pariyo (2007) Contracting for Primary Health Care in Uganda
Three arms: (1) performance bonus, (2) freedom to allocate base grant, (3) control.
Performance bonus based on meeting targets (pre-natal care, attended births, immunizations, etc.)
Results after 18 months: bonus facilities perform the same as
or worse than others; freedom-to-allocate facilities perform
significantly better than others.
29
P4P experiment in Uganda
This performance bonus didn’t work.Amounts not large enough? Scheme too complicated? Not enough time? (some evidence of
learning curve)
Facilities may allocate budgets more effectively than the Ministry of Health
Priority thematic areas
1. Conditional Cash Transfers2. Paying For Performance in Health3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth
Employment6. HIV/AIDS Prevention7. Early Childhood Development
30
Strategies for improving learning outcomes
Supply side policies (classrooms, textbooks, extra tutors, teacher training)
Demand side policies (CCTs?) Child endowments/readiness to
learn -- ECD, de-worming Accountability reforms: Change the
environment in which decisions are made
31
WDR 2004 framework of accountability
32
33
Rationale for Education Accountability Schemes
How do we transform this teacher….
…into this teacher?
School Accountability34
35
3 Modalities on School Accountability
1. School-based Management (SBM)
2. Information for accountability projects
3. Teacher contracting and pay for performance projects
36
How can SBM improve outcomes?
Those at the local level have more or better information and different objectivesKey decisions about school personnelKey decisions about spendingChanges in the educational processResource mobilizationChanges in the school climate
37
School Based Management in Mexico
(Apoyo a la Gestión Escolar, AGE)
Grants + training to Parent Associations (PA).
AGEs increase school autonomy through improved mechanisms for participation of directors, teachers and PA in the management of the schools.
Effects on: School failure, repetition, drop out
Use phasing in of program for evaluation
38
School Base Management in Mexico
Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”
39
School Base Management in Mexico
Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”
40
School Base Management in Mexico
Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”
41
3 Modalities on School Accountability
1. School-based Management
2. Information for accountability (IFA)projects
3. Teacher contracting and pay for performance projects
How could IFA improve outcomes?42
Empower parents and communities vis a vis local & school authorities – Increased importance of education in the political
sphere– Increased attention to results valued by parents
Enable better technocratic decision-making• Help guide allocation decisions
Under “strong accountability” systems:– Threat of sanction will spur increased teaching
effort When there is school choice:
– “Market-pressure” for performance
Newspaper Campaign in Uganda
– In 1995 only 13 % of non-wage recurrent spending reached schools.
– From 1996-2001, amounts and dates of grant to local governments published in local and national newspapers.
43
Impact of information campaign
Source: Reinikka and Svensson (2001), Reinikka and Svensson (2003a)
IMPACT ON LEAKAGE:A school close to a newspaper outlet experienced 25 percentage points less leakage compared to a school one standard-deviation (30 km) further away from a newspaper outlet.
IMPACT ON LEARNING:A one standard deviation reduction (30 km) in distance lead to a 0.09 standard deviation increase in average test scores.
44
45
Information Campaigns in India
Village-level education committees (VECs), representing parents and village leaders, created by state education policy
Presumption that information campaigns are needed to empower parents/communities
Two sets of experiments PRATHAM --Banerjee et al. (2008) “Pitfalls of
participatory programs: Evidence from a Randomized Evaluation in Education in India”
Three states in India –WB + Nike Foundation --Pandey et al. (2008) Community Participation in Public Schools: Impact of Information Campaigns in three Indian States.
46
Impact of Information Campaign (Banerjee et al.)
Program: Advocacy and information about VECs =(1); (1)+teaching how “learning reports cards” are done=(2); (2)+ Teaching: Capacity Building to Directly Improve
Learning (reading classes)=( 3)
Information campaigns ((1) and (2)) had no effect on VEC activism or on learning outcomes
Trained teaching volunteers (3) did have a significant effect on “reading” learning outcomes
47
Impact of Information Campaign (Pandey et al.)
REGION Teacher Attendan
ce
Teacher
Activity
% Receiv
ed Unifor
m
Participation rate of parents in school inspections
Scholarships
Students received
Improvement in
Reading
UP 11 % Unchanged
18 % (girls)
.11 33 %(general caste)
27 %(Grade 3)
MP Unchanged 30% 14 % (general caste)
.02 26 %(SC/ SC)
14 %(Grade 3)
Karnataka
Unchanged Unchanged
N/A Unchanged N/A 15 %(Grade 4)
Pandey, Priyanka et al. Community Participation in Public Schools: Impact of Information Campaigns in three Indian States. South Asia Human Development. Presentation. World Bank. 2008.
47
48
3 Modalities on School Accountability
1. School-based Management
2. Information for accountability projects
3. Teacher contracting and pay for performance projects
49
How could TI improve outcomes?
Accreditation and merit-based incentives lead to better teaching professionals joining the system.
Decentralized hiring and monitoring of teacher performance leads to higher attendance rates.
Career advancement incentives stimulate sustained effort, improving student performance.
Pay-for-performance incentivizes improved quality and quantity of teaching improving student test scores.
Teacher Pay Incentives in Andra Pradesh (Muralidharan and Sundararaman, 2008)
50
Three policies Group and individual pay incentives based on
average improvement in test scores of all students Block grants to schools (extra teachers, more inputs) Assessments conducted by an independent NGO
Results (two rounds) Incentives increase student performance by 0.22 SD 1st year both incentives work well. 2nd year individual
incentives work better Additional inputs improve scores only by 0.08 SD Teachers liked the program
Priority thematic areas
1. Conditional Cash Transfers2. Paying For Performance in Health3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth
Employment6. HIV/AIDS7. Early Childhood Development
51
52
Look for more information….
www.worldbank.org/hdchiefeconomist
..and remember Primo Levi
‘It is better to be content with more modest and less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated'’
53