Embed Size (px)
Citation preview
GOING BEYOND EFFECTIVENESS:
USING IMPACT EVALUATION TO
IMPROVE PROGRAM DESIGN
Impact Evaluation Workshop
Debre Zeit, Ethiopia
Reimar Macaranas
December 9 – 12, 2014
NETWORKING ACTIVITY
Impact Evaluation Workshop
Debre Zeit, Ethiopia
Reimar Macaranas
December 9 – 12, 2014
Getting to know your fellow participants…
1. Find someone you don’t know
(and yes World Bank staff count)
2. Learn that person’s name and how they got their name
3. Share where you work and a project or accomplishment
you are passionate about or proud of
4. Share a funny or interesting fact about yourself
GOING BEYOND EFFECTIVENESS:
USING IMPACT EVALUATION TO
IMPROVE PROGRAM DESIGN
Impact Evaluation Workshop
Debre Zeit, Ethiopia
Reimar Macaranas
December 9 – 12, 2014
Using Impact Evaluations to Improve Program Design
Objective:
Impact Evaluation can be used as a tool for informed and
evidence-base decision making.
Example: Designing Conditional Cash Transfer Programs
Going Beyond Effectiveness
5
Did it work?
How did it work?
Why did it work?
Which design
feature worked
best?
6
Public Works Program
Household Assets
Building Program
Ethiopia’s Productive Safety Net Program
7
Public Works + Asset
Building Program
vs
Public Works Program
Question: What Worked Best?
8
General Findings:
The Program Matters
Food Security
Livestock Assets
Asset Shedding
Biggest gains are found when combining the
programs
Ethiopia’s Productive Safety Net Program
9
Over a 12 Month Period
Beneficiaries with Both Programs (compared to PW only):
1.53 months of additional food security
147 kg of increased output of grain
per household per year
Ethiopia’s Productive Safety Net Program
Using Impact Evaluations to Improve Program Design
Objective:
Impact Evaluation can be used as a tool for informed and
evidence-base decision making.
We can learn how to design programs to make them more effective
Conditional Cash Transfers
Source: World Bank, Fizbein & Schady 2009
Conditional Cash Transfers
Source: World Bank, Fizbein & Schady 2009
Tanzania Social Action Fund
Tanzania Social Action Fund
March 2014: Doubled enrollment to 100,000 poor
households
2014-2016: Expected enrollment of 900,000 poor
households
5.5 million citizens
Program Design Questions
What types of questions related to program design do we
need to be asking?
(This is the audience participation portion of the presentation)
Program Design Questions
What types of questions related to program design do we
need to be asking?
When?
Who?
How much?
How often?
How to transfer?
Learning About Program Design
Conditional Cash Transfers
What should the size of the transfer be?
Who should receive the transfer?
Whom to target?
How to target?
When to distribute the money?
Should it be conditional?
Conditional on what?
What should the size of the transfer be?
The size required to improve attendance is small, possibly only
2-4% of income.
Increasing the size of the transfer does not increase the impact on
attendance
School Attendance - Percentage Point Increase
25 25
8 8
0
10
20
30
$45 $60 $5 $15
Cambodia Malawi
Source: Filmer and Schady (2009), Baird et al., (2010)
When to distribute the money?
• Distribute portion of the money when parents have to
pay for enrollment
Makes it easier for parents to save.
• Distribute portion of money when students reach
university
Incentive to perform better in school.
19
Timing matters.
Timing Matters
20
Bogota, Colombia - Conditional Subsidies for School
Attendance program
Treatment Groups
Standard CCT
Design
Savings Treatment
Tertiary Treatment
• 2/3 Monthly Payment
• Remaining funds
given at enrollment
• Same total cost
• 2/3 Monthly Payment
• Remaining funds
given at enrollment
• Large subsidy for
tertiary
Timing Matters
Savings Treatment vs. Standard CCT Design
21
3.3 3.6
0
1
2
3
4
5
Secondary Tertiary
School Enrollment - Percentage Point Increase
The enrollment gain is larger if money is dispersed at the time where
parents have to pay for enrolment.
Source: Barrera-Osorio et al., (2008)
Timing Matters
Tertiary Treatment vs. Standard CCT Design
22
School Enrollment - Percentage Point Increase
The enrollment gain is larger if money is dispersed at the time where
parents have to pay for enrolment.
3.3
46
0
10
20
30
40
50
Secondary Tertiary
Source: Barrera-Osorio et al., (2008)
Should it be conditional?
Conditions good for increasing attendance and
enrollment.
However, unconditional transfers are better at reducing
drop-outs for teenagers.
23
Conditions should depend on the age and gender of
the pupil.
Should it be conditional?
Conditional Cash Transfer vs Control Group
24
School Attendance - Percentage Point Increase
8 9.2
0
2
4
6
8
10
2009 2010
Conditions good for increasing attendance and enrollment
Source: Baird et al., (2010)
Should it be conditional?
Unconditional Cash Transfer vs Control Group
25
Marriage & Pregnancy - Percentage Point Decrease
Conditions not as good for reducing main sources for female
drop-out rates
8.6 7.7
0
2
4
6
8
10
Marriage Rate Pregnancy Rate
Source: Baird et al., (2010)
26
Using Impact Evaluation for Program Design
Solutions Within Reach
Explore – Ask the right research
questions
Test – Employ rigorous methods
Improve – Go beyond effectiveness
27
Using Impact Evaluation for Program Design
Solutions Within Reach
Thank You!
