Impact Evaluation for Real Time Decision Making

DIME – FRAGILE STATESDUBAI, MAY 31 – JUNE 4

Impact Evaluation for Real Time Decision MakingArianna LegoviniHead, Development Impact Evaluation Initiative (DIME)World Bank

FIRST What is a results chain?

Inputs• Financ

ial and human resources

Activities• Action

s to convert inputs into outputs

Outputs• Tangib

le products, • infor

mation campaigns,

• trainings,

• studies

Inter-mediate outcomes• Use of

outputs by intended population• Take

-up• Use

Final Outcomes• Object

ive of the program• Gro

wth• Soci

al cohesion

• Employment status 2

Implementation (SUPPLY SIDE) Results (DEMAND RESPONSE)

Example of a results chain

Inputs• Financi

al and human resources to build local institutions

Activities• Develo

p com-munication

• Set up grant mechanisms

• Provide technical assistance to communities

Outputs• Comm

unica-tion package

• Grant offerings

• Technical assistance visits

Intermediate outcomes• Grants

awarded and disbursed

• Projects implemented

• Community rallies

Final Outcomes• Reduce

d conflict

• Improved inclusion of ex-combatants

• Expanded employment

3

Monitor whether program is implemented as planned Evaluate whether program is effective

?

The results chain

Developmental hypothesis that helps you define: what you are doing and for what

purpose what needs to be monitored and what needs to be evaluated

4

What is monitoring?

Monitoring tracks indicators over time (in the treatment group)

It is descriptive before-after analysis

It tells us whether things are moving in the right direction

5

What is Impact Evaluation? Impact evaluation tracks outcomes

over time in the treatment group relative to a control group

It measures the effect of an intervention on outcomes relative to a counterfactual what would have happened without it?

It identifies the causal output-outcome link separately from the effect of other time-

varying factors

Monitoring & Impact Evaluation Use monitoring to track implementation efficiency (input-output)

7

INPUTS OUTCOMESOUTPUTS

MONITOR EFFICIENCY

EVALUATE EFFECTIVENESS

$UPPLY

DEMAND RESPONSE

Use impact evaluation to measure effectiveness (output-outcome)

Pick the right method to answer your questions

Descriptive Analy

sis

Monitoring (and process evaluation)• Is

program being implemented efficiently?

• Is program targeting the right population?

• Are outcomes moving in the right direction?

Causal

Analysis

Impact Evaluation• What

was the effect of the program on outcomes?

• How do alternative implementation modalities compare?

• Is the program cost-effective?

8

Discuss among yourselves (5m)What happens if you use monitoring to evaluate impact?

9

Discuss among yourselves (5m)What happens if you use monitoring to evaluate impact?

You get the wrong answer……100% of the times 10

AfterBefore

B

A

t0 t1

B

Intervention

Change

C Impact

NEXT: Do we know ex ante… On Community Driven Development, what

information will get communities to respond? facilitation will results in high quality proposals? rules will increase inclusion in the decision-making

process? monitoring mechanisms and co-payments will

improve local projects and their use of funds? On Disarmament, Demobilization and

Reintegration, are community based or targeted approaches most

effective? should we try to delink combatants from units or

build on unit cohesion? is including or excluding command structures most

effective?

11

Trial and error

We turn to our best judgment for guidance and pick an information campaign, a package of services, a structure of incentives.

Is there any other campaign, package, incentive structure that will do better?

12

The decision process is complexA few big decisions are taken during

design but many more decisions are taken during roll out & implementation

13

DesignEarly roll outImplementation

Pick up the ball:What is a results tree?A results tree is a representation of

the set of results chains that are considered viable during program design or program restructuring.

Is a set of competing policy and operational alternatives to reach a specific objective.

14

Example of a decision tree for a combatant reintegration program

ReintegrationPublic communication campaign to reintegrate ex-combatantsUse combatant unit cohesion

structuresInclude command structuresExclude command structures

Delink combatant and dismantle unit structureSupply-based training

Demand-based trainingCommunity-based campaignUse combatant unit cohesion

structuresInclude command structuresExclude command structures

Delink combatant and dismantle unit structureSupply-based training

Demand-based training

15

How to select between plausible alternatives?Establish which decisions will be

taken upfront and which will be tested during roll-out

Experimentally test critical nodes: measure the impact of one option relative to another or to no intervention

Pick better and discard worse during implementation

Cannot learn everything at onceSelect carefully what you want to

test by involving all relevant partners

16

Walk along the decision tree to get more results out of a reintegration program

17• Take-upMAX • Use of servicesMAX • Benefits from

servicesMAX

ReintegrationPublic communication campaign to reintegrate ex-combatantsUse combatant unit cohesion structures

Include command structuresExclude command structures

Delink combatant and dismantle unit structureSupply-based training

Demand-based trainingCommunity-based campaign

Use combatant unit cohesion structuresInclude command structuresExclude command structures

Delink combatant and dismantle unit structureSupply-based training

Demand-based training

H0w IE can support you

18

Discuss among yourselves (5m)How many times do you make

changes to your program on the feeling that something is not working right?

How useful would it be to know for sure which ways are best?

19

Why evaluate?

Improve quality of programs Test alternatives and inform design in real

time Increase program effectiveness Answer the “so what” questions

Build government institutions for evidence-based policy-making Plan for implementation of options not solutions Find out what alternatives work best Adopt better way of doing business and taking

decisions 20

The market for evidence

21

PM/Presidency:Communicate to constituencies

Treasury/Finance:Allocate budget

Line ministries:Deliver

programs and negotiate budget

Cost-effectiveness of different programs

Effects of government

program

BUDGET

SERVICE DELIVERY

CAMPAIGNPROMISES

Accountability

Cost-effectiveness of alternatives and effect of sector

programs

Shifting Program ParadigmFrom: Program is a set of activities designed to deliver expected results

Program will either deliver or notTo: Program is menu of alternatives with a

learning strategy to find out which work bestChange programs overtime to deliver

more results 22

Shifting Evaluation Paradigm

From retrospective, external, independent evaluation Top down Determine whether program worked or not

To prospective, internal, and operationally driven impact evaluation /externally validated Set program learning agenda bottom up Consider plausible implementation alternatives Test scientifically and adopt best Just-in-time advice to improve effectiveness

of program over time 23

Retrospective (designed & evaluated ex-

post) vs. Prospective (designed ex-ante and evaluated ex-post)

Retrospective impact evaluation: Collecting data after the event you don’t know

how participants and nonparticipants compared before the program started

Have to try and disentangle why the project was implemented where and when it was, after the event

Prospective impact evaluation: design the evaluation to answer the question

you need to answer collect the data you will need later Ensure analytical validity 24

24

Is this a one shot analytical product? Evaluative process to provide useful (actionable)

information at each step of the impact evaluation

25

Discuss among yourselves (5m)

What are some of the entry points in policy-making cycles?

What are some of the answers you would like to have?

26

Ethical considerations

It is not ethical to deny benefits to something that is available and we know works HIV medicine proven to prolong life

It is ethical to test interventions before scale up if we don’t know if it works and whether it has unforeseen consequences Food aid may destroy local markets create

perverse incentives Most times we use opportunities created

by roll out and budget constraints to evaluate so as to minimize ethical considerations AND

We can always and should test alternatives to maximize out results

27

Questions? Comments?

28

0

50

100

150

200

250

300

350

2004 2005 2006 2007 2008 2009 2010

Num

ber o

f ong

oing

IE

Evolution of IE in the Bank

SAR

MNA

LAC

ECA

EAP

AFR