exploring innovations & solutions x real-time policy-making

Arianna Legovini

Development Impact Evaluation (DIME)

World Bank

1 Legovini

What does dime do?

• Evaluate programs at scale to answer policy-

relevant questions

• Build capacity to strengthen country

institutions for evidence-based policymaking

• Build theory in strategic development areas

based on high-quality evidence

Improve Bank’s operations and country policies

2

How does DIME work?

Assess

demand

Identify priority research questions

Define best identification & data generation

strategy

Support implementation

Collect high-quality data

Analyze

results

Disseminate results to direct

client

Disseminate to global

communities of practice

Inform policy design

Guide policy implementation,

make mid-course

corrections

Provide policy feedback to

inform adoption and scale-up

Syst

em

atic

use

of evid

ence

Workshops & clinics with

operational staff and government

clients

Joint research/gov teams, field coordination, research team

visits

Workshops, seminars, papers, briefs, community

of practice

Capacity

build

ing

3

DIME Impact Evaluations % of DIME IE Portfolio

Legovini 4

Discuss

1. Elements of success in policy

design

2. Using impact evaluation (IE) to drive

decisions and results

5 Legovini

What is the Impact of Public

Policy & Investments?

IT DEPENDS

Do people know what services are available?

Do they value them? Use them?

What is their quality?

…

6

Focus on the problem

• what policy do we need ?

• the returns are not fixed

• how to make policy work ?

1. get the delivery right

2. get people to do it

7 Legovini

How to

• Behavioral biases

• limited attention, self-control, cognitive capacity, understanding, asymmetric valuation of gains and losses

• Delivery mechanisms

• targeting rules, centralized/decentralized modality of public service delivery, collective vs. individual or peer-to-peer, private sector vs. public sector delivery, paid vs. voluntary, use of media and technology

• Accountability mechanisms

• top-down accountability (audits, inspections, supervision, performance assessment and feedback, laws and regulations)

• demand-side accountability (information, report cards, user participation and monitoring).

• Incentives

• supply or demand conditional monetary and non-monetary incentives

• Constraints to capital formation and productivity

• credit, financial, cash-flow, risk-management,

• financial literacy and life skills,

• information, inputs,

• institutional, legal, tax burden, corruption, property, public goods,

• managerial, skills, technology, and transaction costs.

Legovini 9

Testing WHAT and HOW

to

We evaluate the effectiveness of packages of interventions (“what”)

and experiment with mechanisms (“why/how”) to make them work

better:

Legovini 10

Legovini 11

SMALL ACTS CAN HAVE

BIG IMPACTS

Reminders increase HIV

adherence by 35%

Legovini 12

How effective are the billions of dollars going to HIV

treatment when 40% of patients adhere to treatment?

With (costless) SMS reminders 53% of patients adhere to

treatment and increase effectiveness by 35%!

Pre-commitment

increases investment

Legovini

60

73.2

0

10

20

30

40

50

60

70

80

Control Precommitment savings

Precommitment and use of agricultural inputs

13

Automatization

increases TAX compliance

without

Automatization

Less compliance

With

Automatization

More compliance

14 Legovini

Default

doubles take up

49%

86%

40%

45%

50%

55%

60%

65%

70%

75%

80%

85%

90%

Before After

Control

Treatment

Legovini 15 [Madrian and Shea, 2001]

Public recognition

increases performance

6.96

7.807 8.177

14.502

4

6

8

10

12

14

16

Sales

Legovini 16 [Ashraf et al., 2011]

What is the POINT?

USE IE EVIDENCE TO INFORM POLICY DESIGN

USE IMPACT EVALUATION TO TEST IF IT WORKS IN SPECIFIC SECTOR/ COUNTRY CONTEXT

17 Legovini

Presentation

1. Elements of success in policy design

2. Using impact evaluation (IE) to

drive decisions and results

18 Legovini

Improve results by learning

19

Prescription-based policy

Evidence-based policy

We know We

learn

Legovini

FOCUS ON the Causal Links

Inputs

• Financial and human resources

Activities

• Actions to convert inputs into outputs

Outputs

• Tangible products

Intermediate outcomes

• Effective access

• Use

• Compliance

• Service quality

Final Outcomes

• Growth

• Lives saved

• Productivity

20

WHAT (SUPPLY SIDE) HOW TO (DEMAND RESPONSE) & RESULTS

Legovini

HOW DO WE TEST? 21

monitoring & Reporting:

trends and correlations

• DO WE NEED MONITORING? YES!

• Monitoring tracks indicators over time

• But only among participants

• It is descriptive before-after analysis

• It tells us whether things are moving in the

right direction

• It does not tell us why things happen 22 Legovini

23 Legovini

Invest in the cause to

get more results

• It is easy to confuse correlation with causation

• By investing in the real causes we get more results

Small business growth

SME training

“Selection”

Entrepreneurial SME owners get more credit and

training

24 Legovini

Impact Evaluation:

causality

• Tracks mean outcomes over time in

• the treatment group relative to

• the control group

• Compares

• what DID happen with

• what WOULD HAVE happened (counterfactual)

• Identifies causal output-outcome link

• separately from the effect of other time-varying

factors

25 Legovini

Counterfactual criteria

Treated & control groups

• Have identical initial average characteristics

(observed and unobserved)

• The only difference is the treatment

• Therefore the only reason for the difference

in outcomes is due to the treatment

26 Legovini

Importance to plan ahead: prospective evaluation

Example SME credit

• What determines loan take up?

• Test different offers (with more or less choice)

• What determines loan repayment?

• Test different risk assessment processes

• What determines SME growth?

• Test whether business services improve credit outcomes

Legovini 27

How to define treatment & control

and measure impact?

• Randomly assign the intervention or different

elements of it to different SMEs to obtain

comparable treatment and control groups

• Use clear measures of outcome (take-up,

repayment, growth)

• Collect data on treatment and control groups

• Compare mean outcomes over time

28 Legovini

identify the elements

that cause more impact

Legovini 29

SME credit/training

programs

Advertise the terms of one

loan

Risk assessment type 1

Business advisory services

Risk assessment type 2

Advertise the terms of different

loan options

Max loan take-up

Max loan repayments

Max SME growth

Max IMPACT

Why iE is important to do?

• Know what improves results

• Take better decisions

• Demonstrate to others

• Invest responsibly/make better

fiscal choices

30 Legovini

CONCLUSIONs

1. Use IE evidence now to improve policy

design

2. Use IE to drive decisions and results

31 Legovini

DIME resources

fb

http://www.facebook.com/IEknow

web

http://www.worldbank.org/dime

blog

http://blogs.worldbank.org/impactevaluations

data

http://microdata.worldbank.org/index.php/catalog/impact_evaluation 32