Setting the framework: Designing interventions to achieve impact

Esteban Diez RouxPatricia Yanez PagansInter-American Development Bank

June 22nd 2015

Setting the framework: Designing

interventions to achieve impact

• Impact Evaluations and the Transport Sector

• Main Issues in Designing IEs for Transport

Effectiveness: What works and what doesn’t work?

Why do we evaluate?

Efficiency: Most efficient way to achieve an outcome?

Evidence-based policy making

What do we want to achieve with our projects?Is it more than reducing transportation costs?Traditional outcomes in transport:– Travel times– Vehicle operating costs

Non-traditional outcomes? – New development agenda– EI’s could be key to understand our impact on

development – But…how far can we get with our investments?– Attribution critical

How can we design projects that are more effective in achieving these non-traditional outcomes and impacts?

First step…

Understand what works and what doesn’t work

How?Impact evaluation!

Attribution: Let’s be careful in getting the right answers

• Traditional approaches in transport:

– Before-and-after methods– Ex-post cost-benefit analysis

• Limitations of these methods?

Impact evaluations complement other approaches

They complement process evaluations and ex-post cost benefit analyses

However, different evaluation instruments answer slightly different questions

8

Process evaluation

Did we achieve all products at the planned costs ?Did outcome indicators move in the right direction?

Ex-post cost-benefit analysis

Did project economic benefits exceed project costs?

Impact evaluation

What would be the situation of beneficiaries without the project?

Learning, accountability, and complementarities

9

Process evaluation

Generates information about costs and trends in indicators. Helps for project accountability.

Ex–post cost-benefit analysisExtracts information of costs and benefits from monitoring. Ideally extracts benefits information from IE. Justifies investments based on their economic return.

Impact evaluationMainly for learning, helps to understand what strategies are more effective. Provides information on actual benefits for economic analysis.

Using instruments in isolation might leave important open questions

Process evaluation

Are there any other external factors that influenced results?

Ex-post cost-benefit analysis

What are the real costs and benefits of the project?

Impact evaluation

Is the program cost-effective?

Setting the framework: Designing

interventions to achieve impact

• Impact Evaluations and the Transport Sector

• Main Issues in Designing IEs for Transport

Designing impact evaluations (IEs) for the transport sector

• Causality or attribution• Mechanisms• Spillovers

=> Need a clear understanding of the theory of change

The theory of change is key for the design of projects and for IEs

Resources at the disposal of the project

Actions taken to convert inputs into outputs

Tangible goods and services project activities produce

Results likely to be achieved once beneficiaries use outputs

Final project goals

StaffBudget

ConstructionRehabilitation

Km of roads paved

Reduction in travel timesReduction in operation costs

Poverty reduction

Inputs Activities Outputs Outcomes Impacts

=> IE questions?

Impact evaluation needed where program attribution is unclear

Resources at the disposal of the project

Actions taken to convert inputs into outputs

Tangible goods and services project activities produce

Results likely to be achieved once beneficiaries use outputs

Final project goals

StaffBudget

ConstructionRehabilitation

Km of roads pavedKm of new roads

Reduction in travel timesOperation costsIncreased mobility?

Poverty reductionIncreased productivity

Under project control

Inputs Activities Outputs Outcomes Impacts

Household level outcomes

Firm level outcomes

Community/Regional level outcomes

What have we learned about non-traditional outcomes in transport?

Picture: Horia Varlan

Some aspects we have learned about household-level outcomes

Road improvement increases…

Household investments in durable assets (Mexico - Gonzales-Navarro and Quintana-Domeneque, 2012)

Changes household production decisions(Peru - Valdivia, 2009)

Use of health and education services (Bangladesh - Khandker et al., 2009)

Some aspects we have learned about firm-level outcomes

Expansion rail system

Changes hiring decisions of firms located close to stations (US - Holzer et al. 2003)

Improved highways

Helps firms reduce stock of input inventories and re-optimize suppliers choice (India - Datta, 2012)

Some aspects we have learned about community & regional level outcomes

Rehabilitation of rural roads

Reduces local prices (Sierra Leone - Casaburi et al., 2013)

Railway construction

Increases wage gaps through increased specialization (Central Europe - Walker, 2015)

Electronic toll collection

Reduces vehicle emissions improving overall child health (US -Currie and Walker, 2011)

Attribution: When compared to other instruments only impact evaluation seeks for causality

Process evaluationNO attribution

Ex-post cost-benefit analysisNO attribution

Impact evaluationYES attribution

How do we seek for causality with an impact evaluation?

What would have happened to our beneficiaries in the absence of the

program?

=> Find the right counterfactual (comparison group)

• Roads and public transportation systems are not randomly placed

• Benefits are difussed, implications on how we define the zone of intervention

• Impacts may take a long time to emerge

IEs in transport can be challenging…

Mechanisms: We usually assume that roads reduce poverty, but how?

Povertyreduction

New employment opportunities?

Market prices?

Increased production?

Access to public services?

Picture by: Klaus Leidorf

Others?

Mechanisms: Distribution of impacts and complementarities

• Are benefits different for men and women?

• Are benefits different for poor and less poor populations?

• How do transport investments interact with other interventions?

Picture: Bob Miller

Picture: Bob Miller

Spillovers: Are we having positive/negative impacts on non-beneficiaries?

Picture: Rakesh Rocky

Spillovers 1: Project externalities

Benefits or costs that are transferred directly from beneficiaries to non-beneficiaries

• Positive? Reducing travel times for users of BRT systems may reduce travel time of non-users in less congested roads

• Negative? Relocation of economic activity closer to highways may have negative impacts on areas further away

Spillovers 2: General equilibrium effects

Program effects are transmitted to the local economy

• Rehabilitation of rural roads reduces local prices (Sierra Leone - Casaburi et al., 2013)

• Railway construction reduces price gaps between regions, but increased specialization increases wage gaps (Central Europe - Walker, 2015)

Spillovers 3: Interactions

Non-beneficiaries may be affected by the program through social and economic

interactions with beneficiaries

• Road safety education programs might have impacts beyond those that receive education

• ICT messages that promote savings might affect the savings decisions of others (friends, family, neighbors)

Spillovers 4: Behavioral effects

Program affects behavioral or social norms within the area of the intervention affecting

both beneficiaries and non-beneficiaries.

• Awareness campaigns to promote safe driving in some neighborhoods might change social norms in a city

• ICTs to promote transparency in some governments offices might promote a general culture of transparency

How to successfully implement impact evaluations?

• Lots of discussion and consensus between key stakeholders

• Continued financial support • Strong local capacity for monitoring and

implementation• Local ownership to answer relevant questions and

guarantee success

Picture: Steve Davis

Key messages

1) Find the right ways to identify causality

2) Design IEs that allow understanding mechanisms and spillovers

3) Work together to guarantee success

Let’s start!

THANK YOU