Experiments and BeyondLocal Economy-wide Impact Evaluation

J. Edward Taylor

University of California, Davis

Agricultural & Resource Economics

Berlin, November 6, 2015

Consider a Social Cash Transfer (SCT) Program

• Cash payments to ultra-poor households

• Inspired by Mexico’s PROGRESA program

• Eligibility: Asset poor, labor poor, other criteria (e.g., orphans and vulnerable children)

• Conditionality: Kids’ enrollment in schools, clinics (PROGRESA)• Most Africa SCTs do not have “hard” conditionalities

Cash Transfer Programs in Sub Saharan

Africa (19 in 13 Countries)

• Malawi SCT* – Mchinji pilot, 2008-2009– Expansion, 2013-2014

• Kenya*– CT OVC, Pilot 2007-2011– CT OVC, Expansion, 2012-2014– HSNP, Pilot 2010-2012

• Mozambique PSA– Expansion, 2008-2009

• Zambia*– Monze pilot, 2007-2010– Child Grant, 2010-2013

• South Africa CSG– Retrospective, 2010– 2012

• Ethiopia*

– PNSP, 2006-2010

– Tigray SPP, 2012-2014

• Ghana LEAP*

– Pilot, 2010-2012

• Lesotho, CGP*

– Pilot, 2011-2013

• Uganda, SAGE

– Pilot, 2012-2014

• Zimbabwe, SCT*

– Pilot, 2013-2015

• Tanzania, TASAF

– Pilot, 2009-2012

– Expansion, 2012-2014

• Niger

– Begun in 2012

* Local-economy (LEWIE) included in evaluation; see http://www.fao.org/economic/ptop/home/en/

The Experiment

• (1) Identify the eligible households • Proxy means tests (PMT), community-based assessment (CBA)

• (2) Conduct a baseline survey with variables of interest to evaluation

• (3) Randomly select treatment and control villages

• (4) Roll out treatment to treatment villages (withhold from controls)

• (5) Conduct follow-on surveys

• (6) Compare outcomes of interest in treatment and control villages

An Illustrative Example (Outcome = Income)

• We might be able to say the SCT program caused the $30 change in income• That is, that the average effect of the treatment on the treated (ATT) is 30

• Two critical conditions to establish causality in this project:• Treatment and control villages are random (randomized control trial, RCT)• There are no spillovers between treatment and control villages (control-group

contamination)• Not a good idea to draw treatment and control groups from same village

• External validity: Nonstarter without knowing why the outcomes happened

Monthly

IncomeDifference

Eligible Households

Before 100

After 140

Treated Villages

40

Monthly

IncomeDifference

100

110

Control Villages

10

Difference in Difference 30

SCT

Rest of Zimbabwe

Eligible Household in Treated

Village

$

$ $

$

Feedback on the Treated?$

Control Village?

$

$

Spillovers to Ineligibles

We Could Use an Experiment to Explore Spillovers

• Angelucci and DiGiorgi, Mexico’s PROGRESA (2009)

• In practice, almost no project does this

Monthly

IncomeDifference

Eligible Households

Before 100

After 140

Treated Villages

40

Difference in Difference

Monthly

IncomeDifference

100

110

Control Villages

10

30

Ineligible Households

Before 200

After 22020

200

21010

Difference in Difference 10Could mean a big total spillover (if there are many ineligible households)

Where Do SCT Income Spillovers Come From?

• Cash raises purchasing power of beneficiary households

• … thus demand in the local economy

• … supply must rise to meet this demand (otherwise, inflation)

• Economic impacts depend critically on the supply response• If local supplies expand to meet demand, SCTs can have a multiplier effect on local

income

• Otherwise the program could be inflationary

• Different kinds of interventions can produce different kinds of spillovers (including non-economic ones)

What Is Local Economy-wide Impact Evaluation (LEWIE)?

• A Simulation Approach, Complement to• Econometric models• RCTs

• Precedents:• Social accounting matrix (SAM) analysis• Computable general equilibrium models

• Model of How “Treated” Economies Work• Micro-actors (beneficiaries, non-beneficiaries)

interact within a LEWIE model

• Construct with micro-survey data• Econometrics estimates give confidence

• Can simulate different policy interventions

Steps to Doing (or Commissioning) a LEWIE• Scoping mission: structure of local economy

• Production activities• How do local markets work?

• Markets convey impacts through local economies

• Modify baseline surveys for LEWIE• Include both eligible and ineligible households• Data to econometrically estimate household expenditure, production functions• The ‘where’ question• May have to supplement household surveys with business surveys

• Construct models of eligible and ineligible households• Draws from rich experience constructing agricultural household models• Integrate these into LEWIE model of local economy (LEWIE) (GE modeling tools)

• Use the LEWIE model to simulate impacts of intervention, test sensitivity to model assumptions, obtain confidence bounds on simulation results

• Optional: Explore complementary interventions• Typically have LEWIE results 4-6 months after baseline survey is complete

Why Is LEWIE Important?

• Lets us identify total local economic impacts, including spillovers• These are part of the cost-benefit analysis of the project

• It’s an option when experiments are not feasible• There is no treatment and control group

• Politics, ethics, implementation challenges, cost

• It provides explanations for why impacts happen• For policy, it’s not enough to know what the impacts are

• We need to learn why and how to influence them

• We want to spot and deal with possible negative impacts

• Timing: LEWIE can be done before the project is implemented• …and inform project design

LEWIE Estimates of SCT Income Multipliers in Seven African Countries*• Cash transfers to

the poor create local income multipliers

• A dollar transferred to a poor household raises local income by significantly more than a dollar

* FAO, Protection to Production (PtoP) Project; http://www.fao.org/economic/ptop/home/en/

Impact

“LEAP has had a positive impact on local economic growth. Beneficiaries spend about 80 percent of their income on the local economy. Every GH1 transferred to a beneficiary has the potential of increasing the local economy by GH2.50.”

- President John Dramani Mahama, opening the Pan-African Conference on Inequalities, April 2014

Where Do We Find the Largest Spillovers?

• High there is high reliance on local supplies of goods and services

• Where the local supply response is high• Or complementary interventions make supply response more elastic• Real impacts are lower in more labor, capital, land, liquidity constrained

contexts

• Increase in demand translates into increase in supply for local goods and services• Prices transmit impacts

• More price action in local economies isolated from outside economies, more nontradables

• More integrated with outside markets, more widely we have to cast our net to find impacts

Most LEAP SCT Spillovers Go to Non-beneficiary Households

0

0.5

1

1.5

2

2.5

Total Beneficiary

households

Non-beneficiary

households

Nominal

Spillover

Transfer

0

0.5

1

1.5

2

2.5

Total Beneficiary households Non-beneficiary

households

Real

FAO, Protection to Production (PtoP) Project; http://www.fao.org/economic/ptop/home/en/

• Many non-beneficiaries are poor, just not poor enough to quality

• Fail to meet other eligibility criteria (LEAP: OVC, elderly, disabled)

A Diversity of LEWIEsCountry Impact of… References

Tigray Social Cash Transfer Pilot Program* Kagin, et al. (2014); Davis, et al. (2016)

Evaluation of economy-wide impacts of

Productive Safety-Nets Program (PSNP)In Progress, 3ie-IFPRI

Cost-benefit Analysis of SCTs In Progress, UNICEF

Galapagos Impacts of Ecotourism Taylor, et al. (2009), (2003), IADB

Livelihood Empowerment Against Poverty

Program (LEAP)*Davis, et al. (2016)

Rural Income Transfers Compared Filipski and Taylor (2012), UNICEF

Child Grants Program (CGP)* Filipski, et al. (2015); Davis, et al. (2016)

Experimental Validation of LEWIE Results In Progress, FAO-UNICEF

Cash Transfer Program for Orphans and

Vulnerable Children (CT-OVC)*Thome, et al. (2013); Taylor, et al. (2013)

Hunger Safety Nets Programme Phase 2 In Progress (UKAID)

Social Cash Transfer Program (SCT)* Thome, et al. (2014); Davis, et al. (2016)

Fertilizer Input Subsidy Program (FISP) Thome, Taylor and Filipski (2014), SOAS

Rural Income Transfers Compared Filipski and Taylor (2012), UNICEF

Migrant Remittances Taylor and Filipski (2014) Hewlett

Maize Price Shocks Dyer and Taylor (2011)

Corruption in PROCAMPO Delivery Taylor and Filipski (2014)

Morocco Saffron Price Shocks, Gender Taylor and Filipski (2014), CIMMYT

Philippines Regulation of Small-scale Fisheries In Progress, Packard

Rwanda Cash Programs in Refugee Camps In Progress, WFP

Tanzania Technology Change in Cotton Production Kagin, et al. (2015), Gatsby

Zambia Child Grant Program (CGP)* Thome, et al. (2014)

ZimbabweHarmonized Social Cash Transfer Program

(HSCT)*Taylor, et al. (2014)

* FAO Protection to Production (PtoP) Program, in collaboration with UNICEF

Ghana

Kenya

Malawi

Mexico

Ethiopia

Lesotho

Beyond Experiments to Answer Big Questions(Mostly not “RCTizable”)

• When poor women get cash, what happens to the local economy?• Do non-beneficiaries (most of whom are likely to be poor) benefit? (Africa)

• Should the WFP shift from food aid to cash in refugee camps?• Good for the refugees? Good for the host country? (Rwanda, Uganda)

• How can we make small-scale fisheries sustainable without adversely affecting fishing communities in the short-run? (Philippines)

• What happens when food price shocks hit poor rural economies? (C. Am)

• Are migrant remittances good for migrant-sending communities? (Mexico)

• If we make poor farmers more productive, are there benefits for the poor villages in which they live? (Tanzania, Ethiopia)

• Can eco-tourism development alleviate poverty in poor regions• …even though tourists rarely buy things from poor people? (Galapagos, Lat America)

• Which are better, cash transfers or fertilizer subsidies? (Malawi)

Each Evaluation Method Has Its Challenges and Drawbacks• Experiments

• Feasibility, ethics, politics, timing• Avoiding control group contamination (control villages beyond influence of the treatment• Keeping control over the experiment and implementation bias

• Harder with big real-world projects, vs. academic ones• Will people’s answers determine whether they are eligible?

• Generalizing: Does the RCT give insight on what scaled-up programs would do?• Tradeoff between internal and external validity

• Understanding why impacts happen• RCTs test whether something works, not why

• Findings not available until some time after the project is implemented

• Econometrics & “Quasi-experimental Methods”• Finding convincing instruments to control for selection bias; the identification problem

LEWIE

• Getting the model right• Understand how local economy works• Structural modeling draws from economic theory

• Rich literature on agricultural household models, imperfect markets

• Need good baseline data to estimate model parameters• Overlap with experiment data needs

• Validation• Confidence intervals on parameter estimates• Tests for structural form• Monte-Carlo method to put confidence intervals around simulation results

• Big advantage in LEWIE: the budget constraint• People spend their income• Most spend it close to home• Potential for large local-economy spillovers and multipliers

• Rich history in economics of modeling expenditures, production, GE linkages

In Vivo in Vitro in Silico• Shift in scientific research from in vivo/vitro to in silico methods*

• Rational drug design, human brain project, climate models

• LEWIE is in silico (RCTs are in vivo; econ lab experiments are in vitro)

• Like other in silico methods, it can benefit from experiments • to obtain better parameter estimates• update parameters that change as a result of a treatment• validate simulation findings

• Contextual variation in how local economies work, but commonalities• The budget constraint, supply and demand, role of prices• Every $1 of SCT goes somewhere; similarities in similar contexts, often predictable

across contexts• Clear procedures to estimate expenditure and production functions

* Witness a sharp increase in usage of ‘in silico’ contrasted with declining use of ‘in vivo’ and ‘in vitro’ on the Google books Ngram viewer. A goal of the EU Human Brain Project is “establish in silico experimentation as a foundational methodology for understanding the brain.”

Learn by Integrating LEWIE with Experiments (A New Gold Standard?)

• LEWIE offers structural explanations for experimental findings

• Findings much sooner than experiments (do not have to wait for follow-on surveys to simulate impacts)

• Experiments can be used ex post to validate, update LEWIE findings

• Example: Lesotho’s CGP• Experimental data showed real-life multipliers are bigger than LEWIE

simulated• Likely: CGP changed the structure of the economy in ways not reflected in the

model• Local economy less constrained than we assumed• Plenty of room to expand livestock in Lesotho• New markets, consumption behavior, behavioral spillovers

Parting Thoughts: From Evaluation to Policy• We see LEWIE as an increasingly important part of the evaluator’s tool kit

• Governments and donors want to know impacts beyond the treated• Including spillovers to ineligible households• We can’t affort to miss impacts of development interventions

• Part of the cost-benefit analysis

• Impacts on the treated include local-economy feedbacks• The economic impacts of many projects are likely to significantly exceed the direct

impacts• (Often) good news for finance ministers and donors

• Structural: Focus on understanding why impacts happen; learn from evaluations

• Want to inform policy • Why impacts happen, and how to influence them

• E.g., productive interventions to enhance impacts of SCTs• LEWIE is a laboratory in which we can explore this

• Biggest impact of LEWIE: Changes the way people think about how their programs create impacts