Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation

Are Disasters Any Different?

Challenges and Opportunitiesfor Post-Disaster Impact Evaluation

Alison Buttenheim, Princeton UniversityHoward White, 3ie

Rizwana Siddiqui, PIDEKatie Hsih, Princeton University

April 1, 2009Cairo

2

3ie post-disaster impact evaluation (PDIE) study

Motivation:

• Frequency and severity of natural disasters

• Quantity of assistance provided in post-disaster settings

• Recent interest from humanitarian and development sectors in more and better impact evaluation

• Opportunity to use Pakistan ERRA experience as case study

3

3ie post-disaster impact evaluation (PDIE) study

Goals:

• Review existing approaches to PDIE

• Develop a framework for rigorous PDIE

• Apply framework to the 2005 Pakistan earthquake case

• Identify a set of principles to guide PDIE

4

Disasters

Natural events:414 reported in 2007

(CRED criteria)

5

Disasters


(CRED criteria)

Human consequences:211 million affected

16,847 lives lostUSD 100+ billion damages

6

Disasters


(CRED criteria)

Human consequences:211 million affected

16,847 lives lostUSD 100+ billion damages

Institutional responses

7

Post-disaster relief and recovery efforts

• USD 5.9 billion (pledged) for 2005 Pakistan earthquake

• USD 13.5 billion (pledged) for 2004 Indian Ocean tsunami

• Actors: Diverse mix of governments, funders, IFIs, aid agencies, humanitarian agencies, int’l/local NGOs.

8

How does PD assistance get evaluated?

• Extensive process evaluations

• Multiple levels of analysis (project, agency, sector, disaster)

• Some joint evaluations (e.g. TEC)

• Review of ALNAP database, etc. suggests few examples of “rigorous” impact evaluation

9

Why so little focus on IE in PD settings?

10

Why so little focus on IE in PD settings?

“Disasters are

different”

“Disasters are

different”

“Disasters are

different”“Disasters

are different”“Disasters

are different”

“Disasters are

different”

11

Are disasters any different?

1. Unpredictable, rapid-onset event

2. Proven life-saving measures cannot be randomized or withheld

3. Mismatch between resources and need (sometimes)

4. Absence of baseline data (usually)

5. Which counterfactual is the right one?

12

Are disasters any different? Maybe not…

1. Nonrandom exposure to disaster event and consequences

2. Nonrandom assignment of interventions

3. Fragile states/vulnerable populations

4. Multiple concurrent interventions

5. Which counterfactual is the right one?

13

• Bangladesh floods, 1998

• Hurricane Mitch, 1998

• Indian Ocean tsunami, 2004

• Hurricane Katrina, 2005

Lessons learned from other PDIE experiences

14

Disaster-related time periods

14

Pre-disaster Immediate post-disaster

Post-intervention (1)

Post-intervention (2)

Emergency

Relief

Recovery/Reconstruction

t-1 t0 t1 t2D

ISASTER

15

Disaster-related populations

15

A Disaster-affected households* that receive assistance or interventions

B Disaster-affected households that do not receive assistance or interventions‡

C Non-affected households† that were similar to A before the disaster

* or communities (or other unit of analysis)‡ or receive them later, or receive different ones† or less-affected households/communities

16

Time Description Disaster-affected households, treated

t0-t-1

t1-t-1

Disaster-related losses

Restoration to baseline

A0-A-1

A1-A-1

t1-t0

t2-t-1

t2-t0

t2-t1

Recovery from disaster

Sustained restoration to baseline

Sustained recovery from disaster

Persistence of recovery

A1-A0

A2-A-1

A2-A0

A2-A1

Within treatment group, single-difference over time

17


t0-t-1

t1-t-1



A0-A-1

A1-A-1

t1-t0

t2-t-1

t2-t0

t2-t1





A1-A0

A2-A-1

A2-A0

A2-A1


ERRA: “Build Back Better”

18


t0-t-1

t1-t-1



A0-A-1

A1-A-1

t1-t0

t2-t-1

t2-t0

t2-t1





A1-A0

A2-A-1

A2-A0

A2-A1


Problems: Recall bias if no baseline; attribution?

19

Time Description Affected treated-Non-affected

t-1 Baseline (pre-disaster) A-1-C-1

t0 Emergency (immediate post-disaster) A0-C0

t1 Relief/Reconstruction (post-intervention #1) A1-C1

t2 Recovery (post-intervention #2) A2-C2

Cross-sectional, single-difference over treatment groups (A vs. C)

20







21







Implied counterfactual: What would “A” households look like if there had been no disaster?

22







Problems: Is there an appropriate “C” group? If so, were they observed? Attribution?

23

Time Description Affected — Unaffected

t0-t-1

t1-t-1



(A0-A-1)

(A1-A-1)

— (C0-C-1)

— (C1-C-1)

t1-t0

t2-t-1

t2-t0

t2-t1





(A1-A0)

(A2-A-1)

(A2-A0)

(A2-A1)

— (C1-C0)

— (C2-C-1)

— (C2-C0)

— (C2-C1)

Difference-in-difference (A vs. C)

24

Time Description Affected — Unaffected

t0-t-1

t1-t-1



(A0-A-1)

(A1-A-1)

— (C0-C-1)

— (C1-C-1)

t1-t0

t2-t-1

t2-t0

t2-t1





(A1-A0)

(A2-A-1)

(A2-A0)

(A2-A1)

— (C1-C0)

— (C2-C-1)

— (C2-C0)

— (C2-C1)

Difference-in-difference (A vs. C)

Controls time-variant factors that are the same between A & C

25

Time Description Affected treated-Affected control

t-1 Baseline (pre-disaster) A-1-B-1

t0 Emergency (immediate post-disaster) A0-B0

t1 Relief/Reconstruction (post-intervention #1) A1-B1

t2 Recovery (post-intervention #2) A2-B2

Cross-sectional, single-difference over treatment groups (A vs. B)

26







27







Implied counterfactual: What would “A” households look like if there had been no intervention?

28







Problems: How were interventions assigned to A but not to B?

29

Time Description Disaster-affected households

“Treated” — “Control”

t0-t-1

t1-t-1



(A0-A-1)

(A1-A-1)

— (B0-B-1)

— (B1-B-1)

t1-t0

t2-t-1

t2-t0

t2-t1





(A1-A0)

(A2-A-1)

(A2-A0)

(A2-A1)

— (B1-B0)

— (B2-B-1)

— (B2-B0)

— (B2-B1)

Difference-in-difference (A vs. B)

30

Time Description Disaster-affected households

“Treated” — “Control”

t0-t-1

t1-t-1



(A0-A-1)

(A1-A-1)

— (B0-B-1)

— (B1-B-1)

t1-t0

t2-t-1

t2-t0

t2-t1





(A1-A0)

(A2-A-1)

(A2-A0)

(A2-A1)

— (B1-B0)

— (B2-B-1)

— (B2-B0)

— (B2-B1)

Difference-in-difference (A vs. B)

Controls time-variant factors that are the same between A & B

31

World Bank impact evaluation of housing and livelihood grants

32

• Instrumental variable approach to disaster impact:

– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES

– Villages at different distance from fault line experienced different earthquake severity.


33





A1-C1

34




• Variation in receipt of relief and recovery funds:– Between-district variation in implementing agency for

housing grant – Threshold eligibility for livelihoods grant of 5

dependents/households: regression continuity design.


A1-C1

35




• Variation in receipt of relief and recovery funds:– Between-district variation in implementing agency for

housing grant – Threshold eligibility for livelihoods grant of 5

dependents/households: regression continuity design.


A1-C1

A1-B1

36

ERRA impact evaluation case study

1. Evaluation opportunities using existing data & HH sample

– Household data collection at t2

– Retrospective household reports of t0

– Use of ongoing government household surveys (e.g., HIES) as baseline

– Randomization of some interventions from 2009

37

ERRA impact evaluation case study

2. Evaluation opportunities in a future disaster– Maintain surveillance sample in disaster-prone regions

– Household-level data collection at t0

– Randomized interventions, e.g,• Timing of interventions:

– Group 1: Housing grant first, followed by livelihood cash grant

– Group 2: Livelihood cash grant first, followed by housing grant

• Conditionality of grants

• Types of interventions, e.g, different formats or recipients of livelihoods cash grant

38

PDIE Guiding Principles

1. PDIE is necessary to ensure that relief and recovery funds are appropriately targeted, effective, and efficient.

2. Each phase of a disaster (emergency, relief, recovery/reconstruction) presents distinct evaluation challenges and therefore may require a different evaluation approach or methodology.

3. “Evaluation preparedness” is an important part of disaster preparedness.

39


5. PDIE should incorporate evaluation of (pre-disaster) investments in disaster mitigation, prevention, and resilience.

6. Rigorous PDIE requires the tools and perspectives of multiple disciplines and sectors.

7. Quantitative PDIE can benefit from the qualitative and mixed-methods approaches.

40


7. Proportionate changes in outcomes over time and over groups can be as instructive as changes in levels.

8. Change-over-time impact evaluations should recognize two distinct baselines: pre-disaster, and immediately post-disaster.

41

PDIE Guiding Principles (ct’d)

9. PDIE will be most successful when the goals of the intervention are clearly defined through a logical framework or similar model; when the interventions are appropriately targeted, and when the purpose/use of the evaluation is clear.

10. Experimental and quasi-experimental approaches are feasible in PDIE if ethical, logistical and “fit” issues are adequately addressed.

Documents

Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation