Evaluation in Africa RISING

Evaluation in Africa RISING

Pascale Schnitzer and Carlo Azzarri, IFPRI

Africa RISING–CSISA Joint Monitoring and Evaluation Meeting, Addis Ababa, Ethiopia, 11-13 November 2013

Outline

• Quantitative (experimental and

quasi-experimental)

• Qualitative

• Mix methods

QuantitativeExperimental

• RCT

• Choice experiments, auctions, games

Quasi-experimental

• Double-Difference (Diff-in-Diff)

• Matching

• RD

• IV and encouragement design

Example: providing fertilizers to farmers

Intervention: provide fertilizer to farmers in district A

Program targets all farmers living in district A

Farmers have to enroll at the local extension office to receive the fertilizer

District B does not receive any intervention

Starts in 2012, ends in 2016, we have data on yields for farmers in district A and district B for both years

What is the effect of giving fertilizer on agricultural yields?

Case I: Before & After

(1) Observe only beneficiaries

(2) Two observations in time: yields at T=0and yields at T=1.

Y

TimeT=2012 T=2016

α = 200

IMPACT=A-B= 200?

B

A

2000

2200

Case I: What’s the problem?Unusual good weather/rain:o Real Impact=A-Co A-B is an overestimate

α = 200

2000

2200

Impact?

Impact?

TimeT=2012 T=2016

B

A

C ?

D ?

Droughto Real Impact=A-Do A-B is an

underestimate

Case II: Those Enrolled & Not EnrolledIf we have post-treatment data on

o Enrolled: treatment group

o Not-enrolled: “comparison” group (counterfactual)

(a) Those that choose NOT to participate

(b) Those ineligible to participate (e.g. neighbor

community)

Did AR have a negative impact?

2200

2500

2800

0

500

1000

1500

2000

2500

3000

Choose to participate Choose not to participate Inelegible to participate

Yields in 2016 by participant type

Case I and IIIn the end, with these naïve comparisons, we cannot tell if the program had an impact

We need a comparison group that is as identical in observable and unobservable dimensions as possible, to those receiving the program, and a comparison group that will not receive spillover benefits.

We need to keep in mind…B&A

Compare: Same individuals Before and After they receive P.

Problem: Other things may have happened over time.

E&NECompare: Group of individuals Enrolled in a program with group that chooses not to enroll.

Problem: Selection Bias. We don’t

know why they are not enrolled.

Both counterfactuals may lead to biased estimates of the impact.


• RCT

= Ineligible

RCT

= Eligible

1. Population

External Validity

2. Evaluation sample

3. Randomize treatment

Internal Validity

Comparison

Treatment

X


• RCT

• Choice experiments, auctions, games

Choice experiments, auctions, games

• An experiment is a set of observations generated in a controlled environment to answer a particular question or solve a particular problem.

• Subjects make decisions that are not part of their day-to-day decision making (typically in a game environment), they know they are part of an experiment, or both.

• Purposes:1. Test theories2. Measure what are considered “unobservables” (e.g. preferences, beliefs)3. Test sensitivity of experimental results to different forms of heterogeneity

Choice experiments, auctions, games

• Examples:-behavioral game theory-ultimatum games-dictator games-trust games-public good games-coordination games-market experiments (auctions)-risk- and time-preference experiments

Quantitative

Quasi-experimental designs


Impact =(A-B)-(C-D)=(A-C)-(B-D)

Pro

bab

ility

of

ado

pti

on

B=0.60

C=0.81

D=0.78

T=0Before

T=1After

Time

Participants

Not participants

Impact=0.11

A=0.74

Impact =(A-B)-(C-D)=(A-C)-(B-D)

Pro

bab

ility

of

ado

pti

on

Impact<0.11

B=0.60

A=0.74

C=0.81

D=0.78

T=0Before

T=1After

Time

Enrolled

Not enrolled

Example from Malawi:Total land used (acres)

Treatment Group(Randomized to

treatment)

Counterfactual (Randomized to

Comparison)

Impact(Y | P=1) - (Y | P=0)

Baseline (T=0) [MARBES] (Y) 3.04 2.13 0.91

Follow-up (T=1) [MARBES] (Y) ?? ?? ??

Quantitative

Non-experimental


• Matching

Propensity-Score Matching (PSM)Comparison Group: non-participants with same observable characteristics as participants. In practice, it is very hard.

There may be many important characteristics!

Match on the basis of the “propensity score”,

Compute everyone’s probability of participating, based on their observable characteristics.

Choose matches that have the same probability of participation as the treatments.

Density

Propensity Score0 1

ParticipantsNon-Participants

Common Support

Quantitative

Non-experimental


• Matching

• RD

RD: Effect of fertilizer program on adoption

Improve fertilizers adoption for small farmers

Goal

o Farms with a score (Ha) of land ≤2 are smallo Farms with a score (Ha) of land >2 are not small

Method

Small farmers receive subsidies to purchase fertilizer

Intervention

Regression Discontinuity: Design at baseline

Not eligible

Eligible

Regression Discontinuity: post intervention

IMPACT

Quantitative

Non-experimental


• Matching

• RD

• IV and encouragement design

Babati (WP2): Timeline and design of an evaluation

Feb 13

July 13

Aug-Oct 13

Nov 13 –Mar. 14 Mar. 2016

Initial planting at demonstration

plots

Follow-up field day: farmers rank preferred seeds

Fertilizer and seed distribution

800 farmers in 11 villages

200 receive improved seeds

200 receive improved seeds and

fertilizerEnd-line survey: measure

impacts200 receive

seeds, fertilizer and contracts

200 receive no additional

intervention

Survey

Outline

• Qualitative

Qualitative• Semi-structured or open-ended in-

depth interviews• Focus groups• Outcome Mapping• Participatory Impact Pathways Analysis

(PIPA)

Outcome Mapping (OM)• Contribution of AR to changes in the

actions, behaviors, relationships, activities of the ‘boundary partners’ (individuals, groups, and organizations with whom AR interacts directly and with whom it anticipates opportunities for influence)

• It is based largely on systematized self-assessment

• OM is based on three stages: 1. Intentional design (Why? Who? What? How?)2. Outcome and performance monitoring3. Evaluation design

Outcome Mapping (OM)

By using OM, AR would not claim the achievement of development impacts; rather, the focus is on its contributions to outcomes. These outcomes, in turn, enhance the possibility of development impacts – but the relationship is not necessarily a direct one of cause and effect.

the relationship is not necessarily a direct one of cause and effect

Qualitative

• Participatory Impact Pathways Analysis (PIPA)

Participatory Impact Pathways Analysis (PIPA)• PIPA begins with a participatory workshop where stakeholders make

explicit their assumptions about how their project will achieve an impact. Participants construct problem trees, a visioning exercise and network maps to help them clarify their 'impact pathways‘ (IPs).

• IPs are then articulated in two logic models:1. The outcomes logic model -> the project's medium term

objectives in the form of hypotheses: which actors need to change, what are the changes, which strategies are needed to attain the changes.

2. The impact logic model -> how, by helping to achieve the expected outcomes, the project will impact on people's livelihoods. Participants derive outcome targets and milestones, regularly revisited and revised as part of M&E.

Outline

• Mix methods

Mixed methods

• Combination of quantitative and

qualitative research methods to

evaluate programs

Conclusions• We cannot do everything in every megasite…

• Quantitative surveys are being conducted/planned in every country

• IFPRI has comparative advantage in quantitative approaches, we shall split the tasks with the research teams on qualitative methods -> mixed methods

• Is IFPRI M&E on the right track? What shall we be focusing on more? What shall we not be doing ?

Africa Research in Sustainable Intensification for the Next Generation

africa-rising.net

Technology

Evaluation in Africa RISING