Slide 1

Felix Naschold University of Wyoming Christopher B. Barrett Cornell University May 2012 seminar presentation University of Sydney A stochastic dominance approach to program evaluation And an application to child nutritional status in arid and semi-arid Kenya

Slide 2

Motivation 1. Program Evaluation Methods By design they focus on mean Ex: average treatment effect (ATE) In practice, often interested in broader distributional impact Limited possibility for doing this by splitting sample 2. Stochastic dominance By design, look at entire distribution Now commonly used in snapshot welfare comparisons But not for program evaluation. Ex: differences-in-differences 3. This paper merges the two Diff-in-Diff (DD) evaluation using stochastic dominance (SD) to compare changes in distributions over time between intervention and control populations 2

Slide 3

Main Contributions 1. Proposes DD-based SD method for program evaluation 2. First application to evaluating welfare changes over time 3. Specific application to new dataset on changes in child nutrition in arid and semi-arid lands (ASAL) of Kenya Unique, large dataset of 600,000+ observations collected by the Arid Lands Resource Management Project (ALRMP II) in Kenya (One of) first to use Z-scores of Mid-upper arm circumference (MUAC) 3

Slide 4

Main Results 4 1. Methodology (relatively) straight-forward extension of SD to dynamic context: static SD results carry over Interpretation differs (as based on cdfs) Only feasible up to second order SD 2. Empirical results Child malnutrition in Kenyan ASALs remains dire No average treatment effect of ALRMP expenditures Differential impact with fewer negative changes in treatment sublocations ALRMP a nutritional safety net?

Slide 5

Program evaluation (PE) methods 5 Fundamental problem of PE: want to but cannot observe a persons outcomes in treatment and control state Solution 1: make treatment and control look the same (randomization) Gives average treatment effect as Solution 2: compare changes across treatment and control (Difference-in-Difference) Gives average treatment effect as:

Slide 6

New PE method based on SD 6 Objective: to look beyond the average treatment effect Approach: SD compares entire distributions not just their summary statistics Two advantages 1. Circumvents (highly controversial) cut-off point Examples: poverty line, MUAC Z-score cut-off 2. Unifies analysis for broad classes of welfare indicators

Slide 7

Stochastic Dominance 7 First order: A FOD B up to iff S th order: A s th order dominates B iff MUAC Z- score Cumulative % of population F A (x) F B (x) 0x max

Slide 8

SD and single differences 8 These SD dominance criteria Apply directly to single difference evaluation (across time OR across treatment and control groups) Do not directly apply to DD Literature to date: Single paper: Verme (2010) on single differences SD entirely absent from the program evaluation literature (e.g., Handbook of Development Economics)

Slide 9

Expanding SD to DD estimation - Method 9

Slide 10

Expanding SD to DD: interpretation differences 10 1. Cut-off point in terms of changes not levels. Cdf orders change from most negative to most positive initial poverty blind or initial malnutrition blind. (Partial) remedy: run on subset of ever-poor/always-poor 2. Interpretation of dominance orders FOD: differences in distributions of changes between intervention and control sublocations SOD: degree of concentration of these changes at lower end of distributions TOD: additional weight to lower end of distribution. Is there any value to doing this for welfare changes irrespective of absolute welfare? Probably not.

Slide 11

Setting and data 11 Arid and Semi-arid districts in Kenya Characterized by pastoralism Highest poverty incidences in Kenya, high infant mortality and malnutrition levels above emergency thresholds Data From Arid Lands Resource Management Project (ALRMP) Phase II 28 districts, 128 sublocations, June 05- Aug 09, 602,000 child obs. Welfare Indicator: MUAC Z-scores Severe malnutrition in 2005/6: Median child MUAC z-score -1.22/-1.12 (Intervention/Control) 10 percent of children had Z-scores below -2.31/-2.14 (I/C) 25 percent of children had Z-scores below -1.80/-1.67 (I/C)

Slide 12

The pseudo panel 12 Sublocation-specific pseudo panel 2005/06-2008/09 Why pseudo-panel? 1. Inconsistent child identifiers 2. MUAC data not available for all children in all months 3. Graduation out of and birth into the sample How? 14 summary statistics for annual mean monthly sublocation - specific stats: mean & percentiles and poverty measures Focus on malnourished children Thus, present analysis median MUAC Z-score of children z 0 Control and intervention according to project investment

Slide 13

Results: DD Regression 13 Pseudo panel regression model where D is the intervention dummy variable of interest NDVI is a control for agrometeorological conditions L are District fixed effects to control for unobservables within major jurisdictions No statistically significant average program impact

Slide 14

DD regression panel results 14 (1)(2)(3)(4)(5) VARIABLES median of MUAC Z

23 2. Poverty and Welfare orderings (Foster and Shorrocks 1988) Let U(F) be the class of symmetric utilitarian welfare functions Then A P B iff A U B Examples: U 1 represents the monotonic utilitarian welfare functions such that u>0. Less malnutrition is better, regardless for whom. U 2 represents equality preference welfare functions such that u0. A transfer is valued more lower in the distribution Bottom line: For welfare levels tests up to third order make sense SD, poverty & social welfare orderings (2)

Slide 24

The data (2) extent of malnutrition 24

Slide 25

DD Regression 2 25 Individual MUAC Z-score regression To test program impact with much larger data set Still no statistically significant average program impact

Slide 26

Results DD regression indiv data 26 Robust p-values in parentheses *** p