Presented by Malte Lierl (Yale University). Making the Most out of Discontinuities. Introduction. How do we measure program impact when random assignment is not possible ? e.g. universal take-up non-excludable intervention treatment already assigned Solutions - PowerPoint PPT Presentation
Making the Most out of Discontinuities
Presented byMalte Lierl (Yale University)DIME FRAGILE STATESDUBAI, MAY 31 JUNE 4This presentation is based on work done by Erich Battistin, Jean-Louis Arcand, Nandini Krishnan and Florence Kondylis.1How do we measure program impact when random assignment is not possible ?e.g. universal take-up non-excludable interventiontreatment already assignedSolutionsMake assumptions about what constitutes a plausible control group (matching on observables, diff-in-diff)Exploit quasi-random aspects of program implementation Quasi-experimentsExample: Regression Discontinuity Design (RDD)IntroductionDiscontinuity = Arbitrarily placed cutoff for program eligibility Regression Discontinuity Design (RDD)vulnerability indexincomeAround the cutoff, beneficiary (treated) and non-beneficiary (untreated) populations are very similar.For the population around the cutoff, RDD can be as credible as a randomized experiment.
vulnerability indexincomePROGRAM IMPACTCUTOFFRDD: Some examplesExample 1: Evaluate reintegration assistance for former child soldiers aged 16 and below.
An ex-combatant aged 16 years and one day would not benefit from the program. RDD would compare individuals just above and just below 16 years of age.
RDD: Some examplesExample 2: If you are elected into parliament, will this make you wealthier?
Cant randomize who gets into parliament. In majoritarian systems such as in the UK, you get into parliament if you have the majority of votes in a district. Some districts have very close election results. Between two candidates with 49.5% and 50.5% of votes it is as good as random who gets into parliament. RDD: compares winners and losers in very close runoffs.
Another RDD exampleExample 3: Minimum legal drinking age in the United States is 21
It is illegal to sell alcohol to people younger than 21People aged 21 and people aged 20, 11 months, 29 days are treated very differently under the drinking age policyBut they are not inherently different (likelihood to go to parties, obedience, propensity to engage in risky behavior, etc.)
What is the effect of alcohol on mortality rates?In effect, the minimum drinking age assigns people into treatment and comparison groupsTreatment group: People between ages 20 years and 11 months and 20 years 11 months and 29 days cannot drink alcohol. Comparison group: People just above 21 can drink.Both groups should be similar in terms of observable and unobservable characteristics that affect outcomes (mortality rates).
If we use the drinking age cutoff as RDD, we can estimate the causal impact of alcohol consumption on mortality rates among young adults.
What is the effect of alcohol on mortality rates?Source: Carpenter & Dubkin, 2009RDDProportion of days drinking, by age
Increased alcohol consumption causes higher mortality rates around the age of 21All deathsAll deaths associated with injuries, alcohol or drug useAll other deathsRDDWhat is the effect of alcohol on mortality rates? Death rates, by ageSource: Carpenter & Dubkin, 2009Internal ValidityIf the cutoff is arbitrary:Individuals directly above and below the cutoff should be very similar in expectationSystematic differences in outcomes are caused by the policy
Major assumptions: Individuals have no precise control over assignment variable Nothing else is happening. In absence of the policy, we would not observe a discontinuity around the cutoff. Might not be the case if: Drinking age is 18, and driving also becomes legal at age 18 Another program provides reintegration assistance for ex-combatants over 16 years.
RDD RequirementsTransparency and precise knowledge of the selection process
Treatment is discontinuous with respect to an assignment variable
Individuals cannot precisely manipulate the assignment variable
All other factors are continuous with respect to the assignment variable (nothing else is happening)
Enough data points around the cutoffSharp discontinuity Discontinuity precisely determines treatment status All people 21 and older drink alcohol and no one else doesAll ex-combatants younger than 16 receive assistance, nobody else doesFuzzy discontinuityPercentage of participants changes discontinuously at cut-off, but not from 0% to 100% (or from 100% to 0%)Some people younger than 21 end up consuming alcohol and/or some older than 21 dont consume at allSome youth ex-combatants under 16 dont participate, and their slots are given to others who are just over 16. Sharp and Fuzzy RDDsSharp and Fuzzy RDDs1010assignment variableassignment variableProbability of being treatedProbability of being treatedSHARP DISCONTINUITYFUZZY DISCONTINUITYExternal validityAre RDD estimates of program impact generalizable?
Counterfactual/control group in RDD: Individuals marginally excluded from benefitsExamples: Ex-combatants over 16, candidates with 49.5% of votesCausal interpretation is limited to individuals/households/villages near the cutoffExtrapolation beyond this group needs additional (often unwarranted assumptions) Or multiple cutoffs! RDD ImplementationData collection: Make sure to have enough observations around the cutoffAnalysis: Observations away from the cutoff should have less weight
Why? Only near the cutoff can we assume that people find themselves to the left and to the right of the cut-off by chance.weightassignment variableoutcomeRDD ImplementationCarefully justify study designBaseline data will be useful to verify assumptionsassignment variableoutcomeassignment variableoutcomeBEFORE PROGRAMAFTER PROGRAMRDD ImplementationCarefully justify study designGraphical analysis is an important toolassignment variableoutcomeSummaryAdvantages of RDDs:RDD can be applied even when randomization is not feasible e.g. to programs with means tests for eligibilityFor the population around the cutoff, RDD is as credible as a randomized experimentRequires fewer assumptions than other non-experimental methodsRDD can be used like a natural experiment to evaluate a program ex-post
SummaryDrawbacks of RDDs:Limited external validity: The estimates of program effects are informative only for the population around the cutoff. RDD requires a lot of data around the cutoffKnowledge about the cutoff may induce behavioral change that can bias your evaluatione.g. ex-combatants misreport their agee.g. candidates become frustrated because they were so close to getting elected
20Thank you!Further reading:
Lee, David and Thomas Lemieux (2009): Regression Discontinuity Designs in Economics, NBER Working Paper No. 14723. http://www.nber.org/papers/w14723