Upload
deirdre-hutchinson
View
232
Download
0
Tags:
Embed Size (px)
Citation preview
PLAN
• Locating impact evaluation• Menu of methods• Case Study: A randomised experiment• Challenges with RCTs in education• Opportunities for research in government
Locating impact evaluation
• Qualitative work– Hoadley (2003,2007); Ensor et al (2009)
• Systemic analysis with mixed methods– Taylor, Vinjevold, Muller (2003); Fleisch (2008)
• Descriptive quantitative work– Reddy (2006); Taylor & Yu (2008); Spaull (2011)
• Correlational analysis– Crouch & Mabogoane (1998); Van der Berg (2008);
Gustafsson (2007); Spaull (2012); Shepherd (2011)• Moving toward causal quantitative analysis
The evaluation problem: knowing a counterfactual
• The evaluation problem:– We cannot observe the counterfactual:– 2 alternative scenarios for the same person or group.
• So we have to identify or construct comparison groups as a “pseudo-counterfactual”.– Or an estimate of the counterfactual
• The big question is: when is a comparison group a valid estimate of the counterfactual?– Internal validity
• Selection bias (endogeneity):– Years of Schooling and IQ– Libraries and learning outcomes
A menu of methods
• Pre & Post• Simple Difference• Difference-in-differences• Regression & matching• Fixed effects• RCT• RDD• IV
}}
}
Non-experimental(observed data)
Experimental
Quasi-Experimental
Background to the “Mind The Gap” study
• Mind the Gap study guides developed during 2012• Aimed at acquiring the basic knowledge and skills
necessary to pass the matric exam• Distributed to schools in some parts of the country
– Mainly underperforming districts in EC, NC, a bit in Gauteng and elsewhere, but not in Mpumalanga
• Impact evaluation using 4 subjects in MP– ACCN, ECON, GEOG, LFSC
The Sampling Frame
• National list of schools that were enrolled for the matric 2012 examination.
• The list was then restricted to only schools in Mpumalanga.• Further restricted to schools registered to write the matric
2012 exam in English.• The final sampling frame consists of 318 schools.• Randomly allocated guides to 79 schools (books were
couriered – delivery reliable)• Leaves 239 control schools• Books delivered late in Year: September
Main Results:OLS regressions with baseline
To summarise: No significant impact in Accounting & Economics;Impacts of roughly 2 percentage points in Geography & Life Sciences
Did impact vary by school functionality?20
3040
5060
Pre
dict
ed s
core
in 2
012
0 20 40 60School mean score in 2011
Control Treatment
2030
4050
60P
redi
cted
sco
re in
201
2
20 30 40 50 60 70School mean score in 2011
Control Treatment
Geography Life Sciences
Matric 2010 simulation
• Roughly a 1 percentage point increase in matric pass rate
5609The number of children who did not pass matric in
2010 but would have passed had Mind The Gap been nationally available Geography and Life Sciences.
Interpreting the size of the impact
• Very rough rule of thumb: 1 year of learning = 0.4 to 0.5 standard deviations of test scores
• Geography: 13.5% SD• Life Sciences: 14.4% SD• Roughly a third of a year of learning• The unit cost per study guide (reflecting material
development, printing and distribution) is estimated to be R41,82
Interpretation of results
• 2 guides had no impact: Interventions do not always impact on desired outcomes
• Interventions are not uniform in effectiveness• The quality of the ACCN & ECON material?
– Or of the GEOG & LFSC materials?• Contextual factors pre-disposing LFSC & GEOG to have an impact
but not ACCN & ECON?• A certain level of school functionality / managerial capacity
needed in order for resources to be effective• Timing of the delivery of guides• External validity
– We are more certain about delivery in MP than if this were taken to scale– Awareness campaigns could increase the impact at scale
Critiques of RCTsExternal validity
• Necessary and sufficient conditions for impact evaluations (internal and external validity)
• Internal validity = causal inference• External validity = transferability to population
– Context: geography, time, etc...?• E.g. Private schools, class size
– Special experimental conditions• Hawthorne effects• Implementation agent• System support
External validity:Recommendations
• Choose a representative & relevant study population
• Investigate heterogeneous impacts• Investigate intermediate outcomes• Use a realistic (scaleable) model of implementation
and cost structure• Work with government... But be careful• No pre-test...? Or use administrative data (ANA &
NSC provide opportunity here for DBE collaboration)
RCTs in Education:Practical challenges
• Fund raising• Stakeholder engagement• Test development• Fieldwork quality• Project management
Evaluations with Government:Advantages
• Accountability• Shifts the focus from inputs (e.g. number of teachers trained)
to outcomes; From form to function (mimicry).• Cooperation between government and other actors
(researchers, NGOs, etc)• Encourages policy-makers to interact with research and
evidence• Thinking about theories of change
– Shifts the focus from did government programme X succeed or fail, to why? The agency of programme recipients to change behaviour.
• Benefits for research: reduces publication bias
Evaluations in Government:Opportunities
• Low hanging RCTs– 1000 libraries; EGRA
• RDD• Encouragement designs
– Online tools; winter schools• Good analysis of existing data
– Grade R evaluation– LOLT paper