Upload
others
View
15
Download
0
Embed Size (px)
Citation preview
Session 2
Theory of Change and Measurement
Impact Evaluation Workshop, Lilongwe, Dec. 14th, 2011
Martin Abel
1
• Session 1: Introduction to impact evaluations
• Session 2: Theory of change and measurement
• Session 3: Group work: Theory of Change
• Session 4: Randomized Evaluation Design
• Session 5: Group work: Evaluation Design
Workshop Schedule – Day 1
2
Recap: Measuring Impact (I)
Time
Pri
mar
y O
utc
om
e
Impact
Intervention
3
Outline
• Theory of Change (ToC)
– Definition
– Case Study 1: Village quotas in India
– Mapping from ToC to indicators
– Case Study 2: Primary education in Madagascar
• Data collection
– When to collect data?
– Data collection instruments
4
Theory of Change (ToC)
• Definition: – A theory of change is a description of how an
intervention is supposed to deliver the desired results.
• ToC: – sets out the causal logic of how and why a
particular project, program, or policy will reach its intended outcomes (hypothesis).
5
Back to Chocolate
• What is the chain of causality from the programme (chocolate) to the outcome (test score)?
• Do you think those who receive chocolates will perform better or worse? Why?
6
People like
chocolate
Participants eat chocolates
The ToC takes us beyond the BLACK BOX
Intervention
Chocolates given to randomly selected participants
Participants are more motivated
Less focused in class
Intervention
Inputs
Intermediary outcomes
Outcomes
Assumptions
Positive effect on learning outcomes
Outputs
Participants are distracted
More focused in class
Negative effect on learning outcomes
Black Box
If we don’t see an effect: where was the causal chain interrupted?
Black Box
8
Case Study 1 Quotas in Village Councils in India
Duflo et al. 2008. “Women as policy makers – evidence from a randomized policy experiment in India”.
• 1993 Constitutional Amendment in India.
• Main characteristics of quota policy – Reservation of leadership to women
– Implemented in randomly selected 1/3 of villages
• Main goals of Village Council – Decentralise decision making, especially public goods
“Every village has to become a self-sufficient republic” (Ghandi)
– Give villagers more say
• Main goals of quota
Why may these goals not be achieved? 9
Case Study Example Quotas in Village Councils in India
10
Needs assessment
Intervention
Output indicators
Intermediary outcomes
Outcomes
Village Quotas: A Possible Theory of Change
Quotas
More female council leaders
Different public goods
Different health and education outcomes?
Low investment in education and health at village level
Women are more empowered
Village Quotas: A Possible Theory of Change
11
Needs assessment
Intervention
Output indicators
Intermediary outcomes
Outcomes
Assumptions
Village leader has decision power
Women have different
preferences
Public goods reflect
women’s preferences
Quotas
More female council leaders
Different public goods
Different health and education outcomes?
Low investment in education and health at village level
Women are more empowered
ToC to Indicators
12
• Once the chain of Theory of Change has been drawn up:
what data should we collect at every step of the way to discriminate between different plausible chains of causality?
Village quotes in India
Theory of Change
Quotas
More female council leaders
Different public goods
Different health and education outcomes?
13
Needs assessment
Intervention
Output indicators
Intermediary outcomes
Outcomes
Low investment in education and health at village level
Women are empowered
14
Gender of leader
Female attendance in council meetings; Time
women speak
Budget allocation; Type of public goods available
Indicators
Village quotas in India
ToC → Indicators
Quotas
More female council leaders
Different public goods
Different health & education outcomes?
Low investment in education and health at village level
Women are empowered
Literacy levels, BMI of children
15
Village quotas in India
ToC → Indicators → Results
More female leaders
Female attendance
Active participation
Allocations shift; More investment in
roads, drinking water
Results
Gender of leader
Indicators Quotas
More female council leaders
Different public goods
Different health & education outcomes?
Low investment in education and health at village level
Women are empowered
Literacy levels, BMI of children
Not yet measured ?
Female attendance in council meetings; Time
women speak
Budget allocation; Type of public goods available
• The ToC can help us to identify potential interventions
• Example: Imagine, you are the mayor of Lilongwe – In the last 5 years, there was a large increase of both
cell phone usage and number of traffic accidents involving pedestrians
What policies could tackle this problem?
16
Example 2: Cell Phone Usage and Pedestrian Accidents
17
Case Study 2 Primary Education Management in Madagascar
Nguyen & Lassibile, 2008. “Improving management in education: evidence from a randomized experiment in Madagascar”.
• Improved access to education but quality remains poor – Pass rate of 63% among Grade 5s – Poor teacher performance identified as key problem
• Madagascar Ministry of Education implemented programme to see whether top-down and/or bottom-up approach is effective in improving teacher performance
18
Primary education management in Madagascar
Context
• Two approaches:
– Top-down approach: provide district and subdistrict administrators with information and tools for supervision
– Bottom-up approach: encourage parental involvement through provision of information and accountability meetings, as well as provide schools with teaching tools
Which method is most effective at increasing teacher effort and learning outcomes?
19
Primary education management in Madagascar
Context
Primary education management in Madagascar
Theory of Change: Top-Down
20 20
Needs assessment
Intervention
Output indicators
Intermediary outcomes
Primary outcome
Intensity and frequency of monitoring increases
Learning outcomes improve
Poor learning levels in primary school
Teacher performance improves
Top-down monitoring programme
Officials receive tools and information
Assumptions
Use of tools increases
monitoring
Increased monitoring causes increased
teacher effort
Intensity and frequency of monitoring increases
Learning outcomes improve
Needs assessment
Intervention
Output indicators
Intermediary outcomes
Primary outcome
Poor learning levels in primary school
Teacher performance improves
Top-down monitoring programme
Primary education management in Madagascar
Theory of Change: Top-Down
Officials receive tools and information
Primary education management in Madagascar
ToC → Indicators
22
Top-down monitoring programme
Learning outcomes improve
Poor learning levels in primary school
Teacher performance improves
Self-reported receipt and usage rates
No. of visits to schools, allocation of time & budget
Attendance, lesson plans, frequency & quality of
evaluations
Indicators
Student attendance, test scores
Officials receive tools & information
Intensity and frequency of monitoring increases
Primary education management in Madagascar
ToC → Indicators → Results
23
Top-down monitoring programme
Learning outcomes improve
Poor learning levels in primary school
Teacher performance improves
Self-reported receipt and usage rates
No. of visits to schools, allocation of time & budget
Attendance, lesson plans, frequency & quality of
evaluations
Indicators
Student attendance, test scores
Schools not visited more often, allocations
unchanged
Teacher behaviour entirely unaffected
Test scores unchanged
Results
Tools were received, tools were used Officials receive tools & information
Intensity and frequency of monitoring increases
Primary education management in Madagascar
Theory of Change: Bottom-Up
24
Community-based monitoring and accountability programme
Parents attend meetings
Learning outcomes improve
24
Needs assessment
Intervention
Output indicators
Intermediary outcomes
Primary outcome
Poor learning levels in primary school
Teacher performance improves
Teaching tools delivered to schools
Parents hold schools accountable
Teachers use of tools
Primary education management in Madagascar
Theory of Change: Bottom-Up
25
Community-based monitoring and accountability programme
Parents attend meetings
Learning outcomes improve
25
Needs assessment
Intervention
Output indicators
Intermediary outcomes
Primary outcome
Poor learning levels in primary school
Teacher performance improves
Teaching tools delivered to schools
Parents hold schools accountable
Teachers use of tools
Parents are interested in
Holding schools accountable influences
teacher effort
Assumptions
Tools make teaching
more effective
Primary education management in Madagascar
ToC → Indicators
26
Community-based monitoring and accountability programme
Parents attend meetings
Learning outcomes improve
Poor learning levels in primary school
Teacher performance improves
Tools delivered to schools
Parents hold schools
accountable
Teachers make use of tools
Indicators
No. of participants at meetings, tools present
Participation of parents, self-reported usage of
tools
Attendance, lesson plans, frequency &
quality of evaluations
Student attendance, test scores
Attendance at meetings good, tools received
Primary education management in Madagascar
ToC → Indicators → Results
27
Community-based monitoring and accountability programme
Parents attend meetings
Learning outcomes improve
Poor learning levels in primary school
Teacher performance improves
Tools delivered to schools
Parents hold schools
accountable
Schools make use of tools
No. of participants at meetings, tools present
Participation of parents, self-reported usage of
tools
Attendance, lesson plans, frequency &
quality of evaluations
Indicators
Student attendance, test scores
Communication with parents unchanged, schools used tools
Attendance unchanged, lesson plan & eval ↑
Attendance and test scores ↑
Results
Theory of Change - Summary
• ToC is useful to help define research questions / hypotheses – Both for experimental and non-experimental research
• ToC important for identifying all outputs, outcomes and assumptions we need to measure
• We learn lessons about underlying mechanism which we can generalize to other programs – “Projects can rarely be replicated, though the
mechanism underlying success or failure will often be replicable and transportable.” (Deaton)
• Different chains of causality can have different policy implications
28
Outline
• Theory of Change (ToC)
– Definition
– Case Study 1: Village quotas in India
– Mapping from ToC to indicators
– Case Study 2: Primary education in Madagascar
• Data collection
– When to collect data?
– Data collection instruments
29
When to collect data?
• Baseline Survey: before the programme is implemented
Target
Population
Not in
evaluation
Evaluation
Sample
Assignment
Treatment
group
Control group
Endline Survey
Measure Impact
Baseline Survey
30
When to collect data?
• Measure inputs and outputs specified in ToC (Process Evaluation)
– Ex: Tutoring program for poor students
• How many tutoring sessions are held? How long is each session? How many students attend? Is the appropriate teaching material available?
• RCT compliance: Do people in treatment group receive the program, and those in control not? 31
Target
Population
Not in
evaluation
Evaluation
Sample
Assignment
Treatment
group
Control group
Endline Survey
Measure Impact
Baseline Survey
When to collect data?
• Endline Survey: after treatment (timing depends on intervention)
– Multiple follow-up surveys possible (medium vs. long-run effects)
• Eg. Village quotas in India – does the perception of women change over time?
32
Target
Population
Not in
evaluation
Evaluation
Sample
Assignment
Treatment
group
Control group
Endline Survey
Measure Impact
Baseline Survey
1. Administrative data
2. Focus groups
3. Survey questionnaires
4. Non-standard instruments
Data collection instruments or methods
33
Instruments (I)
1. Administrative data
– Cheap if routinely collected
– May not be accurate, specific to population of interest, or collect all necessary characteristics
• Case Study 1 - Village quotas in India: 1991 census was used to draw the sample and check that the treatment and control groups were balanced
• Case Study 2 - Primary education management in Madagascar: Scores from the national Grade 5 exam used to measure learning outcomes
34
Instruments (II)
2. Focus group discussions
– Interview with more than one individual at a time
– Can be cheap way to test feasibility, exhaustibility etc
– Useful for understanding local context
3. Survey questionnaires
– Captures information reported by individual
– Can be cross sectional or panel
• Case Study 2 - Primary education management in Madagascar: school questionnaires gathered information on teacher behaviour, usage of tools etc
35
Instruments (III)
• Non-standard instruments:
– Some information can’t easily be collected through standard instruments
– Eg. Corruption, teacher attendance, political participation, service delivery, community trust, female empowerment
• Random spot checks can be useful
– Can make use of unscheduled data collection by:
• Visiting construction sites to measure progress of public goods delivery
• Measuring teacher and student absenteeism
• Visiting police stations incognito to measure discrimination or
corruption
36
Implicit Association Tests (IAT) – Ex: Measuring Discrimination
• Perceptions of marginalised groups is often interesting to researchers, but difficult to measure with explicit questions
“A doctor and his son have an accident and the doctor is killed. The boy is rushed to surgery, and the surgeon says, “this is my son.” What is going on?”
A) The doctor’s wife was unfaithful B) The surgeon is the boy’s mother
Crime
Prison
Education University
Instruments (IV)
Case study:
• Identify Theory of Change
• Identify instruments and outcomes to test the ToC
Your organization’s program:
• Discuss Theory of Change
• What are (intermediary) outcomes and how would you measure them?
Group Work
Group 1: Participation, Accountability and Transparency Group Leader: Cristobal Marshall Organizations: MEJN - Liu Lathu
CCJP PACENET
Group 3: Participation, Accountability and Transparency Group Leader: Martin Abel Organizations: Church & Society
CHRR Trocaire
Group 5: Agriculture / Livelihood Group Leader: Michael Roscitt Organizations: CISANET
ACODE Oxfam
Group 7: Health
Group Leader: Willa Brown Organizations: Norwegian Church Aid
Action Aid Malawi Danish Church Aid
Group Work Group 2: Participation, Accountability and Transparency Group Leader: Clare Hofmeyr Organizations: ACCU TRAC.fm
MISA
Group 4: Economic Governance / Livelihoods
Group Leader: Thomas Coen
Organizations: MEJN Local Development Fund
BMAU
Group 6: Women and Children
Group Leader: Niall Keleher Organizations: PLAN
CARE CISCBE
Annex
40
I. Measurability: a measurable indicator is observable, feasible, and detectable
– Observability: indicators must be observed in the real world.
– Feasibility: indicators must be politically, ethically and financially measurable
– Detectability: study must have instrument and statistical power to measure indicator
Characteristics of good indicators (I)
41
II. Precision: the more exhaustive and exclusive the indicator, the more precise it is
– Exhaustive: most outcomes can be measured by more than one indicator.
– Exclusive: indicators is affected by the outcome of interest and nothing else
42
Characteristics of good indicators (II)
III. Reliability: indicators are reliable when they are hard to counterfeit
– Make sure answer to question is not easily forgettable
– Deliberate misreporting: try to avoid providing any incentive for people to lie in their answer
Use proxy estimators that measures the direct consequence of the same outcome
Remove incentive of respondent to lie by not revealing which is the ‘desirable’ answer
43
Characteristics of good indicators (III)