Upload
measure-evaluation
View
1.846
Download
0
Embed Size (px)
Citation preview
Peter M. Lance, PhDMEASURE Evaluation University of North Carolina at Chapel Hill
MARCH 31, 2016
Fundamentals of Program Impact Evaluation
Global, five-year, $180M cooperative agreement
Strategic objective:
To strengthen health information systems – the capacity to gather, interpret, and use data – so countries can make better decisions and sustain good health outcomes over time.
Project overview
Improved country capacity to manage health information systems, resources, and staff
Strengthened collection, analysis, and use of routine health data
Methods, tools, and approaches improved and applied to address health information challenges and gapsIncreased capacity for rigorous evaluation
Phase IV Results Framework
Global footprint (more than 25 countries)
How Do We Know If A Program Made A Difference? A Brief Helicopter Tour of Methods for Estimating Program
Impact
• The Program Impact Evaluation Challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables
• The Program Impact Evaluation Challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables
• The Program Impact Evaluation Challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables
• The Program Impact Evaluation Challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables
• The Program Impact Evaluation Challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables
• The Program Impact Evaluation Challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables
• The Program Impact Evaluation Challenge
• Randomization
• Selection on observables
• Within estimators
• Instrumental variables
Newton’s “Laws” of Motion
Did the program make a difference?
Did the program cause a change in an outcome of interest Y ?
(Causality)
Our outcome of Interest
What happens if an individual does not participate in a program
What happens if that individual does participate in a program
Potential Outcomes
:
:
:
Our outcome of interest
What happens if an individual does not participate in a program
What happens if that individual does participate in a program
Potential Outcomes
:
:
:
Our outcome of interest
What happens if an individual does not participate in a program
What happens if that individual does participate in a program
Potential Outcomes
:
:
:
Our outcome of interest
What happens if an individual does not participate in a program
What happens if that individual does participate in a program
Potential Outcomes
:
:
:
What happens if
the individual participates
{Causal} Program Impact
Program Impact
What happens if
the individual does not
participate
What happens if
the individual participates
{Causal} Program Impact
Program Impact
What happens if
the individual does not
participate
What happens if
the individual participates
{Causal} Program Impact
Program Impact
What happens if
the individual does not
participate
What happens if
the individual participates
{Causal} Program Impact
Program Impact
What happens if
the individual does not
participate
What happens if
the individual participates
{Causal} Program Impact
Program Impact
What happens if
the individual does not
participate
𝑃 𝑖={1 if individual 𝑖participates ¿0 if individual 𝑖does not participate
Program Participation
𝑌 𝑖=𝑃 𝑖∙𝑌 𝑖1+(1−𝑃 𝑖 ) ∙𝑌 𝑖
0
Observed Outcome
𝑌 𝑖=𝑃 𝑖∙𝑌 𝑖1+(1−𝑃 𝑖 ) ∙𝑌 𝑖
0
Observed Outcome
𝑃 𝑖=1
𝑌 𝑖=1∙𝑌 𝑖1+ (1−1 ) ∙𝑌 𝑖
0
Observed Outcome
𝑃 𝑖=1
𝑌 𝑖=𝑌 𝑖1+0 ∙𝑌 𝑖
0
Observed Outcome
𝑃 𝑖=1
𝑌 𝑖=𝑌 𝑖1
Observed Outcome
𝑃 𝑖=1
𝑌 𝑖=𝑃 𝑖∙𝑌 𝑖1+(1−𝑃 𝑖 ) ∙𝑌 𝑖
0
Observed Outcome
𝑌 𝑖=𝑃 𝑖∙𝑌 𝑖1+(1−𝑃 𝑖 ) ∙𝑌 𝑖
0
Observed Outcome
𝑃 𝑖=0
𝑌 𝑖=0 ∙𝑌 𝑖1+(1−0 ) ∙𝑌 𝑖
0
Observed Outcome
𝑃 𝑖=0
𝑌 𝑖=𝑌 𝑖0
Observed Outcome
𝑃 𝑖=0
{𝑌 𝑖1 ,𝑌 𝑖
0 }
Observed Outcome
{𝑌 𝑖1 ,𝑌 𝑖
0 }
Observed Outcome
{𝑌 𝑖1 ,𝑌 𝑖
0 }
Observed Outcome
{𝑌 𝑖1 ,𝑌 𝑖
0 }
Observed Outcome
{𝑌 𝑖1 ,𝑌 𝑖
0 }
Observed Outcome
Fundamental Identification Problem
of Program Impact Evaluation
{𝑌 𝑖1 ,𝑌 𝑖
0 }
Observed Outcome
Fundamental Identification Problem
of Program Impact Evaluation
Individual Population
Individual Population Hi. They call me
individual i
Individual Population ?!?
{𝑌 𝑖1 ,𝑌 𝑖
0 }
{𝑌 𝑖1 ,𝑌 𝑖
0 }
An expected value for a random variable is the average value from a large number of repetitions of the experiment that random variable represents
An expected value is the true average of a random variable across a population
Expected Value
An expected value for a random variable is the average value from a large number of repetitions of the experiment that random variable represents
An expected value is the true average of a random variable across a population
Expected Value
An expected value is the true average of a random variable across a population
Expected Value
Expectations: Properties
Expectations: Properties
Expectations: Properties
Expectations: Properties
Expectations: Properties
Expectations: Properties
Expectations: Properties
Expectations: Properties
Expectations: Properties
Expectations: Properties
Average Treatment Effect (ATE)
Average Effect of Treatment on the Treated (ATT)
Hi there
Individual Impact
𝑌 𝑖1−𝑌 𝑖
0
𝐸 (𝑌 𝑖1−𝑌 𝑖
0 )
Average Treatment Effect (ATE)
Average Effect of Treatment on the Treated (ATT)
Treatment Effects
Suppose that we have a sample of individuals….
…but for each individual we observe either or …
…but not both
So how do we estimate??
Suppose that we have a sample of individuals….
…but for each individual we observe either or …
…but not both
So how do we estimate??
Remember, however, a key property of expectations:
…but this means that in principle we could estimate and
separately
So how do we estimate??
Remember, however, a key property of expectations:
…but this means that in principle we could estimate and
separately
So how do we estimate??
For instance, suppose that in our sample we have:
participants()
and
non-participants()
(hence )
So how do we estimate??
Then an estimator of is
calculated with the participants out of the sample of individuals
So how do we estimate??
Then an estimator of is
calculated with the participants out of the sample of individuals
So how do we estimate??
Then an estimator of is
calculated with the participants out of the sample of individuals
So how do we estimate??
Then an estimator of is
calculated with the participants out of the sample of individuals
So how do we estimate??
Then an estimator of is
calculated with the participants out of the sample of individuals
So how do we estimate??
Then an estimator of is
calculated with the participants out of the sample of individuals
So how do we estimate??
Similarly, an estimator of is
calculated with the non-participants out of the sample of individuals
So how do we estimate??
So then an estimate of
is
So how do we estimate??
But is it a good estimate??
So we have two samples of size
By random chance, between the two samples we almost surely have
1. A different precise mix of individuals
2. A different number of participants () and non-participants ()
3. Different estimates and of and : 𝑌 1=
∑𝑗=1
𝑛𝑃
𝑌 𝑗
𝑛𝑃 =∑𝑗=1
𝑛𝑃
𝑌 𝑗1
𝑛𝑃
𝑌 0=∑𝑘=1
𝑛𝑁
𝑌 𝑘
𝑛𝑁 =∑𝑘=1
𝑛𝑁
𝑌 𝑘0
𝑛𝑁
So we have two samples of size
By random chance, between the two samples we almost surely have
1. A different precise mix of individuals
2. A different number of participants () and non-participants ()
3. Different estimates and of and : 𝑌 1=
∑𝑗=1
𝑛𝑃
𝑌 𝑗
𝑛𝑃 =∑𝑗=1
𝑛𝑃
𝑌 𝑗1
𝑛𝑃
𝑌 0=∑𝑘=1
𝑛𝑁
𝑌 𝑘
𝑛𝑁 =∑𝑘=1
𝑛𝑁
𝑌 𝑘0
𝑛𝑁
So we have two samples of size
By random chance, between the two samples we almost surely have
1. A different precise mix of individuals
2. A different number of participants () and non-participants ()
3. Different estimates and of and : 𝑌 1=
∑𝑗=1
𝑛𝑃
𝑌 𝑗
𝑛𝑃 =∑𝑗=1
𝑛𝑃
𝑌 𝑗1
𝑛𝑃
𝑌 0=∑𝑘=1
𝑛𝑁
𝑌 𝑘
𝑛𝑁 =∑𝑘=1
𝑛𝑁
𝑌 𝑘0
𝑛𝑁
So we have two samples of size
By random chance, between the two samples we almost surely have
1. A different precise mix of individuals
2. A different number of participants () and non-participants ()
3. Different estimates and of and : 𝑌 1=
∑𝑗=1
𝑛𝑃
𝑌 𝑗
𝑛𝑃 =∑𝑗=1
𝑛𝑃
𝑌 𝑗1
𝑛𝑃
𝑌 0=∑𝑘=1
𝑛𝑁
𝑌 𝑘
𝑛𝑁 =∑𝑘=1
𝑛𝑁
𝑌 𝑘0
𝑛𝑁
So we have two samples of size
By random chance, between the two samples we almost surely have
1. A different precise mix of individuals
2. A different number of participants () and non-participants ()
3. Different estimates and of and : 𝑌 1=
∑𝑗=1
𝑛𝑃
𝑌 𝑗
𝑛𝑃 =∑𝑗=1
𝑛𝑃
𝑌 𝑗1
𝑛𝑃
𝑌 0=∑𝑘=1
𝑛𝑁
𝑌 𝑘
𝑛𝑁 =∑𝑘=1
𝑛𝑁
𝑌 𝑘0
𝑛𝑁
So we have two samples of size
By random chance, between the two samples we almost surely have
1. A different precise mix of individuals
2. A different number of participants () and non-participants ()
3. Different estimates and of and : �̂� 𝟏=
∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋
𝒏𝑷 =∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷
𝑌 0=∑𝑘=1
𝑛𝑁
𝑌 𝑘
𝑛𝑁 =∑𝑘=1
𝑛𝑁
𝑌 𝑘0
𝑛𝑁
�̂� 𝟏 𝑬 (𝒀𝟏 )
�̂� 𝟏=∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋
𝒏𝑷 =∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷
�̂� 𝟏=∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋
𝒏𝑷 =∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷
�̂� 𝟏=∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋
𝒏𝑷 =∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷
𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
�̂� 𝟏=∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋
𝒏𝑷 =∑𝒋=𝟏
𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷
𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (�̂� 𝟏)=𝒏𝑷 ∙𝑬 (∑𝒋=𝟏𝒏𝑷
𝑬 (𝒀 𝒋𝟏 ))
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝑬 (∑𝒋=𝟏
𝒏𝑷
𝑬 (𝒀 𝒋𝟏))
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏
𝑷 ∙𝑬 (𝒀 𝒋𝟏 )
𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)
1ST Rule:
2nd Rule:
𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (�̂� 𝟏)=𝒏𝑷 ∙𝑬 (∑𝒋=𝟏𝒏𝑷
𝑬 (𝒀 𝒋𝟏 ))
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝑬 (∑𝒋=𝟏
𝒏𝑷
𝑬 (𝒀 𝒋𝟏))
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏
𝑷 ∙𝑬 (𝒀 𝒋𝟏 )
𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)
1ST Rule:
2nd Rule:
𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
1ST Rule: 1ST Rule:
2nd Rule:
𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙(∑𝒋=𝟏
𝒏𝑷
𝑬 (𝒀 𝒋𝟏 ))
1ST Rule: 1ST Rule:
2nd Rule:
𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙(∑𝒋=𝟏
𝒏𝑷
𝑬 (𝒀 𝒋𝟏 ))
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏
𝑷 ∙𝑬 (𝒀 𝒋𝟏 )
𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)
1ST Rule:
𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙(∑𝒋=𝟏
𝒏𝑷
𝑬 (𝒀 𝒋𝟏 ))
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏
𝑷 ∙𝑬 (𝒀 𝒋𝟏 )1ST Rule:
𝑬 (�̂� 𝟏)=𝑬 (∑𝒋=𝟏𝒏𝑷
𝒀 𝒋𝟏
𝒏𝑷 )𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝑬 (∑𝒋=𝟏
𝒏𝑷
𝑬 (𝒀 𝒋𝟏))
𝑬 (�̂� 𝟏)= 𝟏𝒏𝑷 ∙𝒏
𝑷 ∙𝑬 (𝒀 𝒋𝟏 )
𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)
𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)
𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (𝒀 𝒋𝟏 )=𝑬 (𝒀𝟏 )
𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)
𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (𝒀 𝒋𝟏 )=𝑬 (𝒀𝟏 )
𝑬 (�̂� 𝟏)=𝑬 (𝒀 𝒋𝟏)
𝑬 (�̂� 𝟏)=𝑬 (𝒀𝟏 )
𝑬 (𝒀 𝒋𝟏 )=𝑬 (𝒀𝟏 )
𝑬 (𝒀 𝒋𝟏 )=𝑬 (𝒀𝟏 )
𝑬 (𝒀𝟏 )
𝑃=0𝑃=0
𝑃=0
𝑃=1𝑃=1
𝑃=1
𝑃=0
𝑃=1𝑃=1
𝑃=1
𝑃=0
𝑃=1
𝑃=1
𝑃=0
𝑃=0
𝑃=1
𝑃=1
𝒀 𝟏
𝑃=0𝑃=0
𝑃=0
𝑃=1𝑃=1
𝑃=1
𝑃=0
𝑃=1𝑃=1
𝑃=1
𝑃=0
𝑃=1
𝑃=1
𝑃=0
𝑃=0
𝑃=1
𝑃=0
𝒀 𝟏
𝑃=1𝑃=1
𝑃=1
𝑃=1𝑃=1
𝑃=1
𝑃=1
𝑃=1
𝑃=1
𝒀 𝟏
Z W
“Z Causes W”
𝑬 (𝑾∨𝒁 )≠𝑬 (𝑾 )
Z W
“Z causes W”
𝑬 (𝑾∨𝒁 )≠𝑬 (𝑾 )
Z W
“Z causes W”
𝑬 (𝑾∨𝒁 )≠𝑬 (𝑾 )
X Y1
X
Y
P
X
Y
P
0
X
Y
P
X
Y
P
X Y1
X Y1
X Y1
X
Y
P
𝑃=1𝑃=1
𝑃=1
𝑃=1𝑃=1
𝑃=1
𝑃=1
𝑃=1
𝑃=1
Selection Bias
The estimator
of
would be biased if some individuals occurred only among participants or non-participants
Or more often among one of the two groups
X
Y
P
X
Y
P
Sir Austin Bradford Hill
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
Strength: How strong is the relationship?Consistency: How consistently is link found?Specificity: How specific is the setting or disease?Temporality: Does the cause precede the effect?Gradient: Does more cause lead to more
effect?Analogy: Do similar “causes” have similar effect?Coherence: Are field and laboratory findings similar?Experiment: Was variation in the cause random?Plausibility: Does theory agree?
Bradford Hill Criteria
We are presented with data in the form of a sample:
Causality: Our Approach
,
We are presented with data in the form of a sample:
Causality: Our Approach
,
Assumptions
ModelE(Y1-Y0),
E(Y1-Y0|P=1),Etc.
We are presented with data in the form of a sample:
Causality: Our Approach
,
Assumptions
ModelE(Y1-Y0),
E(Y1-Y0|P=1),Etc.
Conclusion
Links:
The manual:
http://www.measureevaluation.org/resources/publications/ms-14-87-en
The webinar introducing the manual:
http://www.measureevaluation.org/resources/webinars/methods-for-program-impact-evaluation
My email:[email protected]
MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) under terms of Cooperative Agreement AID-OAA-L-14-00004 andimplemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International, John Snow, Inc., Management Sciences for Health, Palladium Group, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government.www.measureevaluation.org