Embed Size (px)
Citation preview
Evaluability Assessment, Formative & Summative
Evaluation
Laura C. Leviton, Ph.D.
Senior Advisor for Evaluation
■ Summative evaluation:
• Designing and using evaluation to judge merit
■ Formative evaluation:
• Designing and using evaluation to improve intervention
■ Evaluability assessment: Assessing whether
• the intervention is ready to be managed for results
• what changes are needed to do so
• whether evaluation would contribute to improved performance
Definitions
Resources
•Wholey, Hatry & Newcomer, Handbook of Practical Program Evaluation, Wiley 2010
• Leviton et al., Evaluability assessment to improve public health. In Annual Review of Public Health, Volume 31:213-234.
•Leviton, Kettel Khan & Dawkins, New Directions in Evaluation, No. 125, January 2010
Chapter 3 has templates, procedures
Evaluations are Often Handed to Us
“Here, evaluate this.”
Some of the Interventions Just Aren’t Very Good
“Here, evaluate this.”
Why?
• Perhaps 95% of interventions are not fully developed at the time of evaluation.
• Also, ongoing problems with measurement, design and analysis.
• Wilson and Lipsey, 2001 review of 319 meta-analyses:
• Proportion of effect sizes associated with study features
• Variance accounted for:Study methods = features of the intervention
Biggest sources: research design, operationalizing the dependent variable, sampling error
Don’t Rush to Summative Evaluation
• It’s just not cost-effective.
• Good outcome studies are the culmination of careful work
• Start with formative evaluation, or better yet,
• Evaluability assessment.
Steps in Evaluability Assessment
1. Involve intended users of evaluation information
2. Clarify the intended intervention
3. Explore intervention reality
4. Reach agreement on needed changes in activities or goals
5. Explore alternative evaluation designs
6. Agree on evaluation priorities and intended uses of information.
Too Linear – It Works Like This:
Involve end users of
evaluation
Determine scope of project
Review program
documents
Consult stakeholders
Agree-ment on goals?
no
no Create / revise logic model or
theory of change
Agree-ment on model?
yes
Feedback to program manager or policy maker
Interview staff; “scout” the
program reality
yes
Report on assessment of: Plausibility Areas for program development Evaluation feasibility Options for further evaluation Critique of data (quality,
availability)
Develop Program
Stop
Early Steps
Involve end users of
evaluation
Determine scope of project
Review program
documents
Consult stakeholders
Agree-ment on goals?
no
Feedback to program manager or policy maker
Stop
Middle Steps
Review program
documents
Consult stakeholders
Agree-ment on goals?
no
no Create / revise logic model or
theory of change
Agree-ment on model?
yes
Feedback to program manager or policy maker
Develop Program
Stop
Logic Models
Inputs Outputs Outcomes (Impact)
Resources &
Staff
Activities,Outreach to
Target Group
Products,Target Group
Participation
Achieve Short Term Objectives
Achieve Intermediate Objectives
Achieve Long Term Objectives
Theory of Change
Goals Strategies Target Group If-Then Statements
Short Term Outcomes
Long Term Outcomes
Statements How goal will be
accomplished
Define group If (activity) then outcome
Measurement
Was strategy achieved?
Were participants in
the target group?
Did the activity result
in the outcome?
Was short term objective
achieved?
Was long term objective
achieved?
Data Sources E.g., program records
E.g., survey of group
Later Steps
no Create / revise logic model or
theory of change
Agree-ment on model?
Interview staff; “scout” the
program reality
yes
Report on assessment of: Plausibility Areas for program development Evaluation feasibility Options for further evaluation Critique of data (quality,
availability)
What Now?
• Intervention development
• Data collection to inform improvements
• When to do summative (outcome) studies:
• Logic model or TOC is sharpened and agreed to
• The model looks like the reality, and vice versa
• It’s plausible to achieve the outcome(s)
• Formative evaluation indicates intermediate steps are being accomplished.
MeasurementNick Barber15th February 2013
Evaluation of TechnologiesNick Barber15th February 2013
Presentation title set in header
18
When do we measure the effectiveness of this system?
Presentation title set in header
19
Structured approach to IT evaluation
(after Cornford)Technical Stakeholders Institution
Structure
Process Most IT evaluations here
Outcome
Evaluation of TechnologiesNick Barber15th February 2013
