Embed Size (px)
DESCRIPTION
Project Impact: beyond the talk of 'what we invest in' to 'what our investments have achieved.'
Citation preview
© IMPACT CONSULTING 2009
CSI Matters Conference, Johannesburg
25 May 2011
Project impact:Beyond the talk of ‘what we invest in’ to
‘what our investments have achieved’
© IMPACT CONSULTING 2009
Contents
1. Introduction:• Impact Consulting• Defining impact assessment/evaluation
2. Rationale for undertaking project impact evaluations
3. Is impact evaluation neglected/inadequately addressed with the CSI and NPO arena?
4. Whose responsibility?
5. Options for impact evaluation and can they be done affordably?
© IMPACT CONSULTING 2009
Impact Consulting
• We provide services to social development projects and organisations to assist them to maximise their positive social impact
• Services: o M&E: from impact planning to impact evaluation to mentoring
and capacity buildingo Strategy and programme design o Social researcho Information management (DATAFOX)
• Clients: NPOs, donors and government
• Conducted over 40 evaluations in the last 6 years across sectors
© IMPACT CONSULTING 2009
Introduction to impact evaluation
• Impact assessment vs impact evaluation
• Theory of change and theory of action
• Impact vs outcomes
© IMPACT CONSULTING 2009
Theory of change and theory of action
Theory of change
Theory of action
Positive reinforcement
Negative reinforcement
Gold star on chart
Go out for ice-cream
Withhold pocket money
© IMPACT CONSULTING 2009
Theories of change/logic models--------------- Change theory -----------------
----------- Action theory ----------
© IMPACT CONSULTING 2009
Impact evaluation vs other types of M&E
Evaluation Activities
Programme outcomes/
effects
Outcome Evaluation/
ImpactEvaluation (Summative)
Conceptualisationand design of
programme
Clarificatory/theory evaluation
(also needs ax,feasibility study)
First version(s)of programmeimplemented
Processevaluation(Formative)
“Mature”version of
programmeimplemented
ProgrammeMonitoring(Formative/Summative)
Project Cycle
© IMPACT CONSULTING 2009
Why evaluate?
Other intentions?
• marketing/fundraising
• decision-making (re-funding)
Source: Davies/Dart (2005): MSC guide, on http://www.mande.co.uk/docs/MSCGuide.pdfx
© IMPACT CONSULTING 2009
Case study: Infant mortality and nutrition
© IMPACT CONSULTING 2009
Choosing to conduct an impact evaluation
• Is this the appropriate type of evaluation for this project?
• Am I trying to test a theory? (finding gaps, improving)
• Am I trying to maximise the positive impact of my project?
• Am I trying to prove a replicable model?
• Am I wanting to take this project to scale?
© IMPACT CONSULTING 2009
Is impact evaluation a neglected area for CSI and NPOs?
Often not – appropriateness due to short-term funding or
type of project (eg advocacy)
For impactful projects, barriers include:
• Funding
• Technical skills/knowledge
• Capacity (staff and time)
• Data and documentation (primary and secondary)
• Motivation/desire/buy-in (NPO and CSI)
• No culture of learning
© IMPACT CONSULTING 2009
Action learning
© IMPACT CONSULTING 2009
Roles of CSI and NPOs in impact evaluation
Some issues
• Who pays?
• Who decides on evaluation terms and design?
• What methods and questions are valued?
• Which impact/outcomes are valued?
• Who chooses the evaluator?
• Who uses evaluation?
• Who does the M&E?
NPO is ultimate user and driver
Partnership?
© IMPACT CONSULTING 2009
Uses for external evaluators
• Mentoring/capacity building
• Independent “objective” evaluations
• Highly technical studies
NOT IN PLACE OF INTERNAL M&E/LEARNING
© IMPACT CONSULTING 2009
Options for impact evaluation
“Gold standard” approach, focus on quantitative
• NB: mixed methods
• Monitoring to be done internally
• Incorporate with other kinds of M&E
• Focus on learning
• Key impact indicators AND emergent (NB!!)
• Use secondary data where possible (eg education
data)
• Needs investment
© IMPACT CONSULTING 2009
Case study: Infant mortality and nutrition
Case study: Housing and quality of life
© IMPACT CONSULTING 2009
Q
Case study: Impact evaluation design
Rapid evaluation to identify
domains of change:
grounded, exploratory, qualitative
IMPACT EVALUATION
Baseline Year 1 Year 2
300 shack
dwellers
300 shack
dwellers
Community 1
Community 2
100 shack
100 org
100 RDP
100 shack
100 org
100 RDP
100 shack
100 org
100 RDP
100 shack
100 org
100 RDP
Quantitative: Household survey with impact indicators
Qualitative: MSC stories, focus groups, informant interviews
© IMPACT CONSULTING 2009
Results: expected/theoretical
Psychological wellbeing
Safety and
security
Comfort
Health
EducationEconomic wellbeing
Social cohesion?
But was this the NGO’s expectation? Or the donor’s?
...Time...
© IMPACT CONSULTING 2009
• Disappointment that the only significant
QUANTITATIVE findings were for shorter-
term domains of change, despite amazing
qualitative results and indicators towards
longer-term changes (qual not valued)
• Looking for methodological faults to
explain perceived disappointing results
NPO’s response to results
© IMPACT CONSULTING 2009
In conclusion
• Short-term funding = short-term change – this also has to be
reflected in what is “sold”
• Vital to measure expected change and specifically look for
unexpected
• Vital to evaluate throughout the process, not wait till the end
• Funding must be flexible to allow for learning and formative
changes
• Impact evaluation needs to be budgeted for and included
from the beginning
© IMPACT CONSULTING 2009
Contact details
Jerushah Rangasami
Director: Impact Consulting
082 776 2286
021 424 4166
www.impactconsulting.co.za
