Embed Size (px)
Citation preview
Monitoring and Evaluation in Fragile and Conflict-Affected Contexts: The challenges of measurement
David Fleming, Senior Consultant, Itad
Date: 28th January 2015
Seminar Outline
1. Introducing Itad: Life as an M&E consultant2. Introducing/recapping M&E: Why monitor
and evaluate and why important in FCAS?3. Theories of change: what they are, why they
are useful and challenges in FCAS4. M&E approaches and methods: how to
monitor and evaluate in FCAS; examples from peacebuilding and humanitarian work
Learning objectives
1. Come away with a better understanding of why we do M&E and why it’s particularly important in FCAS
2. Learn about and put into practice some of the most important M&E methods and tools for FCAS
3. Be able to better identify the challenges of doing M&E in FCAS and how to overcome these
4. Everyone to leave the room with a burning desire to get involved in M&E at some point in the future!
Life as an M&E consultant…
2. Introducing M&E: Why Monitor and Evaluate?
“After decades in which development agencies have disbursed billions of dollars for social programs, and developing country governments and nongovernmental organizations (NGOs) have spent hundreds of billions more, it is deeply disappointing to recognize that we know relatively little about the net impact of most of these social programs”
‘When will we ever learn?’ Evaluation Gap Working Group, Center for Global Development 2006
• Monitoring: “Collection of data with which managers can assess extent to which objectives are being achieved” (World Bank)– Purpose: Collect information on programme outputs and
outcomes to track and improve performance and results
• Evaluation: “Determination of the value of a project, programme or policy” (World Bank)– Purpose: evidence-based decisions, accountability,
transparency, lesson learning– Types: project, programme, policy, organisation, sector, theme,
formative, summative, impact…
Why is M&E important in FCAS?
1. Development trends in FCAS• By 2015, 50% of world’s poor will live in fragile states
(OECD); by 2030 it might be two thirds (Brookings)• Support to conflict, violence and fragility becoming a key
priority for most major donors• ODA to fragile states is falling in quantity but number of
actors multiplying (OECD)• DFID has been scaling up support to FCAS (commitment to
increase to 30% of ODA by 2015)• DFID strategies include BSOS, cross-Whitehall CSSF, and the
‘Beyond Aid’ agenda
Why is M&E important in FCAS?
2. Increasing emphasis on transparency, accountability and fiduciary risk
• Higher risk to investments in terms of results, security and fiduciary risk
3. More limited evidence base – need for lesson learning and evidence of what works
• Support evidence-informed decisions and better programming by knowing what works and doesn’t and why and in which contexts
What are the biggest challenges?
Risk of exacerbating
conflict
Hawthorne effect
Insecurity
Political objectives
Longer-term nature of results
Measurement challenges
Vulnerability to biases
Lack of existing data
Poor data reliability
Poor data accessibility
Unpredictable chains of causation
Complex and dynamic contexts
M&E within the programme cycleIdentification
Problem analysis
AppraisalEvidence of what
works
DesignMost cost-effective
intervention/s
ImplementationWith M&E built in
from outset
CompletionMeasure results
– did it work?
PostCompletion
Feed lessons into future decisions
Lesson Learningand
Feedback
Challenges of programming in FCASIdentification
Problem analysis contested
AppraisalLittle robust
data and research. No
time
DesignLittle evidence to assess cost effectiveness.
Political imperatives
ImplementationGreat hurry. M&E lags behind. No
baselines/ measurement
strategies
CompletionNot enough data
to say. No inclination to admit failure
PostCompletionNot enough
results published/
stored/ synthesisd.
Disagreement No knowledge management/sharing and
lots of uncoordinated actors
3. Theories of Change (ToC)
Why are ToCs useful for M&E?A ToC is an iterative and collaborative process for thinking through how a programme is expect to work within the context of the broader system. It should create the space for critical reflection and learning and be adjusted and iterated over time.
• Links to assumptions box in LF, but goes beyond this in focusing on iterating through learning shared mental models of how change happens
• Important for developing M&E strategy – test key links and assumptions (intellectual leaps) in the causal chain over the life of the programme
• Important for evaluability – provides foundation for a theory-based evaluation • Important to talk of ‘theories’ not ‘theory’ – i.e. to recognise and manage a
range of theories and multiple drivers of change• Not a tick-box exercise or management tool like the LF but a way of working
and thinking – it’s primarily a process rather than a product
What are the pitfalls in FCAS?• Time and resource-consuming – so they can often be poorly conceived/
too vague• Poorly understood/used – as linear tick-box exercise rather than iterative
approach• Oversimplification of complex contextual (e.g. conflict) factors – reflexivity
and feedback loops in complex conflict systems – black swan idea• Absence of/poor conflict analysis – must underpin project design• Difficulties in evidence gathering/data collection – conflict environments
are often data rich but information poor – insecurity, staff turnover• Difficulties of working with and aiming to influence a range of actors• Unpacking chains of cause and effect in FCAS can be very difficult• Death by diagram• Funnel of attrition
The funnel of attritionOnly these people may experience improved outcomes
4. M&E approaches and methods
Recent explosion of new and innovative approaches to monitoring and evaluation:
1. Use of mobile technology and ICTs for data collection and analysis – e.g. Ushahidi
2. Influence of complexity science – PDIA, DDD – enabling environment for experimentation
3. Remote monitoring and verification4. Rigorous evaluation/impact evaluation designs
Why evaluate?
• White and Waddington (2012):
‘The use of the systematic reviews methodology is comparatively new among social scientists in the international development field, but has grown rapidly in the last 3 years...To date, there has not been a strong tradition of using rigorous evidence in international development. The evidence bar has been rather low, with many policies based on anecdote and ‘cherry picking’ of favourable cases’.
Why evaluate?
• Accountability and lesson-learning– Accountability to taxpayers and beneficiaries– Understanding what works, why, where and for whom
to underpin evidence-based programming– Priority to evaluate interventions with a weak evidence
base• Inform scale up of an intervention or transfer to
another context• Make mid-course corrections• To support spending decisions
What is impact evaluation?“Impact evaluation is a with versus without analysis: what happened with the programme (a factual record) compared to what would have happened in the absence of the programme (which requires a counterfactual)” (White, 2013)
“Impact evaluation aims to demonstrate that development programmes lead to development results, that the intervention has a cause and effect” (Stern et al. 2012)
• Attribution analysis to understand what difference a programme made• Counterfactual construction through experimental/quasi-experimental
methods for large n (comparison groups); causal chain analysis for small n• Theory-based impact evaluation – in ideal world, an RCT should be embedded
in a broader theory-based design that addresses questions across the causal chain (White, 2013)
• Causal chain analysis – rigorous empirical assessment of causal mechanisms and the assumptions that underlie the causal chain
How do we estimate impact?
How do we estimate impact?
Pros and cons of RCTs
• Pros: RCTs are the “gold standard” for addressing attribution when an ex ante design is possible with a large number of units of assignment
• BUT MAJOR DRAWBACKS, ESPECIALLY IN FCAS– Not suited to complex development pathways with
multiple non-linear causal factors– Less appropriate where hard to identify comparison
groups – threat to validity– When extrapolated from their context, RCT findings
lose claims to rigour (Pritchett and Sandefur, 2013)
How best to evaluate in FCAS?
In increasing order of robustness:• Use of evaluation framework and robust
approach to evidence assessment – e.g. humanitarian evaluations
• Use of theories of change and contribution analysis to test causation and assumptions
• Realist evaluation design looking at how different mechanisms operate in contexts
Using an evaluation frameworkQuestions
Theory/Approach
Methods
Tools
Establishing a framework for the evaluation provides a consistent and systematic means to designing the
evaluation, collating and analysing the existing evidence and the new data created, and generating
and interpreting the results. (Magenta Book para 6.1)
Theory or approach• Results-oriented• Theory-based• Participatory/
empowerment• Utilization-focused
Methods• Qualitative• Case study• Experimental• Value for Money• Contribution analysis
Tools• Document review• Key informant interview• FGD• Direct observation• Questionnaire survey• Participatory data collection• SWOT• Forcefield• Stakeholder analysis• Ranking and scoring• [Types of analysis; CBA;
VfM; QCA; etc]]
Methods define which tools and
how to use them
Evaluating peacebuilding
• Most useful definition of impact – understand effects of intervention on conflict drivers
• Conflict analysis is critical – understand/test relevance of intervention to conflict drivers
• Use of ToC to understand/test assumptions about how intervention contributes to change
• Experimental approaches usually not useful – better to look at contribution
M&E Group Exercise
• Split into 4 groups• 2 groups will be responsible for designing an
outline M&E system for a peacebuilding programme
• 2 groups will be responsible for designing an outline proposal to do an external evaluation of the same programme
Further ReadingLiterature on M&E approaches and methods• L. Morra Imas, Rist, R., The Road to Results (World Bank, 2009)• S. Funnell, Rogers, P., Purposeful Program Theory (Wiley, 2011)• E. Stern et al., ‘Broadening the range of designs and methods for impact evaluation’, DFID
working paper 38, April 2012• H. White, Phillips, D., ‘Addressing Attribution of cause and effect in small n impact evaluations’,
3ie Working Paper 15, June 2012• G.Westhorp, ‘Realist impact evaluation: an introduction’, September 2014
Literature on M&E with specific reference to FCAS• DFID, ‘Results in Fragile and Conflict-affected States and Situations’, 2012• DFID, ‘Back to Basics, A compilation of best practices in design, monitoring and evaluation in
fragile and conflict-affected environments,’ March 2013• L. Schreter, Harmer, A., Delivering Aid in Highly Insecure Environments, 2013• S. Herbert, ‘Perceptions surveys in fragile and conflict-affected states’, GSDRC Helpdesk
Research Report, March 2013• DFID, ‘Evaluating impacts of peacebuilding interventions’, May 2014• J. Puri et al. ‘What methods may be used in impact evaluations of humanitarian assistance’, 3ie
working paper 22, December 2014
