Upload
judith-whitely
View
217
Download
1
Embed Size (px)
Citation preview
Outcome Mapping
Bringing learning into development programmes
15-18 September 2009Capetown South Africa
Robert Chipimbi Simon Hearn
Acknowledgements This presentation makes use of various
materials that were shared by various OM community members on the OM learning community website. Without being exhaustive special thanks goes to Terry Smutylo, Enrique Mendizabal, Steff Deprez, Jan Van Ongevalle, Daniel Roduner, Kaia Ambrose and many others.
´´The only real voyage of discovery exists, not in seeing new landscapes, but in having new eyes´´
Marcel Proust
Workshop objectives
1. Understand basic OM concepts & principles
2. Decide if & how OM applies to your work
3. Apply some OM tools
Before we start ...
Outcome Mapping is different from conventional PM&E models
It requires time to absorb the concepts and implications for your work
It might seem very complex at first (new concepts, new language, …)
Requires a change in thinking (new paradigm) Generates different reactions
Interest Confusion / doubt Disagreement Defense Let’s go for it! How can we ever do this? …
DON’T WORRY - ASK QUESTIONS!
Developed between 1998 and 2001 by the International Development Research Centre (IDRC) as an answer to the difficulties faced in evaluation processes of development programs:
Demonstrating results
Proving cause/effect relationships
Measuring & attributing impact
What about unintended results?
…
origins of outcome mapping
Some core principles of outcome mapping
1. Embracing complexity
2. Sphere of influence
3. Boundary partners
4. Outcomes as behavioural changes
1. Embracing Complexity
Looking at the bigger picture
See yourself as a part of an interconnected web
of relationships and systems
Constant change
“It’s not possible to see the same river twice”
Recognizing that change is…
Continuous ComplexNon-linearNot controllableMultidirectional
Keep your eyes wide open…
Being attentive along the journey is as important as the destination
opportunities for P,M,E and learning
beginning
Life cycle of the program
endduring
objectives, inputs,
Activitiesoutputs
results,impact
Often Dealing with social systemsNon-linear process
Complex relationships & interactionsUnpredictable nature of human behaviour
Unknown factors & actors
Impact(changes in
environmental or social conditions)
Objectives(activities, products)
changes in behaviour
Life cycle of the program
different approaches
Logical Frame
Analysis
Results based Mgt
2. Sphere of influence
Inputs Activities Outputs Outcomes Impact
Sphere of control
Sphere of interest
Sphere of influence
Influence of local actors increases
project influence decreases
Relative Influence Along the Results Chain
Inputs Activities Outputs Outcomes Impact
Sphere of control Sphere of interestSphere of influence
Behavioural Changes
3. Focus on boundary partners
Key concept is « boundary partners »
The individuals, groups, and organizations you work with directly and anticipate opportunities for influence
An actor-centered approach to development planning and M&E
“Development is essentially about people (groups and organisations) relating to each other and their environment.”
Although a programme can influence the achievement of outcomes, it cannot control them because ultimate responsibility rests with the people affected.
“Outcome Mapping centred around programme actors and their relationship in development programmes.”
A programme can not control change, it can only influence and contribute to changes at the level of its boundary partners
Implementing team
Boundary partner 1
Boundary
partner 2
Boundary partner 3
Beneficiary 1 Beneficiary 2
Beneficiary 3
Sphere of Control
Sphere of influence
Sphere of interest
4. Outcomes as behavioural changes
Outcome mapping focuses on outcomes as changes in behaviour of those with whom the program or project is working directly (boundary partners)
Why behavioural change?
For each change in state (e.g. security of land for marginalized groups) there are always correlating changes in behaviour of certain people and groups.
Assessing changes in state – as often in logframes – not necessarily provide the kind of information that programmes need to improve their performance and relevance.
Development is done by and for people.
Principles of use Flexibility
OM needs to be adapted to use in your specific context. Not a fixed route but a guide for the journey we take.
Participatory OM implies dialogue and collaboration with partners. We co-create the ´map´ with our partners.
Evaluative thinking Fosters a reflective practice, organisational & social
learning.
principles of use
Flexible: Can be adapted to use & context (Not a fixed route but a guide for the journey we
take)
Complementary: combine with other methods
principles of use
Participatory: seeks dialogue and collaboration with partners (We co-create the ´map´ with our partners)
Evaluative: promotes culture of reflection, results oriented thinking, and social & organizational learning
where is the map?
✓ OM is a guide to the journey we take with our partners. We co-create the map.
✓ It focuses on the intention and what happens along the way
✓ The map is not the territory, it shows the route taken
The outcome mapping framework
Step 1: Vision
stage 1
Intentional DesignIntentional Design
I have a dream!
Martin Luther King, Jr.August 28, 1963
The vision
Description of the universe in terms of human, social, economic or environmental conditions
Large-scale development-related changes
Describes the ideal world
Beyond program's capability
Functions as a lighthouse – something to guide movement
…and a star – something big to aim for
Written in present tense
Example vision statement Education in Zimbabwe is reoriented towards
critical thinking, action competence and responsible behaviour by individuals and groups to achieve sustainable living in a healthy environment. In this way, the Zimbabwean community is empowered to make informed individual and collaborative decisions, which ensures continual effective environmental management.
vision in graphic form, Nagaland (India)
??vision facilitation questions
Imagine that, 5-10 years from now, the program has been extremely successful. Things have improved beyond your most ambitious dreams.
• What changes have occurred?
• What (& how) are your intended beneficiaries doing?
• What are your partners doing?
• Describe the better world you are seeking.
Step 2: Mission
stage 1
Intentional DesignIntentional Design
The mission is that “bite” of the vision statement on which the
project is going to focus.
The Mission
Mission statement
Describes how the program will contribute to the Vision, including its main working areas and the partners with whom it will implement them.
Written in future tense - as something the program will do
Example Mission StatementIn support of this vision, St2eep will create an enabling environment
for sustained EE implementation in Secondary Teachers’ Colleges through the encouragement of active learning processes which promote participation, critical thinking, informed decision-making, action competence and responsible citizenry. St2eep will conduct on-going pre- and in-service capacity enhancement, curriculum review and implementation and strive for the reorientation of assessment of teaching and learning in line with principles of Environmental Education. St2eep will lobby for motivational strategies and policies to ensure that EE is institutionalized in the education system and encourage outreach programs through networking with schools, communities and environmental interest groups. St2eep will ensure that graduates of the Secondary Teachers’ Colleges are able to implement EE in their teaching.
• What areas do you need to work in?
• What do you need to do in these areas?
• Who can you work with?
• How will you stay effective, efficient, and relevant?
??Mission
facilitation questions
Summary
✓ about the future✓ observable✓ idealistic ✓ not about the program
✓ feasible
✓ identifies activities and relationships
✓ about the program
Vision Mission
Exercise: Vision and Mission
step 3: Boundary Partners
stage 1
Intentional DesignIntentional Design
boundary partners
Those individuals, groups, and organizations with whom the program
✓ interacts directly to effect change✓ anticipates opportunities for
influence
✓ engages in mutual learning
Program
= boundary partners
• has direct control over its inputs, activities,... in working with the boundary partners, but
• can not control change at level of its boundary partners and beneficiairies. The ultimate responsibility rest with the people affected.
• hopes to have direct influence / contribute to changes at the level of its boundary partners
• can only indirectly influence change at the level of ultimate beneficiaries (impact)
a programme…..
boundary partners have boundary partners
program
program’s bp
bp’s bp
strategic partners
selected on the basis of their contribution to the mission a person or group with whom the program works directly to achieve the mission, without necessarily wanting to change the partner’s behavior
ExamplesOther donors
Other organisations doing similar work
Media
St2eep impl teamEE coordinators and
VVOB facilitators
Vision
Mission
Beneficiaries: Colleges, Lecturers & staff, Student Teachers, Pilot schoolsCommunities, ….
BP3 EE Steering teams
( EE Coordinator)
BP1Ministry of Higher & Tertiary Education
BP2College
administrations
Strategic partnersMin. Of Education
Min. Of EnvironmentSADC REEP
ZWEECFUniversities
NGO’s ...
Funding Organisations
VVOBSADC REEP
...
Outcome Challenge 3
BP4Dept of Teacher
Education (UZ)
Outcome Challenge 4
Outcome Challenge 2
Outcome Challenge 1
St2eep project 2005-2008
BP5Ministry of
Education, S., C.
Outcome Challenge 5
✓In which individuals, groups, or organizations is your program trying to encourage change as a contribution to the vision?
✓ With whom will you work directly?
✓ Are you choosing boundary partners because you want to influence the ways they help or influence others?
??boundary partners facilitation questions
Boundary partner exercise
Worksheet to be supplied
Step 4: Outcome Challenges
stage 1
Intentional DesignIntentional Design
what indicators do you use for:
Greater awareness…Empowered women…Reduced conflict…Increased collaboration…Gender sensitivity…Equal access…Active participation… Poverty alleviation…Strengthened capacity…
In OM you describe what the BP is DOING DIFFERENTLY and those are your ‘results’.
?
✓ describes the ideal behavioural changes, relationships, actions (professional practices) of a boundary partner for it to contribute to the vision
outcome challenge
✓ one for each boundary partner
✓ describes the boundary partners’ ideal (but possible) contribution to the vision
✓ written like this: “The program intends to see [boundary partner] who [description of behaviors in the active present tense]”
outcome challengecharacteristics
✓ Ideally, in order to contribute to the vision, how would the boundary partner be behaving?
✓ With whom would they be interacting?
✓ What would they be doing to contribute maximally to the vision?
??outcome challenge facilitation questions
Example outcome challenge statement
St2eep intends to see college administrators who continually support EE activities within the
colleges by appointing full–time EE coordinators and providing office space, transport, equipment, finances, communication and other resources for
successful EE implementation. These college administrators are cooperating with St2eep by incorporating EE in the college strategic plan and encouraging lecturers for EE orientation,
workshops and programs. They are attending EE organised functions. They are actively supporting policy development and implementation through the recognition that EE is part and parcel of the
curriculum and college functions.
Outcome challenge exercise Worksheet to be supplied
Program FrameworkVision:
Mission:
Boundary Partner:
Boundary Partner:
Boundary Partner:
Boundary Partner:
Outcome Challenge:
Outcome Challenge:
Outcome Challenge:
Outcome Challenge:
Step 5: Progress Markers
stage 1
Intentional DesignIntentional Design
progress markers
✓ Describe changes in actions, activities and relationships leading to the ideal outcome
✓ Articulate the complexity of the change process
✓ Can be monitored & observed
✓ Enable on-going assessment of partner’s progress (including unintended results)
progress markers (ladder of change)
Expanding influence, helping others, sharing expertise
Actively engaged, learning, commitment
Early encouraging response to program, initial engagement
Love to see
Like to see
Expect to see
progress markers are graduated
✓move from initial to more profound
changes in behaviour
✓show transformation in a single boundary partner
✓more informative than a single indicator
Example Progress Markers
The Project Expects to See College Administrators: Authorising lecturers to participate in EE activities. Providing office space and equipment for the EE coordinators.
The program would Like to See College Administrators : Providing transport, finances and other resources for EE activities. Including and positioning EE, high on the agenda of staff meetings.
The program would Love to See College Administrators : Incorporating EE in the college strategic plan. Appointing full time EE coordinators.
Do not require consultant to write quantifiable indicators
Show directionality of change
Support learning function
Can be monitored & observed from the start of the program
Permit on-going assessment of partner’s progress (including unintended results)
Provide basis for dialogue with partners
some advantages of progress markers
How many progress markers?
It depends.
Suggestion: total of 15, with most occurring in the “like to see” range.
Life of Program
Expect4
Like8
Love3
Exercise Progress Markers
Identify 3 strips that are not progress markers Divide remaining strips between
Expect to see Like to see Love to see
Outcome challenge and progress
markers
Boundary Partners
...
Programme Team
Interventions/strategies??
Step 6: Strategy Maps
stage 1
Intentional DesignIntentional Design
Strategy Maps
- Which intervention strategies are we developing in order to support/influence the boundary partners towards the desired changes?
- Outcome Mapping provide a framework to develop a mixed set of strategies to assure greater potential for success:1. Strategies directly aimed at individuals, teams, organisation of the
boundary partner2. Strategies aimed at the environment in which the boundary
partner is working
- Programme interventions most likely change during the course of an initiative
causal persuasive supportive
Iaimed at individual boundary partner
Eaimed at boundary partner’s
environment
6 types of support strategiescausal persuasive supportive
Iaimed at individual boundary
partner
directInfluence
(e.g. funding, prepare a report, …)
arouse new thinking;
build skills, capacity
(e.g. skill enhancement,
methodological workshops, training)
on-going support
supporter/mentor who guides change over
time,involvement is more frequent and sustained,
nurturing for self-sufficiency
(e.g. program member who provides regular guidance and input, coaching, expert, …)
Eaimed at boundary partner’s
environment
alter the physical or regulatory
environment(e.g. technical transfer, policy change, internet access, terms of reference, rules,
guidelines)
broad information
dissemination/access to new
info(e.g. Radio, TV,
website, publications, conferences, workshops)
create / strengthen
peer networkspartners working
together and collectively
supporting each other)
(e.g. research network, …)
causal persuasive supportive
Iaimed at individual boundary partner
How to produce an “immediate effect in the
BP”?
How to build capacity, to
influence new ways of
thinking or doing things?
How will sustained support,
guidance or mentoring be
provided?
Eaimed at boundary partner’s
environment
How to alter the physical or
policy environment?
How to influence the quality, availability,
accessibility of information?
How to influence the media?
What networks or relationships
will be established,
strengthen or utilized?
strategies: facilitation questions ??
I-2I-3I-1
E-1 E-2 E-3
causal persuasive supportive
Iaimed at individual boundary partner
•Fund research projects
•Run workshops on quantitative and
qualitative methods•Offer internet
reesarch courses•Coordinate training
on participatory methods
•Offer gender sensitivity training to those working with
HIV-infected women
•Hire a professional writer to work on
dissemination strategies with
researchers•Hire a fundraiser to help identify donors
and develop a fundraising strategy
•Provide ongoing technical assistance
Eaimed at boundary partner’s
environment
•Provide computers and internet access•Include work with
women and youth as a condition for the
grant
•Organise regional conferences for
HIV/Aids research community
•Develop internet site with tools and
methods•Publish ‘special
paper’ series
•Establish fromal mentorship program
that partners experienced and
junior researchers.•Facilitate the
development of an electronic HIV/AIDS
policy research network
Example strategy map
value of strategy maps
✓ Clarify what the project is doing, with whom and why
✓ Articulate the range, mix and logic of the strategies
✓ Detect strategy gaps and over extension of resources
✓ Encourage multiple interventions to increase effectiveness
Outcome challenge and progress
markers
Boundary Partners
Programme Team
Organisational practices?Strategies &
activities
Step 7: Organisational Practices
stage 1
Intentional DesignIntentional Design
Organisational practices
Supporting change in boundary partners requires that the program/organisation be able to change and adapt as well
Move beyond being efficient & effective (operational capacities) towards being relevant (adaptive capacities)
Therefore OM incorporates an ongoing assessment and reflection of the own practices & capacities into the design of the programme
what you do as an organization to
✓ keep learning
✓ foster creativity & innovation
✓ seek better ways to assist your partners
✓ maintain your niche
✓ maintain high level support
✓ build relationships
organizational practices
1. Prospecting for new ideas, opportunities, and resources
2. Seeking feedback from key informants
3. Obtaining the support of your next highest power
4. Assessing and (re)designing products, services, systems, and procedures
…organizational practices
5. Checking up on those already served to add value
6. Sharing your best wisdom with the world
7. Experimenting to remain innovative
8. Engaging in organizational reflection
Exercise – organisational practices
Worksheet to be supplied
85
mission vision
Girls & Women
Community Leaders
Women’s Self Help Groups
Families
PoliceState NGOs
Banks
Public Health Clinics
Strategic Partners
Strategies
Project’s Outcomes
Boundary Partners
BP’s outcomes
BAIF
Overview of Stage 1
Outcome Mapping: Main Elements
Boundary Partners
Vision
Mission
O.C. OutcomeChallenge
O.C.
Progress Markers
Strategies
Organizational Practices
project team
Interest
Influ
ence
Contro
l
stage 2
Outcome and Performance MonitoringOutcome and Performance Monitoring
4. Monitoring & Evaluation in OM
« Not everything that counts can be counted.
and not everything that can be counted,
counts. »
Albert Einstein
Monitoring and evaluation
An outcome mapping based M&E system looks both at the processes that are going on within the programme and how the programme (through its processes) is contributing to certain outcomes (no causal relationship)
M&E – what’s the difference?Monitoring Ongoing, continuous, more regular Internal activity Responsibility of project staff and management Continuous feedback to improve programme & report on
performance
Evaluation Periodic and time bound Can be internal, often external Responsibility of evaluator with staff and management Periodic feedback A more in depth study and assessment of an issue
Outcome challenge and progress
markers(outcome journal)
Strategies & activities(Strategy Journal)
Organisational practices
(Performance Journal)
Boundary Partner
Program
Planning and monitoring opportunities in OM
Change in behaviour
Internal performance
contextual information
Accountability & Learning: A Balancing Act
Changing the M&E perspectiveOld M&E: M&E is mainly for external
funding body Concentrates on activities
and financial reporting Lots of data and little
analysis (big reports!) Little learning takes place Little stakeholder
participation M&E is boring and not very
useful for project staff and beneficiaries
New M&E M&E is for everybody
involved in project Also look at effects and
outcomes and why there is success and failure
Learning is a key function of M&E
Good analysis on how to improve project
Lots of stakeholder participation
M&E is active, interesting and useful
-> Evaluative thinking throughout the working cycle
Accountability & Learning: A Balancing Act
Exercise Monitoring tools Monitoring Journals
Monitoring Plan
Monitoring Priority
Who will use
Purpose When is it needed
Who will
collect & how
How often
Proposed Tool
Strategies Strategy Journal
Org’l Practices Performance Journal
Boundary Partner Outcomes
Outcome Journal
Three additional questions:1. How will we learn from the monitoring information, how do we make sure
lessons learned feed into future planning?2. How is the reporting going to be done?3. Who gets the report?
Exercise – Monitoring Planning See worksheet
Learning & Reporting(Info to users/events/audiences)
Clarifying Intent(Both interventions & outcomes)
M&E Planning(Choosing what to track)
M&E Implementation (Data collection & interpretation)
Useful M&E Depends On
104
reviewing the intentional design1. Read the vision statement
Does this still reflect the program's dream?
2. Read the mission statement
Is this the greatest contribution our program can make? Have we been doing this? Why? Why not? Should we add anything or take anything away?
3. Review boundary partners
Is this who we are working with directly? Do we need to add or drop any boundary partners?
4. Review outcomes Do these accurately reflect transformations in our boundary partners as they strengthen their contributions to the vision?
5. Review progress markers
Was the change process we set out accurate and useful? What now needs to be added or taken out?
6. Review strategies What did we plan to do? Have we implemented these activities? Do we need to add, remove any?
7. Review organizational practices
Are we doing everything we can to maintain & enhance our capacity to support our partners?
stage 3
Evaluation PlanningEvaluation Planning
« Evaluative thinking is a learned behaviour that you have to practice and improve at on an on-going basis! »
Michael Quinn Patton
Step 12: Evaluation Plan A short description of the main planning
elements of an evaluation study
For Whom? What? Where? How? By Whom? When? How Much?
Elements of an Evaluation Plan
Who Will UseEvaluation,
How, & When?
Questions to be
Answered
Info. Sources
EvaluationMethods
Who WillConduct &
Manage theEvaluation?
Date(Start &Finish)
Cost
Evaluation Issue:
Setting Evaluation Priorities
Criteria to select evaluation issue include:
Learning needs Accountability or Reporting requirements Partners’ information needs
Guiding Principles for Evaluation
Balancing learning & accountability
Action oriented
Each study is unique
Process not only product
Build capacity at all levels through participation
Points to Remember:
1. Stay in touch with reality.
2. Celebrate the contributions of your team and your partners.
3. Be “idealistic realists.”
Social change has multiple, interacting causes. Trying to determine the “impact” achieved by a single intervention may be a waste of time and resources.
Seek, instead, to deepen understanding of how the actors and factors influence each other in relation to the program’s vision.
OM’s underlying message:
ongoing OM applicationshttp://www.outcomemapping.ca
Ghana Korea Switzerland
Kenya Ecuador United Kingdom
Madagascar Mexico Brussels
Mali Egypt Netherlands
Namibia Bhutan Australia
Uganda Honduras Sri Lanka
Zimbabwe Guatemala India
http://web.idrc.ca/en/ev-27705-201-1-DO_TOPIC.html
examples of use:
for more information and to share your experiences
www.outcomemapping.ca www.idrc.ca/evaluation
www.mapeodealcances.net
Thank You Very Much