Outcome Mapping Bringing learning into development programmes 15-18 September 2009 Capetown South...

Outcome Mapping

Bringing learning into development programmes

15-18 September 2009Capetown South Africa

Robert Chipimbi Simon Hearn

Acknowledgements This presentation makes use of various

materials that were shared by various OM community members on the OM learning community website. Without being exhaustive special thanks goes to Terry Smutylo, Enrique Mendizabal, Steff Deprez, Jan Van Ongevalle, Daniel Roduner, Kaia Ambrose and many others.

´´The only real voyage of discovery exists, not in seeing new landscapes, but in having new eyes´´

Marcel Proust

Workshop objectives

1. Understand basic OM concepts & principles

2. Decide if & how OM applies to your work

3. Apply some OM tools

Before we start ...

Outcome Mapping is different from conventional PM&E models

It requires time to absorb the concepts and implications for your work

It might seem very complex at first (new concepts, new language, …)

Requires a change in thinking (new paradigm) Generates different reactions

Interest Confusion / doubt Disagreement Defense Let’s go for it! How can we ever do this? …

DON’T WORRY - ASK QUESTIONS!

Developed between 1998 and 2001 by the International Development Research Centre (IDRC) as an answer to the difficulties faced in evaluation processes of development programs:

Demonstrating results

Proving cause/effect relationships

Measuring & attributing impact

What about unintended results?

…

origins of outcome mapping

Some core principles of outcome mapping

1. Embracing complexity

2. Sphere of influence

3. Boundary partners

4. Outcomes as behavioural changes

1. Embracing Complexity

Looking at the bigger picture

See yourself as a part of an interconnected web

of relationships and systems

Constant change

“It’s not possible to see the same river twice”

Recognizing that change is…

Continuous ComplexNon-linearNot controllableMultidirectional

Keep your eyes wide open…

Being attentive along the journey is as important as the destination

opportunities for P,M,E and learning

beginning

Life cycle of the program

endduring

objectives, inputs,

Activitiesoutputs

results,impact

Often Dealing with social systemsNon-linear process

Complex relationships & interactionsUnpredictable nature of human behaviour

Unknown factors & actors

Impact(changes in

environmental or social conditions)

Objectives(activities, products)

changes in behaviour

Life cycle of the program

different approaches

Logical Frame

Analysis

Results based Mgt

2. Sphere of influence

Inputs Activities Outputs Outcomes Impact

Sphere of control

Sphere of interest

Sphere of influence

Influence of local actors increases

project influence decreases

Relative Influence Along the Results Chain

Inputs Activities Outputs Outcomes Impact

Sphere of control Sphere of interestSphere of influence

Behavioural Changes

3. Focus on boundary partners

Key concept is « boundary partners »

The individuals, groups, and organizations you work with directly and anticipate opportunities for influence

An actor-centered approach to development planning and M&E

“Development is essentially about people (groups and organisations) relating to each other and their environment.”

Although a programme can influence the achievement of outcomes, it cannot control them because ultimate responsibility rests with the people affected.

“Outcome Mapping centred around programme actors and their relationship in development programmes.”

A programme can not control change, it can only influence and contribute to changes at the level of its boundary partners

Implementing team

Boundary partner 1

Boundary

partner 2

Boundary partner 3

Beneficiary 1 Beneficiary 2

Beneficiary 3

Sphere of Control

Sphere of influence

Sphere of interest

4. Outcomes as behavioural changes

Outcome mapping focuses on outcomes as changes in behaviour of those with whom the program or project is working directly (boundary partners)

Why behavioural change?

For each change in state (e.g. security of land for marginalized groups) there are always correlating changes in behaviour of certain people and groups.

Assessing changes in state – as often in logframes – not necessarily provide the kind of information that programmes need to improve their performance and relevance.

Development is done by and for people.

Principles of use Flexibility

OM needs to be adapted to use in your specific context. Not a fixed route but a guide for the journey we take.

Participatory OM implies dialogue and collaboration with partners. We co-create the ´map´ with our partners.

Evaluative thinking Fosters a reflective practice, organisational & social

learning.

principles of use

Flexible: Can be adapted to use & context (Not a fixed route but a guide for the journey we

take)

Complementary: combine with other methods

principles of use

Participatory: seeks dialogue and collaboration with partners (We co-create the ´map´ with our partners)

Evaluative: promotes culture of reflection, results oriented thinking, and social & organizational learning

where is the map?

✓ OM is a guide to the journey we take with our partners. We co-create the map.

✓ It focuses on the intention and what happens along the way

✓ The map is not the territory, it shows the route taken

The outcome mapping framework

Step 1: Vision

stage 1

Intentional DesignIntentional Design

I have a dream!

Martin Luther King, Jr.August 28, 1963

The vision

Description of the universe in terms of human, social, economic or environmental conditions

Large-scale development-related changes

Describes the ideal world

Beyond program's capability

Functions as a lighthouse – something to guide movement

…and a star – something big to aim for

Written in present tense

Example vision statement Education in Zimbabwe is reoriented towards

critical thinking, action competence and responsible behaviour by individuals and groups to achieve sustainable living in a healthy environment. In this way, the Zimbabwean community is empowered to make informed individual and collaborative decisions, which ensures continual effective environmental management.

vision in graphic form, Nagaland (India)

??vision facilitation questions

Imagine that, 5-10 years from now, the program has been extremely successful. Things have improved beyond your most ambitious dreams.

• What changes have occurred?

• What (& how) are your intended beneficiaries doing?

• What are your partners doing?

• Describe the better world you are seeking.

Step 2: Mission

stage 1

Intentional DesignIntentional Design

The mission is that “bite” of the vision statement on which the

project is going to focus.

The Mission

Mission statement

Describes how the program will contribute to the Vision, including its main working areas and the partners with whom it will implement them.

Written in future tense - as something the program will do

Example Mission StatementIn support of this vision, St2eep will create an enabling environment

for sustained EE implementation in Secondary Teachers’ Colleges through the encouragement of active learning processes which promote participation, critical thinking, informed decision-making, action competence and responsible citizenry. St2eep will conduct on-going pre- and in-service capacity enhancement, curriculum review and implementation and strive for the reorientation of assessment of teaching and learning in line with principles of Environmental Education. St2eep will lobby for motivational strategies and policies to ensure that EE is institutionalized in the education system and encourage outreach programs through networking with schools, communities and environmental interest groups. St2eep will ensure that graduates of the Secondary Teachers’ Colleges are able to implement EE in their teaching.

• What areas do you need to work in?

• What do you need to do in these areas?

• Who can you work with?

• How will you stay effective, efficient, and relevant?

??Mission

facilitation questions

Summary

✓ about the future✓ observable✓ idealistic ✓ not about the program

✓ feasible

✓ identifies activities and relationships

✓ about the program

Vision Mission

Exercise: Vision and Mission

step 3: Boundary Partners

stage 1

Intentional DesignIntentional Design

boundary partners

Those individuals, groups, and organizations with whom the program

✓ interacts directly to effect change✓ anticipates opportunities for

influence

✓ engages in mutual learning

Program

= boundary partners

• has direct control over its inputs, activities,... in working with the boundary partners, but

• can not control change at level of its boundary partners and beneficiairies. The ultimate responsibility rest with the people affected.

• hopes to have direct influence / contribute to changes at the level of its boundary partners

• can only indirectly influence change at the level of ultimate beneficiaries (impact)

a programme…..

boundary partners have boundary partners

program

program’s bp

bp’s bp

strategic partners

selected on the basis of their contribution to the mission a person or group with whom the program works directly to achieve the mission, without necessarily wanting to change the partner’s behavior

ExamplesOther donors

Other organisations doing similar work

Media

St2eep impl teamEE coordinators and

VVOB facilitators

Vision

Mission

Beneficiaries: Colleges, Lecturers & staff, Student Teachers, Pilot schoolsCommunities, ….

BP3 EE Steering teams

( EE Coordinator)

BP1Ministry of Higher & Tertiary Education

BP2College

administrations

Strategic partnersMin. Of Education

Min. Of EnvironmentSADC REEP

ZWEECFUniversities

NGO’s ...

Funding Organisations

VVOBSADC REEP

...

Outcome Challenge 3

BP4Dept of Teacher

Education (UZ)

Outcome Challenge 4

Outcome Challenge 2

Outcome Challenge 1

St2eep project 2005-2008

BP5Ministry of

Education, S., C.

Outcome Challenge 5

✓In which individuals, groups, or organizations is your program trying to encourage change as a contribution to the vision? 

✓ With whom will you work directly? 

✓ Are you choosing boundary partners because you want to influence the ways they help or influence others?

??boundary partners facilitation questions

Boundary partner exercise

Worksheet to be supplied

Step 4: Outcome Challenges

stage 1

Intentional DesignIntentional Design

what indicators do you use for:

Greater awareness…Empowered women…Reduced conflict…Increased collaboration…Gender sensitivity…Equal access…Active participation… Poverty alleviation…Strengthened capacity…

In OM you describe what the BP is DOING DIFFERENTLY and those are your ‘results’.

?

✓ describes the ideal behavioural changes, relationships, actions (professional practices) of a boundary partner for it to contribute to the vision

outcome challenge

✓ one for each boundary partner

✓ describes the boundary partners’ ideal (but possible) contribution to the vision

✓ written like this: “The program intends to see [boundary partner] who [description of behaviors in the active present tense]”

outcome challengecharacteristics

✓ Ideally, in order to contribute to the vision, how would the boundary partner be behaving?

✓ With whom would they be interacting?

✓ What would they be doing to contribute maximally to the vision?

??outcome challenge facilitation questions

Example outcome challenge statement

St2eep intends to see college administrators who continually support EE activities within the

colleges by appointing full–time EE coordinators and providing office space, transport, equipment, finances, communication and other resources for

successful EE implementation. These college administrators are cooperating with St2eep by incorporating EE in the college strategic plan and encouraging lecturers for EE orientation,

workshops and programs. They are attending EE organised functions. They are actively supporting policy development and implementation through the recognition that EE is part and parcel of the

curriculum and college functions.

Outcome challenge exercise Worksheet to be supplied

Program FrameworkVision:

Mission:

Boundary Partner:

Boundary Partner:

Boundary Partner:

Boundary Partner:

Outcome Challenge:

Outcome Challenge:

Outcome Challenge:

Outcome Challenge:

Step 5: Progress Markers

stage 1

Intentional DesignIntentional Design

progress markers

✓ Describe changes in actions, activities and relationships leading to the ideal outcome

✓ Articulate the complexity of the change process

✓ Can be monitored & observed

✓ Enable on-going assessment of partner’s progress (including unintended results)

progress markers (ladder of change)

Expanding influence, helping others, sharing expertise

Actively engaged, learning, commitment

Early encouraging response to program, initial engagement

Love to see

Like to see

Expect to see

progress markers are graduated

✓move from initial to more profound

changes in behaviour

✓show transformation in a single boundary partner

✓more informative than a single indicator

Example Progress Markers

The Project Expects to See College Administrators: Authorising lecturers to participate in EE activities. Providing office space and equipment for the EE coordinators.

The program would Like to See College Administrators : Providing transport, finances and other resources for EE activities. Including and positioning EE, high on the agenda of staff meetings.

The program would Love to See College Administrators : Incorporating EE in the college strategic plan. Appointing full time EE coordinators.

Do not require consultant to write quantifiable indicators

Show directionality of change

Support learning function

Can be monitored & observed from the start of the program

Permit on-going assessment of partner’s progress (including unintended results)

Provide basis for dialogue with partners

some advantages of progress markers

How many progress markers?

It depends.

Suggestion: total of 15, with most occurring in the “like to see” range.

Life of Program

Expect4

Like8

Love3

Exercise Progress Markers

Identify 3 strips that are not progress markers Divide remaining strips between

Expect to see Like to see Love to see

Outcome challenge and progress

markers

Boundary Partners

...

Programme Team

Interventions/strategies??

Step 6: Strategy Maps

stage 1

Intentional DesignIntentional Design

Strategy Maps

- Which intervention strategies are we developing in order to support/influence the boundary partners towards the desired changes?

- Outcome Mapping provide a framework to develop a mixed set of strategies to assure greater potential for success:1. Strategies directly aimed at individuals, teams, organisation of the

boundary partner2. Strategies aimed at the environment in which the boundary

partner is working

- Programme interventions most likely change during the course of an initiative

causal persuasive supportive

Iaimed at individual boundary partner

Eaimed at boundary partner’s

environment

6 types of support strategiescausal persuasive supportive

Iaimed at individual boundary

partner

directInfluence

(e.g. funding, prepare a report, …)

arouse new thinking;

build skills, capacity

(e.g. skill enhancement,

methodological workshops, training)

on-going support

supporter/mentor who guides change over

time,involvement is more frequent and sustained,

nurturing for self-sufficiency

(e.g. program member who provides regular guidance and input, coaching, expert, …)

Eaimed at boundary partner’s

environment

alter the physical or regulatory

environment(e.g. technical transfer, policy change, internet access, terms of reference, rules,

guidelines)

broad information

dissemination/access to new

info(e.g. Radio, TV,

website, publications, conferences, workshops)

create / strengthen

peer networkspartners working

together and collectively

supporting each other)

(e.g. research network, …)

causal persuasive supportive

Iaimed at individual boundary partner

How to produce an “immediate effect in the

BP”?

How to build capacity, to

influence new ways of

thinking or doing things?

How will sustained support,

guidance or mentoring be

provided?

Eaimed at boundary partner’s

environment

How to alter the physical or

policy environment?

How to influence the quality, availability,

accessibility of information?

How to influence the media?

What networks or relationships

will be established,

strengthen or utilized?

strategies: facilitation questions ??

I-2I-3I-1

E-1 E-2 E-3

causal persuasive supportive

Iaimed at individual boundary partner

•Fund research projects

•Run workshops on quantitative and

qualitative methods•Offer internet

reesarch courses•Coordinate training

on participatory methods

•Offer gender sensitivity training to those working with

HIV-infected women

•Hire a professional writer to work on

dissemination strategies with

researchers•Hire a fundraiser to help identify donors

and develop a fundraising strategy

•Provide ongoing technical assistance

Eaimed at boundary partner’s

environment

•Provide computers and internet access•Include work with

women and youth as a condition for the

grant

•Organise regional conferences for

HIV/Aids research community

•Develop internet site with tools and

methods•Publish ‘special

paper’ series

•Establish fromal mentorship program

that partners experienced and

junior researchers.•Facilitate the

development of an electronic HIV/AIDS

policy research network

Example strategy map

value of strategy maps

✓ Clarify what the project is doing, with whom and why

✓ Articulate the range, mix and logic of the strategies

✓ Detect strategy gaps and over extension of resources

✓ Encourage multiple interventions to increase effectiveness

Outcome challenge and progress

markers

Boundary Partners

Programme Team

Organisational practices?Strategies &

activities

Step 7: Organisational Practices

stage 1

Intentional DesignIntentional Design

Organisational practices

Supporting change in boundary partners requires that the program/organisation be able to change and adapt as well

Move beyond being efficient & effective (operational capacities) towards being relevant (adaptive capacities)

Therefore OM incorporates an ongoing assessment and reflection of the own practices & capacities into the design of the programme

what you do as an organization to

✓ keep learning

✓ foster creativity & innovation

✓ seek better ways to assist your partners

✓ maintain your niche

✓ maintain high level support

✓ build relationships

organizational practices

1. Prospecting for new ideas, opportunities, and resources

2. Seeking feedback from key informants

3. Obtaining the support of your next highest power

4. Assessing and (re)designing products, services, systems, and procedures

…organizational practices

5. Checking up on those already served to add value

6. Sharing your best wisdom with the world

7. Experimenting to remain innovative

8. Engaging in organizational reflection

Exercise – organisational practices

Worksheet to be supplied

85

mission vision

Girls & Women

Community Leaders

Women’s Self Help Groups

Families

PoliceState NGOs

Banks

Public Health Clinics

Strategic Partners

Strategies

Project’s Outcomes

Boundary Partners

BP’s outcomes

BAIF

Overview of Stage 1

Outcome Mapping: Main Elements

Boundary Partners

Vision

Mission

O.C. OutcomeChallenge

O.C.

Progress Markers

Strategies

Organizational Practices

project team

Interest

Influ

ence

Contro

l

stage 2

Outcome and Performance MonitoringOutcome and Performance Monitoring

4. Monitoring & Evaluation in OM

« Not everything that counts can be counted.

and not everything that can be counted,

counts. »

Albert Einstein

Monitoring and evaluation

An outcome mapping based M&E system looks both at the processes that are going on within the programme and how the programme (through its processes) is contributing to certain outcomes (no causal relationship)

M&E – what’s the difference?Monitoring Ongoing, continuous, more regular Internal activity Responsibility of project staff and management Continuous feedback to improve programme & report on

performance

Evaluation Periodic and time bound Can be internal, often external Responsibility of evaluator with staff and management Periodic feedback A more in depth study and assessment of an issue

Outcome challenge and progress

markers(outcome journal)

Strategies & activities(Strategy Journal)

Organisational practices

(Performance Journal)

Boundary Partner

Program

Planning and monitoring opportunities in OM

Change in behaviour

Internal performance

contextual information

Accountability & Learning: A Balancing Act

Changing the M&E perspectiveOld M&E: M&E is mainly for external

funding body Concentrates on activities

and financial reporting Lots of data and little

analysis (big reports!) Little learning takes place Little stakeholder

participation M&E is boring and not very

useful for project staff and beneficiaries

New M&E M&E is for everybody

involved in project Also look at effects and

outcomes and why there is success and failure

Learning is a key function of M&E

Good analysis on how to improve project

Lots of stakeholder participation

M&E is active, interesting and useful

-> Evaluative thinking throughout the working cycle

Accountability & Learning: A Balancing Act

Exercise Monitoring tools Monitoring Journals

Monitoring Plan

Monitoring Priority

Who will use

Purpose When is it needed

Who will

collect & how

How often

Proposed Tool

Strategies Strategy Journal

Org’l Practices Performance Journal

Boundary Partner Outcomes

Outcome Journal

Three additional questions:1. How will we learn from the monitoring information, how do we make sure

lessons learned feed into future planning?2. How is the reporting going to be done?3. Who gets the report?

Exercise – Monitoring Planning See worksheet

Learning & Reporting(Info to users/events/audiences)

Clarifying Intent(Both interventions & outcomes)

M&E Planning(Choosing what to track)

M&E Implementation (Data collection & interpretation)

Useful M&E Depends On

104

reviewing the intentional design1. Read the vision statement

Does this still reflect the program's dream?

2. Read the mission statement

Is this the greatest contribution our program can make? Have we been doing this? Why? Why not? Should we add anything or take anything away?

3. Review boundary partners

Is this who we are working with directly? Do we need to add or drop any boundary partners?

4. Review outcomes Do these accurately reflect transformations in our boundary partners as they strengthen their contributions to the vision?

5. Review progress markers

Was the change process we set out accurate and useful? What now needs to be added or taken out?

6. Review strategies What did we plan to do? Have we implemented these activities? Do we need to add, remove any?

7. Review organizational practices

Are we doing everything we can to maintain & enhance our capacity to support our partners?

stage 3

Evaluation PlanningEvaluation Planning

« Evaluative thinking is a learned behaviour that you have to practice and improve at on an on-going basis! »

Michael Quinn Patton

Step 12: Evaluation Plan A short description of the main planning

elements of an evaluation study

For Whom? What? Where? How? By Whom? When? How Much?

Elements of an Evaluation Plan

Who Will UseEvaluation,

How, & When?

Questions to be

Answered

Info. Sources

EvaluationMethods

Who WillConduct &

Manage theEvaluation?

Date(Start &Finish)

Cost

Evaluation Issue:

Setting Evaluation Priorities

Criteria to select evaluation issue include:

Learning needs Accountability or Reporting requirements Partners’ information needs

Guiding Principles for Evaluation

Balancing learning & accountability

Action oriented

Each study is unique

Process not only product

Build capacity at all levels through participation

Points to Remember:

1. Stay in touch with reality.

2. Celebrate the contributions of your team and your partners.

3. Be “idealistic realists.”

Social change has multiple, interacting causes. Trying to determine the “impact” achieved by a single intervention may be a waste of time and resources.

Seek, instead, to deepen understanding of how the actors and factors influence each other in relation to the program’s vision.

OM’s underlying message:

ongoing OM applicationshttp://www.outcomemapping.ca

Ghana Korea Switzerland

Kenya Ecuador United Kingdom

Madagascar Mexico Brussels

Mali Egypt Netherlands

Namibia Bhutan Australia

Uganda Honduras Sri Lanka

Zimbabwe Guatemala India

http://web.idrc.ca/en/ev-27705-201-1-DO_TOPIC.html

examples of use:

for more information and to share your experiences

www.outcomemapping.ca www.idrc.ca/evaluation

www.mapeodealcances.net

Thank You Very Much