Monitoring and Evaluation Impact

8/2/2019 Monitoring and Evaluation Impact

1/22

This is the second guidance note in a our-part series o notes related to impact evaluation developed

by InterAction with nancial support rom the Rockeeller Foundation. The other notes in this series are:

Introduction to Impact Evaluation; Introduction to Mixed Methods in Impact Evaluation; and Use of

Impact Evaluation Results. The complete series can be ound on InterActions website at:

www.interaction.org/impact-evaluation-notes.

Impact Evaluation Notes No. . April

LINKING MONITORING AND EVALUATIONTO IMPACT EVALUATION

Burt Perrin
http://www.interaction.org/impact-evaluation-noteshttp://www.interaction.org/impact-evaluation-notes


2/22

Photo: Manuel Meszarovits


3/22

| Linking Monitoring and Evaluation to Impact Evaluation |

Contents

Introduction

. How can monitoring and other orms o evaluation support impact evaluation?

1.1. Main characteristics o monitoring, evaluation, and impact evaluation 21.2. How M&E can contribute to impact evaluation 6

. How to build impact evaluation into M&E thinking and practices

2.1. Articulate the theory o change 7

2.2. Identiy priorities or undertaking impact evaluation 10

2.3. Identiy inormation/data needs 10

2.4. Start with what you have 13

2.5. Design and implement the impact evaluation, analyze and interpret the ndings 14

2.6. Use the ndings 152.7. Review, refect, and update 15

. Engaging all parts o the organization

3.1. M&E: A core management unction requiring senior management leadership and support 16

3.2. An active role or program sta is required 16

Summary

Reerences and Other Useul Resources 8

Annex Contribution analysis 9


4/22

| Linking Monitoring and Evaluation to Impact Evaluation | | |

IntroductionThe purpose o nongovernmental organizations (NGOs) and other

development actors is to help people improve their lives, communi-

ties, societies and environments. NGOs are typically interested in

contributing to social change such as increased justice or economic

and social security while recognizing that these represent long-term

outcomes that go beyond the achievement o short-term results.

As Guidance Note 1 indicates, that is what im-

pact and impact evaluation are all about: sys-

tematically and empirically identiying the eectsresultingrom an intervention, be they intended or

unintended, direct or indirect. Impacts are usually

understood to occur later than and as a result

o intermediate outcomes. There is an increasing

realization that good intentions are not enough.

Impact evaluation goes beyond considering what

agencies are doing to what happens as a result of

these activities, and the extent to which these inter-

ventions are indeed making a dierence in the lives

o people and the conditions in which they live.

Most NGOs engage in a variety o monitoring and

evaluation (M&E) activities. The extent o these ac-

tivities can vary considerably. In many cases, what

goes under the rubric o M&E consists mainly o

monitoring and reporting, although organizations

are increasingly also engaged in a variety o evalu-

ation activities, examining various aspects o their

perormance. Yet there is signicantly less atten-

tion to the evaluation o impact.

This guidance note will illustrate the relationship

between routine M&E and impact evaluation in

particular, it will indicate how both monitoring and

evaluation activities can support meaningul and

valid impact evaluation, and even make it possible.

Impact evaluations are typically external, carried

out in whole or in part by an independent expertrom outside an agency.

1Nevertheless, M&E has a

critical role to play, such as:

Identiying when and under what circum-

stances it would be possible and appropriate

to undertake an impact evaluation.

Contributing essential data to conduct an im-

pact evaluation, such as baseline data o vari-

ous orms and inormation about the nature o

the intervention. Contributing necessary inormation to inter-

pret and apply ndings rom impact evalua-

tion. This includes inormation about context,

and data like the quality o implementation,

needed to understand why given changes have

or have not come about and what we can do

to make our eorts even more eective in the

uture.

Section 1 o this guidance note discusses some o

the characteristics o impact evaluation and how

For example, see the USAID Evaluation Policy, January 2011. Nev-ertheless, impact evaluations can and sometimes are also carriedout internally, or in combination. Guidance Note 1 has more to sayabout who should conduct an impact evaluation.


5/22


this diers rom monitoring and rom other orms

o evaluation, and more specically how these

dierent but complementary approaches can be

linked to support meaningul impact evaluation.

Section 2 provides guidance and ideas about the

various steps involved and approaches that can beused to maximize the contribution o routine M&E

to impact evaluation.

One primary audience or this guidance note is

M&E staffin NGOs, to assist them in identiying

steps that they can take to better link M&E activi-

ties to impact evaluation. This note may also be

o interest to sta in other types o organizations,

such as oundations.

In addition, this note will also be relevant or NGO

management, as well as or program staff. Why?

First, as noted in Section 3, there is little point in

conducting impact evaluation unless the entire or-

ganization thinks in terms o impact and continual

improvement, questioning how to improve on

what is being done. The organization must be will-

ing to act on implications rom impact evaluation

and other orms o evaluation.

Second, M&E is not something to be let just to

evaluation specialists. Program sta can and must

play a very active role in collecting data and record-

ing their observations and experiences, which can

help ensure that impact evaluations are address-

ing the right questions and accurately refect what

programs are doing.

1. How can monitoring and other orms o

evaluation support impact evaluation?

Monitoring and evaluation can make essential con-

tributions to impact evaluation. Indeed, meaningful

impact evaluation2

is not possible without signi-

cant support rom an organizations regular M&E

activities. While the scope o this note is too o-

cused to discuss the nature and diversity o moni-

toring and evaluation, it is important to recognize

some signicant dierences between monitoring

and evaluation, which make dierent contribu-tions to impact evaluation. Thus, it is helpul to

consider some basic characteristics, including di-

erences and opportunities or complementarities,

beore identiying more specically how M&E

can contribute to impact evaluation.

1.1. Main characteristics o monitoring, evaluation,

and impact evaluation

The ollowing table summarizes some o the most

common and generally agreed-upon characteristicso monitoring, evaluation and impact evaluation.

This table should be viewed with care. As some o

the wording (e.g., typical, can, generally) and

the text below suggests, there is oten some varia-

tion in how these concepts are viewed and applied.

Monitoring generally involves tracking progress

with respect to previously identied plans or objec-

tives, using data easily captured and measured on

an ongoing basis. While monitoring most re-quently makes use o quantitative data, monitoring

qualitative data is also possible and some agencies

do this regularly.

Monitoring is carried out or a variety o dierent

purposes, generally having little to do with impact

evaluation. Some o the most requent reasons or

monitoring include:

Internal use by project managers and sta to

Impact evaluation that is meaningul: would appropriatelyrepresent the intervention in question and its context; is valid inall respects (e.g., providing or both internal and external validity);and, above all, provides useul inormation that can help inormuture directions.


6/22


better understand and track how things are

proceeding, mainly to identiy i the project ison target or not. This includes tracking data

on what services actually are being provided,

the quality o services being provided, who

is being served, and related considerations.

Monitoring data can also oten serve as an

early warning system, and in the case o nega-

tive or unexpected ndings may suggest the

need to consider a change in approach while

the project or program is still underway.

Internal organizational use at the regional,

national and/or international HQ level so the

agency can track a projects or activitys status

against plans and expectations; or planning

and management purposes; and to address

accountability needs o the agencys board,

unders and the public. Addressing external requirements or compli-

ance and control, such as donor demands or

reporting and accountability.

Monitoring can also take other orms. Rather than

ocusing specically on what the organization is

doing, or example, monitoring could include citi-

zens report cards with respect to public services,3

advocacy services o many organizations that

may track corruption practices, environmental ap-

proaches, employment o child workers in local

business practices, etc.

Further reading: Samuel Paul. Holding the State to Account: Citi-zen Monitoring in Action. Bangalore: Public Aairs Centre, 2002.

Monitoring, Evaluation and Impact Evaluation: Some Basic Characteristics

Monitoring Evaluation Impact Evaluation

Periodic, using data gath-ered routinely or readily ob-

tainable, generally internal,usually ocused on activitiesand outputs, although indi-cators o outcome/impactare also sometimes used

Assumes appropriatenesso program, activities, objec-tives, indicators

Typically tracks progressagainst a small numbero preestablished targets/

indicators

Usually quantitative

Cannot indicate causality

Dicult to use by itsel orassessing impact

Generally episodic, otenexternal

Goes beyond outputs to as-sess outcomes

Can question the ratio-nale and relevance o theprogram, objectives andactivities

Can identiy both unintendedand planned eects

Can address how andwhy questions

Can provide guidance oruture directions

Can use data rom dierentsources and rom a widevariety o methods

A specic orm o evaluation

Sporadic, inrequent

Mostly external

Generally a discrete researchstudy

Specically ocused on at-tribution (causality) in someway, most oten with acounteractual

Generally ocused on long-term changes, such as in

the quality o lie o intendedbeneciaries

Needs to take into accountwhat was actually done (e.g.,through M&E) as well asidentiy impacts


7/22


Some denitions o monitoring:

A continuing unction that uses systematiccollection o data on specied indicators toprovide management and the main stake-holders o an ongoing development interven-

tion with indications o the extent o progressand achievement o objectives and progressin the use o allocated unds. (OECD/De-velopment Assistance Committee [DAC]Glossary)

The tracking o project outputs and outcomesas indicators o project eectiveness, or theextent to which the project achieves its statedobjectives. (USAID)

While these are all legitimate and important rea-sons or monitoring, none are concerned with

contributing to impact evaluation. As a result,

oten the types o data that are collected, and the

ways in which these are kept and reported, are

dicult to use or evaluation. As the box to the

right suggests, one needs to plan in advance in

order or monitoring data to be useul or impact

evaluation. In ideal circumstances, those con-

ducting an impact evaluation can contribute to

the design and structuring o an interventionsmonitoring system.

At the same time, it is important to acknowledge

the limitations o monitoring. Monitoring mainly

tracks progress against predened objectives and

indicators, and assumes these are appropriate.

But monitoring alone is insucient or drawing

conclusions about attribution, or or identiying

the reasons why changes have or have not taken

place (such as the extent to which these changes

are a result o the intervention or due to other

causes). It is also usually unable to identiy unin-

tended eects, gaps in service, etc. For this, one

usually needs evaluation.

An all too common scenario: an impact evalu-ation planned at the end o a project ndsthat no baseline inormation or other appro-priate data has been collected or appropriatelydisaggregated to demonstrate changes oreven provide necessary inormation about

program activities, who has been served, andother needed inormation. This leaves no wayto identiy impact or attribute it in any way tothe program.

Evaluation involves systematic, evidence-based

inquiry that can describe and assess any aspect

o a policy, program or project. Evaluation uses a

wide variety o both quantitative and qualitative

methods, providing more comprehensive inorma-

tion about what is taking place, why, and whether itis appropriate or not, and to provide guidance or

uture directions.

Evaluation can be carried out or many dierent

purposes and take a variety o orms. Some o

the ollowing types o evaluation (the list is not

inclusive there are also many other evaluation

approaches) may contribute to impact evaluation

under certain circumstances.4

Needs assessments involve assessing or

evaluating the needs or problem situation,

oten prior to the initial development o the

project design. Such assessments requently

identiy ways in which these needs can be

addressed.

Process (or implementation) evaluations

describe the nature o the intervention as it is

The denitive resource or denitions o key terms used in de-velopment evaluation, such as those used in this note, is: OECD/Development Assistance Committee. Glossary of Key Terms inEvaluation and Results Based Management, http://www.oecd.org/dataoecd/29/21/2754804.pd. Also see: Michael Scriven. EvaluationThesaurus. 4th Ed. Sage Publications, 1991. USAIDs EvaluationPolicy groups all orms o evaluation other than impact evaluationunder the category o perormance evaluation.
http://www.oecd.org/dataoecd/29/21/2754804.pdfhttp://www.oecd.org/dataoecd/29/21/2754804.pdfhttp://www.oecd.org/dataoecd/29/21/2754804.pdfhttp://www.oecd.org/dataoecd/29/21/2754804.pdf


8/22


actually implemented in practice. To a certain

extent, monitoring may be able to provide

data about program activities that can be

useul or impact evaluation. However, inter-

ventions are rarely applied exactly as initially

intended and requently change over time,

oten or good reasons. It can be surprisinglydicult to determine what is actually taking

place, who is being served, in what ways or

to what extent, and what else is going on.

Process evaluation can go into more depth

than is possible with monitoring, oten explic-

itly using questions arising rom monitoring

as a starting point.

Programs requently need to change, some-times considerably, in response to inorma-tion rom needs assessments and other ormso eedback rom monitoring and evaluation.For example, a program targeting orphansand vulnerable children initially assumedthat children living alone would be most vul-nerable, but quickly learned that there werehouseholds headed by adults where childrenwere even more vulnerable.

Without understanding what the programreally is, even the most sophisticated and sta-

tistically rigorous impact evaluation will have

little meaning. Evaluations that clearly outline

the program, along with reasons or diver-

gence rom original expectations, can provide

invaluable inormation to help understand

how the programs interventions might have

made an impact (or, perhaps, to indicate

some challenges in a programs implemen-

tation or some o its underlying assump-

tions that may impede its ability to make an

impact). For example, i a programs impacts

were limited, data rom process evaluation

can help ascertain i this was because o a

problem in the theory o how the program

was expected to work, or due to limitations in

how it was implemented.

Some denitions o evaluation:

The systematic and objective assessment oa planned, on-going or completed project,program or policy, its design, implementationand results. (OECD/DAC Glossary)

The systematic collection and analysis oinormation about the characteristics andoutcomes o programs and projects as a basisor judgments, to improve eectiveness, and/or inorm decisions about current and utureprogramming. (USAID)

Formative evaluation, carried out partwaythrough implementation, is intended to

improve perormance during the subsequent

steps o a program or project. This can help

identiy intermediate outcomes, at what point

(i any) the intervention seems likely to make

an impact and what else may be needed to

enhance eectiveness. Consequently, this can

make or more meaningul impact or summa-

tive evaluation5

carried out at a later date.

Organizational evaluation looks at an orga-nizations overall eectiveness, or perhaps

that o an organizational unit. Organizational

actors (e.g., governance, management,

human resources, nances, intra- and inter-

organizational relationships, etc.) oten may

have more to do with an interventions suc-

cess than its design does these actors are

essential inormation that must be taken into

account during the design and interpretation

o impact evaluation.

Summative evaluation, oten equated with impact evaluation,involves assessing the extent to which anticipated outcomes wereproduced at the end o a planned intervention and making a judge-ment as to its worth.


9/22


In addition, the OECD Development Assistance

Committee (DAC), one o the most infuential

bodies concerning development evaluation, has

identied ve basic evaluation criteria (or ques-

tions): relevance, eectiveness, eciency, im-

pact and sustainabil ity.6

Note that evaluation o

impact represents just one o these criteria. It israrely possible to conduct an impact evaluation

ocused specically on attribution without also

having undertaken other orms o evaluation to

better understand what has taken place. There is

little point in documenting that an intervention is

making an impact i it is no longer relevant or i

there may be more eective or less costly ways o

addressing the basic need, or i the intervention

and/or its results are not likely to continue or to

be sustainable.7

Guidance Note 1 notes circum-stances under which impact evaluation could be

useul.

Impact evaluation generally shares the basic char-

acteristics o other orms o evaluation, and should

be one o an NGOs evaluation approaches. How-

ever, as the table on page 3 suggests, there are

typically some signicant dierences, underscoring

the need to use other orms o M&E to make an

impact evaluation meaningul.

Guidance Note 1 observes that impacts and im-

pact evaluation are sometimes dened in dier-

ent ways. Nevertheless, an essential aspect o im-

pact evaluation concerns attribution, the linking

o documentable impacts in a cause-and-eect

One o many sources with more inormation about thesecriteria is the DAC Criteria for Evaluating Development As-sistance. This and many other DAC evaluation resources

can be ound at http://www.oecd.org/document/35/0,3746,en_21571361_34047972_31779555_1_1_1_1,00.html.

Frequently, interventions subject to impact evaluations or otherintensive research are resourced at a higher level than normalwork, in a way that is unsustainable or that can readily be scaledup or applied in other contexts.

manner to an intervention. Simply knowing that

impacts have come about because o an interven-

tion, however, is insucient. To be able to apply

the ndings rom impact evaluation in other

settings and/or to other groups o beneciaries,

one needs to know why and how the given re-

sults came about, as well as the characteristics othose who did (or did not) benet.

This is where meaningul M&E can be helpul

provided that one identies in advance, as speci-

cally as possible, the types o inormation that will

be needed and how various orms o M&E can

provide these.

1.2. How M&E can contribute to impact evaluation

As the above discussion indicates, there are manytypes o data that M&E can, indeed must, contrib-

ute to enable impact evaluation. There are our

basic categories o inormation that M&E can

contribute to impact evaluation.

1. Inormation about the nature o the interven-

tion, which is essential or impact evaluation to

causally link any documented outcomes to what

was done. Basic inormation rom routine M&E

is required about: the actual nature o servicesprovided; baseline data, as well as changes over

time, with respect to who is served (or addressed,

such as with advocacy initiatives); the number

and characteristics o beneciaries reached,

including relevant subgroups; program outputs;

and how the intervention has progressed. These

may be dierent rom what was originally expect-

ed. As previously noted, the planned interven-

tion can vary signicantly rom what was actually

implemented, either in response to eedback rom

M&E or rom changes in circumstances. The

targeted population may also change during the

lie o the program, as organizations learn more

about who should be eligible or a program or
http://www.oecd.org/document/35/0,3746,en_21571361_34047972_31779555_1_1_1_1,00.htmlhttp://www.oecd.org/document/35/0,3746,en_21571361_34047972_31779555_1_1_1_1,00.htmlhttp://www.oecd.org/document/35/0,3746,en_21571361_34047972_31779555_1_1_1_1,00.htmlhttp://www.oecd.org/document/35/0,3746,en_21571361_34047972_31779555_1_1_1_1,00.html


10/22


become better at reaching them. Without taking

this into account, any ndings rom an impact

evaluation may be meaningless.

Much o this data is quantitative in nature and

can come rom monitoring, along with qualita-

tive descriptions about the nature o the programor project. Process evaluation in particular oten

can provide more in-depth inormation about the

extent to which services (or messages, in the case

o an advocacy program) are actually reaching the

intended audiences, how these are viewed and

how they vary across dierent groups o people.

2. Inormation about the context o the interven-

tion. What other interventions and actors may

infuence whether or not an intervention canwork? Can dierent groups o people be aect-

ed dierentially (e.g., aid provided ater a disaster

disproportionately going to those best able to

access it; services and other orms o assistance

requiring people to go to a site, thereore leaving

people with mobility limitations unserved)? These

kinds o considerations must be taken into ac-

count in the design o impact evaluations.

Similarly, what infuencing actors may be specialabout the context or this intervention? For exam-

ple, the successul delivery o ood aid and sub-

sequent lives saved requently depends on things

that are well beyond the programs control, such

as logistics, weather, support (or intererence)

rom governments or terrorist organizations, etc.

This kind o inormation is essential or meaningul

interpretation o impact evaluation data, and or

identiying the ndings implications or other set-

tings or situations. These types o data are mainly

(but not necessarily) qualitative. They sometimes

can come rom monitoring, but in many cases

would require evaluation to enable more in-depth

exploration. Guidance Note 3 provides more

inormation about potential methods that can be

used to collect these and other orms o data.

3. What we already know (or at least suspect)

about impact. Sometimes there already is some

evidence or at least a strong suggestion rom

monitoring and/or evaluation that some kind ochanges may be taking place. In such cases, an im-

pact evaluation could document and conrm what

is really going on and the extent to which these

changes have been a result o program activities.

In contrast, M&E may raise questions about the vi-

ability o an impact evaluation, at least at the pres-

ent time, indicating that it might still be premature

to look or impact. Once an impact evaluation is

underway, data rom monitoring and other orms

o evaluation should o course be taken into ac-count, along with any other data collected.

4. What other evidence may inuence the de-

cision to undertake an impact evaluation. As

indicated above, evaluation inormation about an

interventions relevance, eciency and sustain-

ability should be taken into account beore setting

priorities or impact evaluations.

2. How to build impact evaluation into M&Ethinking and practices

As identied above, M&E is vital to impact evalu-

ation but its contributions are not automatic.

This section provides guidance and ideas or how

to plan and carry out M&E so that it can support

meaningul impact evaluation. These are organized

around the steps identied in the diagram below,

and discussed in the text that ollows.

2.1. Articulate the theory o change

As Guidance Note 1 indicates, a theory of change

is a model that explains how an intervention is

expected to lead to intended or observed impacts.


11/22

| Linking Monitoring and Evaluation to Impact Evaluation | | 8 |

Oten reerred to as the program theory, results

chain, program logic model or attribution logic, the

theory o change illustrates the series o assump-

tions and links identiying the presumed relation-

ships between:

Inputs (e.g., unding, sta, volunteers, tangibleand in-kind support rom others, etc.);

Activities and their immediate outputs (i.e.,

what is done, such as training sessions

conducted);

Intermediate outcomes at various levels (e.g.,

teachers trained, teachers applying what they

have learned in the classroom situation); and

The intended impact (such as reduction in

child labor, amilies and communities that

have become nancially sel sucient, im-

proved health status, ewer people dying in

emergency situations, etc.).

Guidance Note 1 has identied key elements and

various ways in which a theory o change can

improve impact evaluation. This, in turn, has some

very direct implications or M&E that can be sum-

marized under the ollowing main categories.

. Articulate thetheory o change

. Identiy prioritiesor undertaking IE

. Identiy inormation/data needs

. Start withwhat you have

. Design & implementIE, analyze & interpret

ndings

. Use the ndings(e.g., to improve

policies/programs

. Review, refect,and update


12/22


A theory o change or advocacy eorts, simi-or advocacy eorts, simi-

lar to those or direct service projects, should

identiy the sequence o intermediate steps (e.g.,

awareness, other partners agree to take comple-

mentary action, community mobilized, dialogue

with government ocials created) and other

intervening actors (e.g., activities by others) bywhich this would be expected to lead to changes

in government policies and practices, or per-

haps in community practices.

1. Identiy the stage in the trajectory o an

intervention when it would be appropriate to

undertake various orms o evaluation includ-

ing impact evaluation by identiying the various

intermediate steps and other actors that may alsoinfuence impact (such as interventions o other

allies, contextual variables, etc.). M&E may be able

to track progress regarding activities, outputs and

initial outcomes, indicating when it could be pos-

sible to consider impact evaluation. This can avoid

the all too common situation o evaluations that

attempt to measure impact prematurely, some-

times even beore a program has a chance to get

well established.

Michael Woodcock o the World Bank has em-

phasized the importance o determining the

timerames and trajectories o impact that is

reasonable to expect. It is critical to recognize that

dierent projects may ollow very dierent trajecto-

ries. As he observes, Some projects may, o their

nature, yield high initial impacts while others

may inherently take ar longer, even decades, to

show results, not because they dont work ater

three years, but because its simply how long it

takes (Woolcock 2011). It is also important to

bear in mind that an organizations sta may be

overly optimistic about the time it takes to set up

a new program, or or results to ollow. To attempt

impact evaluation prematurely is to set up a pro-

gram or ailure.

2. Describe the manner in which the intervention

is expected to work using a theory o change.

Oten reerred to as the results (or impact) path-

way, this can help suggest the types o M&E dataneeded to document the nature o the interven-

tion and how it has evolved in practice. The theory

o change can indicate which aspects o imple-

mentation need to be checked or quality, to help

distinguish between implementation ailure and

theory ailure. It also provides a basis or identiy-

ing where along the impact pathway (or causal

chain) an intervention may stop working. This type

o inormation is essential to draw a causal link be-

tween any documented outcomes or impacts andthe intervention. It is also essential to explain and

interpret the meaning and implications o impact

evaluation ndings.

3. Develop a common understanding o the

intervention by involving staf and other key

stakeholders in the development o the theory o

change. I a theory o change is developed just by

an M&E specialist relying extensively on documen-

tation, there may be no ownership o the model.It also may not refect the reality o what is taking

place. It is advisable, where possible, to involve

program, eld sta and some other key stakehold-

ers in developing the theory o change. There is a

crucial role or the M&E specialist in the process,

as an active acilitator and drawing up the actual

model. Sometimes there may be diering views

about how a program may be expected to work,

resulting in alternative theories o change. This can

suggest questions to be explored by M&E, perhaps

beore undertaking the actual impact evaluation.

In addition, i a participatory approach is taken, the

development o the theory o change can help all


13/22


participants think in outcome terms. The process

can help develop ownership and a common un-

derstanding o the programs purpose and what is

needed or it to be eective.

2.2. Identiy priorities or undertaking impact

evaluationMany actors help establish priorities or undertak-

ing impact evaluations, such as interests o key

stakeholders or donors, the availability o unding

or opportunities to collaborate with others. But a

key consideration is when it would be most likely

to contribute useul inormation that cannot be

obtained through simpler and less costly means,

and that can provide useul guidance or uture

strategies or program approaches.

M&E plays an important role in this process. The

theory o change can indicate which questions

are appropriate to address when during a pro-

gram. Monitoring and evaluation data can help to

indicate how ar interventions have progressed,

at what stage they are ready or impact evalu-

ation, and when there is sucient supporting

M&E documentation to make this possible. Start

by identiying inormation needs, then see how

much inormation rom routine M&E is alreadyavailable beore deciding to undertake an impact

evaluation. At this stage, one can assemble the

perormance story to clariy how much is already

known, pinpoint gaps in inormation and unan-

swered questions, and identiy when an impact

evaluation would be most applicable. This may

be to provide hard evidence to conrm what

is suspected, or in situations when there are real

questions about an interventions value that can-

not be answered through other means.

2.3. Identiy inormation/data needs

What types o M&E data will be needed to sup-

port impact evaluation? The simple answer is: itdepends. There is no one-size-ts-all prescription;

that would be a surere route to irrelevancy. This

is why this guidance note has emphasized the

importance o articulating the theory o change,

preerably involving program sta and other stake-

holders in the process, and using techniques such

as a realist evaluation mindset and a contribu-

tion analysis approach as described below. These

techniques will help identiy questions or evalua-

tion, which in turn can help identiy the necessarytypes o data, and M&E techniques and methods

to obtain these.

Guidance Note 1 lists examples o key evaluation

questions or impact evaluation. Section 1 o this

note identied various ways M&E data can con-

tribute to impact evaluation. But there is no magic

ormula. It is necessary to identiy, very specically,

what types o data will actually be required or

impact evaluation, and the extent to which existingmonitoring procedures and evaluation practices

may need to be modied to provide this inorma-

tion (or to indicate that this is not possible). The

ollowing table can serve as a summary checklist

o the types o inormation requirements that M&E

approaches should be able to provide in most

situations.


14/22


Baseline data

Identiy the initial state along dimensions relevant to the intended impact o the program (the theory o change should beable to assist in identiying these dimensions).

In situations where baselines have not been established, are there ways these could be recreated retrospectively?*

Nature o the program/intervention, as actually implemented

Identiy changes rom how the intervention was initially expected to be delivered, and to the extent possible, the reasonsor these modications.

Identiy various modications and changes along the way (and i there were none, was the initial concept really perect inall respects, or was the program unable to respond to changing circumstances?).

Identiy other descriptive inormation about the implementation process, including phasing o various activities orvariations across sites or types o beneciaries.

Who did the program serve?

Disaggregate the beneciaries characteristics to the extent possible, and certainly along characteristics identied duringdevelopment o the theory o change and the perormance story.

There are oten some dierences to track (e.g., gender, age, dierent tribal/cultural or socio-economic groups) that maybe evident, but oten others that may be very relevant to a particular situation (e.g., children with or without living parents,living with relatives or on their own).

What else was going on that could afect the programs impact, positively or otherwise, e.g.:

How has the program worked in conjunction with other agencies or programs, including identied partners?

Have there been other actors including interventions o other actors, government policies, private sector initiatives(such as the creation o employment opportunities, or perhaps the closure o the major employer in an area), natural andman-made disasters, etc. that have aected the ability o the program to progress positively or negatively, or perhapsmay have required a somewhat dierent approach than initially envisioned?

To what extent have the above actors helped or hindered the program?

What outcomes or impacts can be documented, in the short-, medium- and long term?

These should relate to the theory o change, taking into account what types o outcomes were expected at given points in time.

Are these as intended, or not? Any data, or even subjective assessments, as to why this might be?

Are impacts likely to be sustainable? What is the evidence or this?

How strong is the data? What else is needed to make a more convincing case o impact ollowing in some way rom theprogram intervention?

What else happened (unintended/unexpected efects)?

To what extent were these positive or negative?

To what extent were these potentially under the control o the program?

Should the program design and the theory o change be revised to take these into account or the uture?

How appropriate and relevant was the program as designed and implemented?

E.g., to what extent did the program address beneciaries requirements, community or national needs, or otheridentied local/global priorities?

Have needs and contextual considerations changed that may suggest that the program should be modied in some way?

What else is needed?

Are there other needs that the program itsel should be addressing?

Are there other needs that it might be more appropriate or others to address (e.g., implications or advocacy)?

Other considerations

as identied initially in the theory o change.

that might arise later, such as in response to preliminary ndings, through refection meetings, etc.

that address concerns or perspectives o other stakeholders, including critics o the program.

* Further reading: RealWorld Evaluation techniques such as discussed in Bamberger et al. 2012.


15/22


Suggestions in this table may well appear as a

ormidable list o what needs to be monitored.

But not every program needs to collect every orm

o data. The questions need not be asked all at

once, or can be staged throughout the liecycle o

the program.

Also, it is important to prioritize inormation

requirements, and to ocus on need to know

inormation. While developing a monitoring plan,

it should be clear in advance how every piece o

inormation can be used. Trying to obtain too

much inormation can create unintended conse-

quences. For example, it may overwhelm the sta

who need to collect the data, who are then orced

to take shortcuts that can aect the quality and

usability o any inormation provided.

Realist (or realistic) evaluation. When identiy-

ing data necessary to support meaningul impact

evaluation, it is helpul to bear in mind the tenet

orealist evaluation,8

which starts rom the prem-

ise that saying a program works or not is an

oversimplication, and that a statistically signi-

cant dierence between two groups9

may mask

considerable variation. Invariably, some people

gain and others do not, and almost all programsmay work or some people in some situations,

but not or others. Realist evaluation states that

outcomes arise through various combinations o

context and mechanisms. For example, gunpow-

der will only re i it is lit in a certain way. Thus,

a major implication is that impact evaluation

should identiy what works or whom, under what

conditions and through what mechanisms.

8 Further reading: Pawson and Tilley, Realistic Evaluation (1997).

9 Which, statistically, is based on comparing the average scores oeach group.

There are many illustrations o approaches(e.g., community mobilization) that need tobe adapted based on the livelihoods o thetarget group, or example pastoralists vs. sed-entary populations (such as armers). Similar-ly, an approach to improve youth reproductive

health may work with some cultural groupsbut not with others, or be dependent uponother actors.

Realist evaluation principles have major implica-

tions or the design o M&E systems, particularly

or types o data that need to be collected and

kept disaggregated. For example, some pro-

gram approaches that are eective with boys in

certain situations may be detrimental to girls.

Some programs that work in urban areas maynot work in rural areas. The relevance o some

dimensions (or example, gender, age or degree

o poverty) may be evident (but still not always

tracked), while others may be less obvious. For

example, an educational approach that may assist

some children may not do so or others, such as

children who may have been traumatized in some

way. In addition to characteristics o communi-

ties and beneciaries, it is also important to keep

track o context, or in this case what services wereprovided and in what circumstances.

NGO sta, given their close connection to the

eld and to beneciaries, are oten very well

placed to be able to observe, or even suspect,

which actors can infuence approaches that work

or some but not or others. Documenting this

type o inormation through ongoing M&E can be

an important contribution to impact evaluation. It

is essential, not only to understanding what has

actually taken place with a given program, but

also to learn rom this experience and to be able

to apply this inormation in dierent situations.


16/22


2.4. Start with what you have

Beore undertaking an expensive and complex

impact evaluation, start with the inormation

already available, rom M&E and other sources.

It may be that there is already sucient evidence

and an impact evaluation would not be worth the

investment. At a minimum, this can help ocus animpact evaluation on those questions and issues

that cannot be addressed in any other way. It may

suggest other orms o evaluation that might be

more appropriate to try rst. Here are a couple o

suggested systematic approaches.

Contribution analysis involves the ollowing six

steps, explained in more detail in Appendix 1:

1. Develop the theory o change.2. Assess the existing evidence on results.

3. Assess alternative explanations.

4. Assemble the perormance story.

5. Seek additional evidence.

6. Revise and strengthen the perormance story.

In essence, this involves using evidence rom

existing M&E to see what it can reveal about the

outcomes or even impact o an intervention, while

also considering what else besides the interven-tion could have created the identied outcomes.

From this, a provisional perormance story can be

developed about the extent to which it is reason-

able to assume that the programs actions could

have contributed to the observed outcomes, and

to identiy weaknesses and where addition data

would be useul. This is a powerul way o using ex-

isting data to determine what is known and where

data is needed rom additional orms o M&E, or i

necessary rom an impact evaluation, to provide a

more convincing picture.

Reective discussions. A contribution analy-

sis approach need not be a major undertaking,

particularly or a small and contained project.

Systematic refective discussions (or review or

monitoring meetings) are one good example o

this approach. These could orm part o regular

sta meetings, or perhaps be the ocus o special

meetings set up periodically. They could involve an

inormal discussion o what seems to have beenworking or not; the basis or drawing these conclu-

sions, taking into account competing explanations

or any observed or demonstrated outcomes (in

other words, a simple version o the perormance

story); and then identiying what other orms o

evidence will be needed in the uture to provide a

more convincing case o the programs impact. In

some cases, such as when sta and stakeholders

rom dierent areas have been brought together,

it might be appropriate to use a more ormalizedapproach and an external acilitator. This kind o

qualitative monitoring can be invaluable in under-

standing what may have led to outcomes or im-

pacts (and whether these same results would have

occurred anyway in the absence o the project),

which is essential inormation in order to apply the

ndings o impact evaluations.

AWARD, a ellowship program or womenscientists in agricultural research and devel-opment in Arica, has been using periodicrefection sessions to give the ellows andtheir mentors in dierent countries an op-portunity to consider progress, achievementsto date, and what has and has not been goingwell. These add to the management teamsmonitoring data, as well as their own ongoingrefections on perormance. This has providedopportunities to test the programs theory ochange, and has also helped the managementteam to work in a continuous improvement

mode, identiying the need or any remedialsteps or changes in direction.


17/22


Eliminate rival plausible explanations. Step 3 o the

contribution analysis approach involves assessing

alternative possible explanations. This is a simple

but powerul concept, well anchored in the evalua-

tion literature. It was rst articulated by Donald T.

Campbell,10

who urther said that one should use

the simplest and least costly approaches that canprovide necessary condence in the ndings.

Some approaches to ruling out rival plausi-ble explanations

Anticipate potential questions o theaudience(s) or the evaluation.

Address threats to internal and externalvalidity.

Use a theory-based approach. Identiy what types o evidence would be

convincing. Provide multiple lines o evidence Ensure ace validity.*

*This reers to ndings that are presented in such a way that

they appear to make sense at ace value to non-researchers

First identiy other plausible explanations or

how impact may have come about other than the

program or project intervention, and then produce

the necessary evidence to rule out the likelihood o

these alternative explanations. It only needs to ad-dress other plausible explanations. One can saely

assume that changes in youth employment pat-

terns in a district were not caused by the ghost o

Elvis! But the opening or closing o a new business

operation, or changes in the policies o an existing

operation, might be relevant in some situations. It

is important to realize, however, that stakeholders

can dier on what they view as plausible or not,

0 For example: Campbell, D. T. and J. C. Stanley. 1963. Experi-mental and Quasi-Experimental Designs for Research. Boston:Houghton-Mifin. Campbell is perhaps best known as the athero modern program evaluation and a proponent o experimentaland quasi-experimental methods. He was the rst to identiy thelatter concept, as well as with the concepts o internal and externalvalidity.

and that this is something that should be taken

into account when developing an M&E approach.

2.5. Design and implement the impact evaluation,

analyze and interpret the ndings

As previously indicated, M&E has an important

role to play in making sure that an impact evalua-tion asks the right questions and that the design is

realistic about what relevant M&E data might exist

or could be practically provided. Complementary

evaluation exercises involving both qualitative and

quantitative approaches may be appropriate, con-

tributing to a mixed methods approach (discussed

in Guidance Note 3) that could provide or triangu-

lation o impact evaluation ndings and later to as-

sist in interpretation. M&E specialists undoubtedly

would need to be involved, at least at some stagesduring the impact evaluation process, to acilitate

additional data collection. M&E specialists also

need to be ully aware o issues and tradeos

involved in establishing any orm o validity.11

A common assumption is that once an impactevaluation design is drawn up, everything mustremain static, oten or a period o years. Thereality, however, is that programs do evolveand change over time, and indeed must do so

in order to be responsive to changed circum-stances, eedback and new inormation. Impactevaluation designs need to be able to refect andadapt to changes in the nature o the interven-tion, otherwise the ndings may have limitedmeaning. As Guidance Note 3 indicates, someapproaches to impact evaluation can be morefexible and adaptable than others. Nevertheless,it is important to bear in mind that impact eval-uation requires cooperation rom the programin order to succeed, and to the extent possiblesignicant changes should be discussed withthe evaluation team.

A very useul very recent resource is: Huey T. Chen, StewartI. Donaldson, and Melvin M. Mark (eds.). Advancing Validity inOutcome Evaluation: Theory and Practice. New Directions for Evalu-ation. Jossey-Bass and American Evaluation Association. 2011.


18/22


M&E specialists (and through them, other orga-

nization sta and other stakeholders) should play

a signicant role in the interpretation o impact

evaluation ndings. It is useul to ask i the nd-

ings make sense and are consistent with other

inormation. I not, it would be appropriate to ask

what this might mean, and what could explain anyapparent contradictions.

As Section 1 has emphasized, M&E is needed to

provide inormation about relevant contextual

actors and the nature o the intervention, which

will be essential or drawing any causal links. In

particular, M&E is needed to provide explanations

or ndings, to indicate the whys and hows,

essential inormation to act upon impact evalua-

tion ndings. Are assumptions made about theintervention still valid? Counteractual designs are

oten particularly unable to explain the dierences

between experimental and control groups. Even i

there is no dierence between the mean (average)

scores o the experimental and control groups,

it is likely that some people nevertheless have

done better, and it is still possible that a majority

o people could do better. Complementary M&E

inormation can help get inside the black box

in order to be able to understand and interpretthe dierences and what they mean or uture

directions.12

Such considerations might suggest

additional subgroup analyses that may still be pos-

sible, or at a minimum aid in appropriate interpre-

tation o the ndings.

2.6. Use the ndings

Use o impact evaluation is the ocus o Guidance

Note 4 and is largely beyond the scope o this

For example, impact evaluations are oten undertaken with theobjective o indicating i a given approach should be continuedand/or replicated in a dierent setting (generalizability or externalvalidity). But without knowing how and why impact occurred (ornot), it is impossible to make inormed decisions about suchchoices.

note. Nevertheless, it is worth noting that M&E

can have a signicant role to play in this process.

Other M&E inormation may be able to help iden-

tiy the extent to which an impact evaluations nd-

ings can be generalized or may be applicable to

other settings or populations, or even at the same

place, given the inevitable changes in context, thenature o the intervention, and the target group

over the liespan o a multiyear impact evaluation.

M&E specialists can help animate sessions involv-

ing sta and management to consider implica-

tions o impact evaluation ndings in light o other

inormation.

2.7. Review, reect, and update

No matter how much eort one has put into the

initial articulation o the theory o change, develop-ing an evaluation plan and design(s), and putting

data collection mechanisms in place, this does not

mean that one can or should then orget about

it. It is useul to periodically consider whether the

assumptions underlying the impact evaluation

approach are still valid, or i any changes may

be needed. Responsive programs rarely remain

static they adapt their approaches to changes

in context, to address emerging needs and op-

portunities, and in response to eedback rombeneciaries, other stakeholders and M&E. I

identied early enough, it may still be possible or

an impact evaluation to take these into consider-

ation. Appropriate impact evaluation methodolo-

gies should be chosen that are fexible enough to

accommodate changes in program direction that,

in many cases, are almost inevitable.

It is appropriate to conduct interim reviews (which

can be as simple as a ocused sta meeting) to

consider any relevant changes to the program,

and to revisit the theory o change to check i the

original assumptions are still valid or need updat-

ing. As suggested above, this review can be an


19/22


opportunity to develop or to update the peror-

mance story, and to identiy gaps in inormation

that should be addressed in some way. Interim

reviews, in addition to providing an opportunity to

refect on how the program seems to be progress-

ing, can also serve as an opportunity to review, and

i necessary update, the M&E plan itsel.

This note has emphasized the role that M&E can

play to support meaningul impact evaluation.

But the reverse is also true. Impact evaluation

can invariably suggest types o data that should

be monitored in the uture, as well as raise ques-

tions that can be addressed by other orms o

evaluation. It is appropriate, at the conclusion o

an impact evaluation, to consider implications or

uture M&E priorities.

3. Engaging all parts o the organization

M&E and impact evaluation clearly require an

active role or M&E specialists to put in place ap-

propriate evaluation designs, and to ensure that

all aspects o the process are undertaken in ways

that maximize the validity and utility o the data.

But there are also important roles or management

and program sta that sometimes are not wellappreciated.

3.1. M&E: A core management unction requiring

senior management leadership and support

M&E, and thinking and acting in impact terms,

are much too important to be let just to M&E

specialists. First and oremost, M&E represents a

management tool that can help in setting direc-

tions; assessing progress; learning about the types

o approaches that appear to work or not in varying

circumstances; making decisions; and in many

other ways. This means that NGO senior manage-

ment must take ownership, provide leadership,

and encourage everyone within the organization

to think in terms o outcomes or results by asking

how they can know their work is making a dier-

ence and what can be done to improve. Learning

and change imply an openness to acknowledging

and accepting that, on occasion, some approaches

will have not worked well and require modication.

But this requires management to encourage andsupport sta to innovate and not to punish sta

i not everything turns out as expected provided

that there is openness to learn rom experience,

and rom evaluation. Such support is essential i

impact evaluation, along with M&E, will have any

real impact on how programs are implemented.

Managers at all levels in an agency will need to

provide the necessary support, resources and lead-

ership to enable meaningul M&E and impactevaluation to take place. Senior managers in par-

ticular have a key role to ensure that this is not just

a paper exercise that can breed cynicism and pas-

sive compliance, but to demonstrate their commit-

ment to doing and to using all orms o results

inormation. Carrying out impact evaluation, even

i there is a dedicated budget or this, will also have

workload and possible resource implications or

program sta and M&E specialists. Management

should be aware o these implications when agree-ing to undertake impact evaluation. To a large

extent, the relevance and success o impact evalu-

ation, and the ability o routine M&E to contribute

to this success, is dependent upon the attitude

and support o senior management.

3.2. An active role or program staf is required

Meaningul impact evaluation, together with

routine M&E, requires the active involvement o

program sta. In most cases, it is program sta

who will be responsible or the actual collection,

recording, and reporting o much o the data. It is

important to acknowledge this key role and what it

implies. Imposing requirements on busy program


20/22


managers and sta is in practice oten looked

on as yet another administrative task o dubious

value and meets with resistance which may

result in issues with respect to data quality and

completeness.

Making M&E and impact evaluation helpul to

program staf rather than viewed as a chore

Involve them in the process.

Address questions that they eel are rel-evant and can help them.

Provide opportunity to participate in theinterpretation o M&E/impact evaluationdata.

Provide eedback when data is submitted(even an acknowledgement or a thankyou).

Provide recognition or sta who assistwith impact evaluation.

Share inormation about how the datahas been used.

As suggested above, appropriate leadership rom

an organizations senior management is needed

to set the stage or meaningul data collection. To

acilitate the needed cooperation o program sta

in data collection, they should be given incen-tives and the resources needed (time, sta and

unding) to ensure the collection and recording o

quality data, along with appropriate quality control

mechanisms. To make the data collection relevant

and meaningul, program sta should be engaged

in the process o identiying questions and issues

that they eel are important and relevant to them

and to the communities in which they are work-

ing. The accompanying box lists some ideas or

making M&E, and impact evaluation, meaningul.

For example, sta in an education-or-all project

in Uganda began to appreciate the importance o

the monitoring data they were asked to provide,

and to improve its quality, once some o the initial

evaluation reports were shared with them.

To the extent possible, considerations regarding

the role and support o program sta should be

built into the M&E/impact evaluation approach.

Program sta who will be required to collect

the data are oten best placed to identiy whichinormation is easy to obtain and which may be

more problematic, and what requests or data are

ambiguous and can be interpreted in diering

ways, or may even be so unreliable as to be use-

less. Through involvement in the process, M&E is

more likely to be viewed as a support rather than

a burdensome chore with no value, or something

that is done to them with no discernable value.

Summary

Impact evaluation is sometimes seen by some

researchers as well as by many program manag-

ers and sta as a very technical exercise that can

only be carried out by external experts working at

some distance rom the program, its sta and its

constituency. And there are some situations where

this is how it is approached.

But as this note has indicated, impact evaluationrepresents just one orm o evaluation. Only in

rare circumstances can meaningul impact evalu-

ation be carried out without drawing on data rom

other ongoing M&E activities. But the value and

use o M&E to impact evaluation is not automatic.

Advance planning is needed to identiy the types

o data that will be required in order to be able to

determine the impact o policies, programs and

projects and how these data can be obtained.

And as this note has emphasized, when everyone

within an agency thinks in impact or results terms,

this alone can help support the eectiveness o the

agencys operations and help improve the lives

o people around the world.


21/22


Reerences and Other Useul Resources

Bamberger, M., Rugh, J. and Mabry, L. (2012).

RealWorld Evaluation: Working Under Budget,

Time, Data, and Political Constraints (2nd ed.).

Sage Publications.

Chen, H., Donaldson, S., Mark, M. (eds.) (2011).

Advancing Validity in Outcome Evaluation: Theory

and Practice. New Directions for Evaluation. No.

130. Jossey-Bass and the American Evaluation

Association.

Funnell, S. and Rogers, P (2011). Purposeful

Program Theory: Effective Use of Theories

of Change and Logic Models. Jossey-Bass

Publications.

Jones, H. (2011). A guide to monitoring and

evaluating policy infuence. London: Overseas

Development Institute. http://www.odi.org.uk/

resources/docs/6453.pd

Mayne, J. (2001). Addressing attribution through

contribution analysis: Using perormance mea-

sures sensibly, Canadian Journal of Program

Evaluation, 16(1), 1-24.

Mayne, J. (2011). Contribution Analysis:

Advancing Cause and Eect. In Evaluating the

Complex: Attribution, Contribution, and Beyond,

edited by Robert Schwartz, Mita Mara, and Kim

Forss. London: Transaction Publishing.

OECD/DAC. (2011). Evaluating Impacts: An

Overview. http://www.oecd.org/document/17/0,37

46,en_21571361_34047972_46179985_1_1_1_1,00.

html

OECD/DAC. (2002). Glossary of Key Terms in

Evaluation and Results Based Management.

http://www.oecd.org/dataoecd/29/21/2754804.

pd

Malhotra, A. (2010). Connecting Routine M&E

with Impact Evaluation, International Center

or Research on Women. Presentation at

InterAction/3ie Impact Evaluation Design Clinic,

Washington, D.C., April 2010. http://www.

interaction.org/sites/deault/les/18/Anju_

Malhotra_-_Connecting_Routine_M&E_with_

Impact_Evaluation.pd

Paul, S. (2002). Holding the State to Account:Citizen Monitoring in Action. Bangalore: Public

Aairs Centre.

Pawson, R. and Tilley, N. (1997). Realistic

Evaluation. Sage Publications.

Scriven, M. (1991). Evaluation Thesaurus (4th ed).

Sage Publications.

Woolcock, M. (2011). Guest Post on TheImportance o Time and Trajectories in

Understanding Project Eectiveness.

Development Impact. Blogs.worldbank.org.

http://blogs.worldbank.org/impactevaluations/

guest-post-michael-woolcock-on-the-importance-

o-time-and-trajectories-in-understanding-proj-

ect-ee.
http://www.odi.org.uk/resources/docs/6453.pdfhttp://www.odi.org.uk/resources/docs/6453.pdfhttp://www.oecd.org/document/17/0,3746,en_21571361_34047972_46179985_1_1_1_1,00.htmlhttp://www.oecd.org/document/17/0,3746,en_21571361_34047972_46179985_1_1_1_1,00.htmlhttp://www.oecd.org/document/17/0,3746,en_21571361_34047972_46179985_1_1_1_1,00.htmlhttp://www.oecd.org/dataoecd/29/21/2754804.pdfhttp://www.oecd.org/dataoecd/29/21/2754804.pdfhttp://www.interaction.org/sites/default/files/18/Anju_Malhotra_-_Connecting_Routine_M&E_with_Impact_Evaluation.pdfhttp://www.interaction.org/sites/default/files/18/Anju_Malhotra_-_Connecting_Routine_M&E_with_Impact_Evaluation.pdfhttp://www.interaction.org/sites/default/files/18/Anju_Malhotra_-_Connecting_Routine_M&E_with_Impact_Evaluation.pdfhttp://www.interaction.org/sites/default/files/18/Anju_Malhotra_-_Connecting_Routine_M&E_with_Impact_Evaluation.pdfhttp://blogs.worldbank.org/impactevaluations/guest-post-michael-woolcock-on-the-importance-of-time-and-trajectories-in-understanding-project-effehttp://blogs.worldbank.org/impactevaluations/guest-post-michael-woolcock-on-the-importance-of-time-and-trajectories-in-understanding-project-effehttp://blogs.worldbank.org/impactevaluations/guest-post-michael-woolcock-on-the-importance-of-time-and-trajectories-in-understanding-project-effehttp://blogs.worldbank.org/impactevaluations/guest-post-michael-woolcock-on-the-importance-of-time-and-trajectories-in-understanding-project-effehttp://blogs.worldbank.org/impactevaluations/guest-post-michael-woolcock-on-the-importance-of-time-and-trajectories-in-understanding-project-effehttp://blogs.worldbank.org/impactevaluations/guest-post-michael-woolcock-on-the-importance-of-time-and-trajectories-in-understanding-project-effehttp://blogs.worldbank.org/impactevaluations/guest-post-michael-woolcock-on-the-importance-of-time-and-trajectories-in-understanding-project-effehttp://blogs.worldbank.org/impactevaluations/guest-post-michael-woolcock-on-the-importance-of-time-and-trajectories-in-understanding-project-effehttp://www.interaction.org/sites/default/files/18/Anju_Malhotra_-_Connecting_Routine_M&E_with_Impact_Evaluation.pdfhttp://www.interaction.org/sites/default/files/18/Anju_Malhotra_-_Connecting_Routine_M&E_with_Impact_Evaluation.pdfhttp://www.interaction.org/sites/default/files/18/Anju_Malhotra_-_Connecting_Routine_M&E_with_Impact_Evaluation.pdfhttp://www.interaction.org/sites/default/files/18/Anju_Malhotra_-_Connecting_Routine_M&E_with_Impact_Evaluation.pdfhttp://www.oecd.org/dataoecd/29/21/2754804.pdfhttp://www.oecd.org/dataoecd/29/21/2754804.pdfhttp://www.oecd.org/document/17/0,3746,en_21571361_34047972_46179985_1_1_1_1,00.htmlhttp://www.oecd.org/document/17/0,3746,en_21571361_34047972_46179985_1_1_1_1,00.htmlhttp://www.oecd.org/document/17/0,3746,en_21571361_34047972_46179985_1_1_1_1,00.htmlhttp://www.odi.org.uk/resources/docs/6453.pdfhttp://www.odi.org.uk/resources/docs/6453.pdf


22/22

Annex 1 Contribution analysis

John Mayne, then with Canadas Oce o the

Auditor General, has suggested using contribu-

tion analysis as one approach by which one can

make best use o existing data about outcomes

and impact. He has summarized this approach as

ollows. More detailed inormation about a contri-

bution analysis approach can be ound in Mayne

(2001), rom where this table has been copied

verbatim, or in Mayne (2011).

Step 1: Develop theresults chain

Develop the program theory model/program logic/results chain describing how the program issupposed to work. Identiy as well the main external actors at play that might account or theoutcomes observed. This program theory should lead to a plausible association between theactivities o the program and the outcomes sought. Some links in the results chain will be airlywell understood or accepted. Others will be less well understood or subject to explanationsother than that the program was the cause. In this way you acknowledge that attribution isindeed a problem.

Step 2: Assess the

existing evidence onresults

The results chain should provide a good idea o which intended results (outputs, intermediate

and end outcomes) could be measured. What evidence (inormation rom perormancemeasures and evaluations) is currently available about the occurrence o these various results?The links in the results chain also need to be assessed. Which are strong (good evidenceavailable, strong logic, or wide acceptance) and which are weak (little evidence available, weaklogic, or little agreement among stakeholders)?

Step 3: Assessthe alternativeexplanations

Outcomes by denition are infuenced not only by the action o the program but also byexternal actors other programs, as well as social and economic actors. In addition toassessing the existing evidence on results, there is a need to explicitly consider the extent oinfuence these external actors might have. Evidence or logical argument might suggest thatsome have only a small infuence and that others may have a more signicant infuence on theintended results.

Step 4: Assemble

the perormancestory

With this inormation, you will be able to set out your perormance story o why it is reasonable

to assume that the actions o the program have contributed (in some ashion, which youmay want to try and characterize) to the observed outcomes. How credible is the story? Doreasonable people agree with the story? Does the pattern o results observed validate theresults chain? Where are the main weaknesses in the story? There always will be weaknesses.These point to where additional data or inormation would be useul.

I getting additional evidence is not possible (at least or now), then this is the most you cansay about the extent to which the program has made a dierence.

Step 5: Seek outadditional evidence

To improve your perormance story you will need additional evidence. This could involveinormation on both the extent o occurrence o specic results in the results chain and thestrength o certain links in the chain. A number o strengthening techniques that you might beable to adopt are outlined in this work.

Step 6: Revise andstrengthen theperormance story

With the new evidence, you should be able to build a more credible story, one that a reasonableperson will be more likely to agree with. It will probably not be oolproo, but will be strongerand more credible.

Documents

Monitoring and Evaluation Impact