Pme Concepts Terms 00

8/11/2019 Pme Concepts Terms 00

1/36

Planning, Monitoring and Evaluating

Programmes and Projects

Introduction to Key Concepts, Approaches andTerms

Working DraftVersion 1 March 2000

Global Monitoring and Evaluation Initiative

Jim WoodhillIUCN M&E Facilitator forEast and Southern Africa


2/36

Introduction to PM&E Concepts, Approaches and Terms

Working Draft Version 1 March 2000 IUCN Global M&E Initiative

IUCN The World Conservation Union

Founded in 1948, The World Conservation Union brings togetherStates, government agencies and a diverse range of non-governmental organizations in a unique world partnership: over900 members in all, spread across some 138 countries.

As a Union, IUCN seeks to influence, encourage and assistsocieties throughout the world to conserve the integrity anddiversity of nature and to ensure that any use of natural resourcesis equitable and ecologically sustainable.

The World Conservation Union builds on the strengths of itsmembers, networks and partners to enhance their capacity and tosupport global alliances to safeguard natural resources at local,regional and global levels.

The IUCN Monitoring and Evaluation (M&E) Initiative

Through an approach which fosters questioning and reflectionand engages stakeholders at the regional and global levels, theIUCN M&E Initiative aims to:

develop a common understanding of M&E within IUCN

develop a reflective culture within IUCN

improve project/programme design and implementationthrough the use of methods and tools in project, systemsand institutional assessments

assess the relevance of the Unions work against the broader picture of ecosystem and human wellbeing

improve learning processes and reporting of lessonslearned

put an overall M&E System in place for the Union.

Publications from the M&E Initiative are available on-line on theIUCN website http://iucn.org/themes.html

AcknowledgementsWritten by Jim Woodhill IUCN Monitoring and EvaluationFacilitator for East and Southern Africa.

Email [email protected]

: (2000) IUCN International Union for Conservation of Nature and Natural Resources.

Reproduction of this publication for educational and non-commercial purposes is authorised without prior permission fromthe copyright holder provided the source is fully acknowledged.Reproduction for resale or other commercial purposes is

prohibited without the prior written permission of the copyrightholder.


3/36

Introduction to PM&E Concepts, Approaches and Terms


Contents

1 INTRODUCTION ............................................................................................. .............................................. 1

2 APPROACHES TO PLANNING MONITORING AND EVALUATION ............................................... 2

3 KEY PLANNING, MONITORING AND EVALUATION CONCEPTS................................................. 3

3.1 A PARTICIPATORY LEARNING APPROACH ........................................................................................ ............. 33.2 T HE MANAGEMENT CONTEXT .......................................................................................................... ............. 43.3 P ROGRAMME AND PROJECT CYCLE ............................................................................................. .................. 63.4 P ROGRAMME AND PROJECT LOGIC ................................................................................................... ............. 63.5 O BJECTIVE HIERARCHIES AND ASSUMPTIONS .............................................................................................. . 83.6 A CCOUNTABILITY AND CONTROL ................................................................................................ ................ 153.7 K EY ASPECTS OF EVALUATION .................................................................................... ................................ 163.8 D EVELOPING AN OVERALL M&E STRATEGY /PLAN ................................................................................ .... 173.9 D EVELOPING AND MONITORING EVALUATION QUESTIONS AND I NDICATORS ........................................... 193.10 O PEN E NDED EVALUATION ACTIVITIES .................................................................................................. 21

4 LOGICAL FRAMEWORK APPROACH (LFA) AND ZOPP................................................................ 23

5 RESULTS BASED MANAGEMENT ......................................................................................................... 27

5.1 CIDA RBM TERMINOLOGY ......................................................................................... ................................ 285.2 USAID ROA TERMINOLOGY ....................................................................................... ................................ 32


4/36


5/36

Introduction to PM&E Concepts, Approach and Terms 1


1 Introduction

This document has been developed to assist IUCN staff and partnersnavigate their way through the terminology that surrounds the practice of

planning, monitoring and evaluation (PM&E) in environment and

development programmes and projects. It should be considered a workingdraft and will be revised and improved on the basis of feedback from thosewho use it. A separate glossary of terms accompanies this document.

There is no question that planning, monitoring and evaluation are fieldslittered with terminology that is often unclear and which is used withdifferent meanings by different groups. This problem cant be solved by thisdocument, however, what it can do is to help explain the differentapproaches to PM&E and how terms are used by different organisations.Unfortunately confusion around terminology often makes PM&E seemmuch more complex and difficult than is actually the case. In fact there are

really only a handful of key concepts that need to be understood aboutPM&E. If these concepts are understood then it is easy to make sense of thedifferent terminology and to translate terms between different approaches toPM&E.

Monitoring and Evaluation (M&E) is often considered as a separate functionand responsibility from planning. However, if a project is poorly planned itis very difficult to monitor and evaluate. Very often M&E staff andspecialists find themselves having to go back to basic planning principles

before they can assist with M&E. Hence the attention in this document to planning and M&E.

Within IUCN it would simplify the difficulties of confusing terminology if astandard set terminology could be used. The Global M&E Initiatives isworking towards such a set of concepts and terms. However, the reality isthat IUCN will always be working with different donors who demand theuse of their particular approach and terminology. This means that it willalways be necessary for staff to understand the underlying concepts ofPM&E and be able to translate between the approaches and terminology ofdifferent donors and partner organisations.

The confusion about PM&E can also be reduced by understanding clearlythe way concepts and terms are used by different approaches and donors.

Consequently the later part of this document explores a number ofapproaches in some detail.


6/36



2 Approaches to Planning Monitoring and Evaluation

Broadly there are three main approaches to PM&E in use by the major donoragencies:

1. The logical framework approach (LFA) which is the most commonand widely used.

2. The German ZOPP, a close derivative of LFA. The acronym standsfor the German equivalent of objective oriented project planning.

3. Results Based Management (RBM) or managing for results, whichhas become the favoured model of the Canadians and Americans inrecent years.

However, even within each approach there are often differences in the use ofterminology and many adaptations have been made as different groups put

the approaches into practice. Further, those within agencies who shouldunderstand the approach being used are often not as clear in theirunderstanding as would be ideal. This difficulty is compounded whenagencies are in a transition from one approach to another.

However, while there are certainly differences between the approaches, theunderlying principles of PM&E that they are each trying to promote areremarkably similar. In essence, they are:

1. To develop programmes and projects based on a thoroughunderstanding of the situation in which an intervention is planned.

2. To involve stakeholders in a participatory process of programme or project design and evaluation.

3. To develop a set of clear logical objectives that can realistically beachieved within a particular timeframe and within an allocated

budget and which will make a significant and sustained contributionto a higher level development objective.

4. To make explicit the cause and effect (means ends) relationships andexternal factors that underpin the programme or project and whichmust hold true if planned activities are going to lead to desiredresults and impacts.

5. To establish a monitoring and evaluation system, includingindicators, which will show if the objectives have been achieved and

provide information to support effective management and learning.


7/36



3 Key Planning, Monitoring and Evaluation Concepts

3.1 A Participatory Learning Approach

The approach to PM&E adopted by IUCN is one that emphasises the

participation of stakeholders in continually learning how to improve performance. Monitoring and evaluation is seen very much as a learning process and not as an external top down policing function. It is recognisedthat given the complexity of conservation and natural resource managementand an extremely rapidly changing wider environment an adaptive approachto programme and project management is essential. PM&E should be seenas a process of helping people to learn how to do things better.Consequently the theory and practice of adult learning is very important tothe monitoring and evaluation approach being developed within IUCN.Given the breadth of this field in itself, the glossary will not attempt to coverit in any detail. However, when using the glossary it will help to keep in

mind the idea of a participatory learning approach to monitoring andevaluation.

A participatory learning approach also means that there is much more toM&E than just identifying and monitoring quantitative indicators. Learningimplies understanding, analysis, questioning, being critical and trying toexplain why things have worked or failed. Certainly quantitative indicatorsare important and can be helpful but very often they provide only a small

part of the information needed for learning. Also for higher level objectivesor goals, such as improving protected area management or community well-

being it is just simply not possible to develop simple quantitative indicators

that have any real meaning. Unfortunately there is a widespread view thatdeveloping an M&E plan for a programme or project is primarily aboutdeveloping a set of such quantitative indicators. This document explicitlychallenges this perception. For example, well-facilitated review meetingswith staff, or the use of qualitative inquiry methods with beneficiaries, willoften provide much more valuable information for learning.


8/36



3.2 The Management Context

PM&E are essential functions of good management and should enable a programme or project to achieve a high level of performance as illustrated inthe box below.

It is important to see monitoring and evaluation as tools to be integrated intoall aspects of programme and project management, as illustrated in the next

box. The starting point is to ask; what information is required for effective

management and what sort of M&E system is required to provide it?

Management Functions and M&E

FunctionsMonitoring

&Evaluation

StaffingStaffing

OrganisingOrganisingControllingControlling

LeadingLeading

PlanningPlanning

PM&E(learning)

Performance

PM&E - Critical Tools for Management

Management

should support

resulting in


9/36



Unfortunately M&E is often erroneously viewed as an annoying task ofsimply providing donors with the information they require. Certainlyaccountability to funding bodies is one function of an effective M&E system

but it is certainly not the only or the most important function. A list of purposes for M&E is given below.

Purposes of Monitoring and Evaluation

Ensuring planned results are achieved

Improving and support management

Generating shared understanding

Generating new knowledge and support learning

Building the capacity of those involved

Motivating stakeholders

Ensuring accountability

Fostering public and political support


10/36



3.3 Programme and Project Cycle

The diagram below illustrates a generic programme or project cycle. Itemphasises the importance of starting with detailed scoping, situationanalysis and design stages. It also illustrates the importance of consideringM&E at all stages in the cycle. Importantly it also illustrates the need forconstant cycles of planning, acting, monitoring and evaluation (in otherwords learning) during implementation.

3.4 Programme and Project Logic

There is an overall logic to any programme or project as illustrated below.This logic can be described as follows. First, there is a situation that a groupof stakeholders wish to improve the reasons for a programme or project.This situation is defined, to a significant extent, by the problems and visionsof the stakeholder groups. There may or may not be common perceptions ofwhat the problems are or what would actually constitute an improvement,which is why participatory approaches to planning are so important.Developing a detailed and holistic situation analysis is a critical aspect of

programme or project planning.

The understanding of the situation will lead to a programme or project plan.This will usually include:

The goal a summary of what in the long term the programme or project is contributing towards related to impact.

The Programme/Project Cycle

OrganisationMission

Financing andcontracting

Final Evaluation

Gov/Donor Goals/Policies

BeneficiaryNeeds

Plan

Act

Monitor

Evaluate

Implementation

Monitoring andEvaluation

Scoping

Formulation /Design

M&EStrategy

Mobilisation andImplementation Planning

M&EPlan


11/36



The Purpose a summary statement of overall what the programmeor project should achieve given its timeframe and resources theoverall outcome.

A set of Results the main things that must be achieved for the programme or project to realise its purpose, there may be several

levels of results. Results are also referred to as outputs and outcomes.

A set of Activities what must actually be done for the results to berealised.

A set of Inputs the resources required for the activities to beundertaken.

The programme or project is then implemented according to this planinvolving a process of inputs being used to undertake activities that lead toactual results. Of course few projects go exactly as planned and there willneed to be constant cycles of planning, acting, monitoring, evaluating, re-

planning and so on. In some cases it may be necessary completely revise theoriginal plan.

The actual results should lead to a set of impacts that will improve theoriginal situation. In most projects there will also be unanticipated impactsthat may be positive or negative, which are also important to track.

Understanding this basic logic of a programme or project is the starting pointfor understanding PM&E.

The Situation toImprove

Problems and Visions

Plan(Goal, Purpose,

Resultsand Activities)

Inputs Activities Actual Results

Reasonsfor Programme

Project

ProgrammeProjectand

ImplementatioProcess

IMPACT

Project/Programme Make Up and Logic


12/36



3.5 Objective Hierarchies and Assumptions

Any programme or project has an objective hierarchy, as shown below. Thismaps out the way low level tasks or activities contribute to higher levelobjectives and how meeting these objectives leads to the achievement of theoverall purpose or goal of a project or programme. Such a hierarchy of

objectives is also referred to as the intervention logic or the narrativesummary of a programme or project. It shows the cause and effect, or means

ends relationships of an intervention. Developing a clear logical objectivehierarchy is fundamental to good programme and project design andessential for M&E. It is called an objective hierarchy because any levelwithin it can be seen as being an objective. Higher level objectives are, orshould be, a consequence of achieving lower level objectives. Alternativelyit could be considered a results hierarchy. Although as will be shown later adistinction is sometime made between an objectives hierarchy being what is

planned to be achieved and a results hierarchy being what is actuallyachieved.

An objective hierarchy can be thought of as like the structure of a tree, withthe leaves or twigs being the detailed activities or tasks (low levelobjectives) and the trunk being the overall goal (highest level objective) and

the branches being different intermediate level objectives. In theory therecan be many levels to an objective hierarchy, larger and more complex

programmes or projects require more levels than do small simple projects.In practice, most planning approaches find four or five levels adequate. For a

programme one might imagine the trunk and main branches as being the programme objectives and the sub-branches twigs and leaves as being the projects that contribute to the programme objectives. In a very large project,it may be helpful to think of having sub-projects. For a complex programme

Objective Hierarchy

Purpose

Goal

Key

Result

Key

Result

SubResult

SubResult

SubResult

A C T I V

I T Y

A C T I V

I T Y

A C T I V

I T Y

A C T I V

I T Y

A C T

I V I T Y

A C T

I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

Key

Result

A C T I V

I T Y

A C T I V I T

Y

A C T I V

I T Y

A C T I V I T

Y

SubResult

A C T I V I T Y

A C T I V I T Y

SubResult

SubResult

SubResult

SubResult

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

A C T I V I T Y

Assumptions(hypotheses)


13/36



with large projects, it is conceivable that it may be necessary to have a programme, project and sub-project level of planning and each of theselevels having six levels of an objective hierarchy. In the end, planning needsto be taken down to sufficient detail to enable day to day workplans foractivities and tasks. A well-developed objective hierarchy makes it clearwhat must be done to achieve results and in reverse, along with indicators,what results are achieved from completing activities.

An objective hierarchy also has a sideways logic. The outputs, products orresults from one strand of the hierarchy will often be critical inputs intoanother strand. The conventional representation of an objective hierarchy(as shown above) does not illustrate this sideways flow leading to thecriticism that such an approach to design is too linear and restrictive.However it is equally possible to map out a project from a systems

perspective showing a series of interconnecting systems that have inputs andoutputs. It is important that the linear criticism is not used as an excuse for

poorly developed intervention logic.

Certainly, for complex programmes it will not always be possible to arrive ata simple hierarchical logic, such as the one illustrate above, that adequatelyexpresses all the dimensions that need to be communicated. There may needto be a number of parallel logics or a matrix structure to the programmeframework. A detailed discussion of this is beyond the scope of thisdocument.

To assist in the planning process different levels within an objectivehierarchy are given different terms (goal, purpose, outcomes, outputs,results, specific objectives, activities, etc). It is the lack of consistency in the

way terms are used for the different levels in an objective hierarchy thatcreates much of the PM&E confusion. However, as long as the concept ofdifferent levels in a hierarchy is understood and the meaning of differentterms is understood by the stakeholders within a particular context, it doesntreally matter what they are called. Some examples of different objectivehierarchies that have a different number of levels and which use differentterminology are given on the next page.


14/36



Assumptions

An objective hierarchy should reflect cause and effect relationships betweenlower and higher level objectives. The lower level activities/objectives arethe means for achieving the ends of the higher level objectives. Therewill always be assumptions about the cause - effect (means ends)relationships in an objective hierarchy and these assumptions should bemade explicit in the design of a programme or project.

There are two types of assumptions, which are often confused in the logicalframework approach to planning. The first type of assumption is that relatedto the internal logic of the programme or project. For example a projectdesign might be based on an assumption that by reducing poaching in anarea an endangered species will be preserved. This may be a correctassumption or it may be incorrect because the main threat to the speciesmight in fact be reduced habitat or disease.

The second type of assumption relates to the external factors or externalenvironment that must exist for a project to succeed. For example, for a tree

planting project it may be assumed that rainfall will not be significantly

below average. For any project it will be assumed that there will besufficient political stability for the project to operate effectively.

For either type of assumption it is often helpful to identify, what are referredto as, killer assumptions. These are the assumptions that if they are wrongthe project will fail completely.

Examples of Different Objective Hierarchies

Goal

Activities

Outputs

Purpose

Vision

Activities

Objectives

Goal(s)

Goal

Outputs

Outcomes

Purpose

Activities

Activities

Vision

StrategicObjectives

Mission

Goals

KeyResultAreas

Results

Goal

KeyResults

Purpose

Activities

SubResults


15/36



Good project design looks very carefully at assumptions and makes themexplicit. Likewise an important part of M&E is checking on the validity oforiginal assumptions.

Sorting Out Objective Hierarchy Terminology

Objective hierarchy terminology, as already mentioned, is the source of mostof the confusion around PM&E. To help sort this out it will be helpful todiscuss some basic ideas about different levels in an objective hierarchy

before getting too caught up in the terminology.

At the highest level of an objective hierarchy it is helpful to place the programme or project within the context of some larger human endeavour.For example an integrated conservation and development project may becontributing towards the protection of the ecological, economic and culturalvalues of a particular forest. However the project itself will not be able tofully realise this highest level objective or goal, it can only make a

contribution. This level provides clarity about why a programme or projectis being undertaken. It helps to provide a sense of vision about the future forthose engaged with a programme or project. This level is commonlyunderstood as the goal for a programme or project. Some planningframeworks call this level the vision and reserve the goal for the next leveldown.

Impact is generally used to refer to the extent to which a programme or project in fact does make a contribution towards the goal. However this maynot occur during the life of a programme or project and where others are alsocontributing it may be difficult to desegregate the contributions made by oneinitiative from those of another. This makes impact evaluation particularlydifficult, but nevertheless important.

Below this top level of an objective hierarchy is what can be considered the purpose of a programme or project. This is overall what a programme or project should achieve if it is successful. It is generally considered helpful totry and summarise the purpose as a single statement to ensure focus andclarity. For example to develop the institutional frameworks and humanand organisational capacity for sustainable forest management. This iswhat a programme or project should be able to achieve within its availableresources and implementation period. If planning for an organisation thislevel will usually be called the mission of the organisation.

Below this level are a series of major outcomes or results that need to beachieved for the purpose to be realised. These should be the actualobservable changes in for example behaviour, institutions, economiccircumstances or physical conditions. Again these should be achievablewithin the resources and timeframe of the programme or project. Forexample, staff within the forest department effectively carrying out theirresponsibilities. In the LFA this level is subsumed into the purpose level


16/36



and it is argued that there are outcomes at the purpose level. Other planningframeworks refer to this level as the main objectives.

Each of these major outcomes or results arise from the logical consequenceof the programme or project delivering a set of products or services outputs

which arise directly from programme or project activities. For example

forestry department staff trained in the skills required to carry out theirresponsibilities.

Not all planning frameworks make this distinction between outcomes andoutputs. It is this middle level of an objective hierarchy where terms andconcepts are most problematic. While outcomes generally refer to the higherorder changes or effects that arise from delivering outputs, which aregenerally considered as tangible or concrete products and services, thedistinction is not always so clear cut. There is not a neat dividing line

between what can be considered outputs and are outcomes. In fact ratherthan there being just two cause and effect steps there may be many, ie there

are also higher and lower level outputs and outcomes. While a slightlyartificial, the output and outcome distinction is still helpful in summarisingwhat can be expected to result at different levels within an objectivehierarchy. However, it needs to be remembered that it is a shorthandsummary and like all categorisations an approximation of reality.

This ambiguity around outputs and outcomes is why IUCN finds it lessconfusing to talk of key-results and sub-results within the objectivehierarchy. The relationship between these terms is illustrated below.

Objective and Results HierarchiesObjective Hierarchy - what isplanned to be achieved

Results Hierarchy - what isactually achieved(also called impact or hierarchy and results

Actual Overall Result Outcomes

Actual Key ResultsOutcomes

Actual Sub ResultsOutputs

Goal

Purpose

(Planned)Key Results

(Planned)Sub Results

Activities

Impact

Outputs

Outcomes

Outcomes

Outcomes

Outputs

This is whereM&E terminology

can becomes veryconfusing

Inputs


17/36



The lowest level of an objective hierarchy is the activity level. For detailedwork planning it will often be necessary to break activities down the sub-activities and/or tasks that will occur, for example over a particular year orquarter.

From an M&E perspective it is very helpful to think in terms of impacts,

outcomes and outputs. However it is not necessary to limit impact questionsto the goal level of the objective hierarchy or outcome questions to the key-result/outcome level of the hierarchy. This is easiest explained with anexample. Take the sub-result of a large integrated conservation anddevelopment project - sustainable livelihood activities adopted in targetvillages. An activity of this sub result may the promotion of bee keeping.The real reasons for this may be to try and reduce the negative effects ofwild honey collection on a forest. The impact of such an activity wouldrelate to the extent to which a reduction of wild honey collection leads to animprovement in the ecological values of the forest. The outcomes mayinclude the reduced level of actual wild honey collection, the level of successadoption of bee keeping and extent to which bee keeping increaseshousehold/village income. The outputs would be the number of farmerstrained in bee keeping, the formation of a bee keeping group, the provisionof hives or the development of a honey processing facility. In other wordseven down to the activity level of an objective hierarchy it is possible toexamine impacts, outcomes and outputs.

In development work objective hierarchy thinking has been very muchinfluenced by the logical framework approach which has emphasised theoutput level of planning and subsumed the outcome level into project

purpose. The idea was to focus on the tangible products and services that a

particular project management team should be directly responsible fordelivering. This has problems that manifest themselves in two ways. One,the higher level results or outcomes that a project needs to achieve are notmade explicit and project management becomes very activity/output drivenand hence may not adapt its implementation strategy to achieve higher levelresults when circumstances change. Two, in trying to overcome this

problem and attempting to focus on higher level achievements outputs cometo be used as essentially equivalent to outcomes. These issues relate to thequestions of accountability and control that will be discussed in the nextsection.

The following table summarises the key terms discussed above.


18/36



Term DefinitionGoal The longer term, high level improved situation that a

programme or projects is contributing towards. The goalhelps explain why a programme or projects is beingundertaken. Generally the goal can only be achievedthrough the combined efforts of others and a programmeor project cannot be solely responsible for the goal beingrealised. Sometimes the term goal is used to refer to thehighest level direct results of a programme or project (ie itis used interchangeably with what in this table is definedas the purpose). Vision is sometimes used in place offgoal. Example The ecological, economic and culturalvalues of XXX forest protected for current and futuregenerations.

Impact The extent to which a programme or project, or some partof it, actually makes a contribution towards the goal.Impact is concerned with intended and unintended and

positive and negative contributions.Purpose Overall what a programme or project, within the

timeframe and resources available, should achieve.Example XXX forest being used and managed in asustainable way.

Outcome The observable changes in, for example, behaviour,institutions, economic circumstances or physicalconditions that need to result from a programme or projectin order for it to realise its purpose and make acontribution to the goal. Result or key result is also usedinterchangeably with outcome. Example Forestryofficers carrying out their responsibilities competently andeffectively.

Output The direct services or products that must be delivered forthe outcomes to realised. Result or sub result is also used

interchangeably with output. Example Forestry officerstrained to develop collaborative management agreementswith local communities.

Result Used to refer generally to both outputs and outcomes.There can be lower (sub) and higher (key) level results.Example as for outcome and output.

Objective Used generally to refer to anything that should beachieved by a programme or project. Example as for

purpose, outcome output depending on use.Activity Specific actions that need to be undertaken for outputs to

be produced or outcomes/results/objectives to be realisedExample Training workshop on collaborativemanagement conducted.

Sub Activity/ Tasks

A detailed breakdown of activities to the level required foryearly/quarterly/weekly/daily workplanning.


19/36



3.6 Accountability and Control

As you move up an objective hierarchy it becomes increasingly difficult todirectly control external factors that influence the achievement of results orthe goal of a programme or project. For example, a goal of a project might

be to improve the capacity of a government department responsible for

conservation. Part of this project might involve training. The project candirectly control the hiring of a training venue, the preparation of materials,the provision of a trainer, and the notification of potential participants. It hasless control over whether potential participants will actually attend andconsiderably less control, if any, over whether the skills the participantslearn will actually be used back in the organisational setting.

The issue of accountability and control relates very closely to the notion ofassumptions. For example assumptions will have been made that the

participants would use there skills back in the organisational settingotherwise there would be not rationale for the project (or at least that

particular project activity).

There are two dimensions of accountability and control that are veryimportant to distinguish. The first relates to what a programme or projectmanagement team should be accountable for if they have been given a planto implement. In this case they can really only be held accountable forcarrying out the activities and for the products or services that flow directlyfrom these activities - ie the outputs. The second dimension relates to theoverall accountability of a programme or project, which rests with those whodesign, fund or have overall management responsibility. At this level thereshould be accountability for the higher level results (outcomes) and the

purpose, even though the achievement of these is dependent on actions byothers over which there may be no direction control by the programme or

Withinprojectcontrol

Beyondprojectcontrol

Goal

Activities

Purpose

Key Results(Outcomes)

Sub Results(Outputs)

Tasks

what overall the project canreasonably beaccountable for achieving

Ends what the project

is contributing towards

Means Adapted from Materials Developed by ITAD

The Limits of Control and Accountability

what iswithin thedirect management control of a project


20/36



project. However, the programme or project has been established onassumptions that these actions by others will occur and the programme andshould be held accountable for the assumptions within the design.

Traditional logical framework approaches tend to emphasise the former andsuggest that the purpose of a project is outside the control of a management

team. Results based management approaches tend to emphasise managing toachieve the higher level objectives (results).

3.7 Key Aspects of Evaluation

In developing any monitoring and evaluation system there are five aspects ofevaluation to consider as illustrated below. If you can provide informationon each of these you will be able to judge the overall performance of a

programme or project.

Relevance - Was/is the programme or project a good idea given thesituation to improve? Was the logic of the intervention logic correct? Whyor Why Not?

Effectiveness - Have the planned results been achieved? Why or Why Not

Efficiency - Have resources been used in the best possible way? Why orWhy Not?

Impact - To what extent has the programme or project contributed towardsits longer term goals? Why or Why Not? Have there been any unanticipated

positive or negative consequences of the project? Why did they arise?

Sustainability - Will there be continued positive impacts as a result of the programme or project once it has finished? Why or Why Not?

The Situation toImprove

Problems and Visions

Plan(Goal, Purpose,

Resultsand Activities)

Inputs Activities Actual Results

4. IMPACT

Key Aspects of Evaluation

3. Efficiency

1. Relevance

2. Effectiveness

5. Sustainability


21/36



3.8 Developing an Overall M&E Strategy/Plan

To effectively monitor and evaluate any programme or project it is necessaryto develop an overall M&E strategy or plan. A common failing for many

projects is that the only reference to M&E is the list of indicators andmonitoring mechanisms in the logical framework matrix table. This just

does not provide enough information to guide the actual implementation of aM&E system. The boxes below illustrate the process for developing and thegeneral content for an M&E plan.

Steps for Developing an M&E Plan1. Establish use and scope of M&E system

2. Check project objectives and logic

3. Establish overall evaluation requirements and questions

4. Establish requirements for regular monitoring of implementation

and progress towards desired results5. Test overall M&E strategy with potential users and refine 3 and 4

5. Establish the information and indicators needed for 3 and 4

6. Develop and test regular data gathering / monitoring mechanisms

7. Design open-ended and/pr periodic evaluation activities

8. Design information management system

9. Design a learning and feedback process

10. Decide how to evaluate the evaluation

- key evaluation questions- focussing questions for learning lessons- indicators and monitoring mechanisms- open-ended evaluation activities- participation and responsibilities

Contents for an Overall Project M&E PlanPurpose and scopeOverview of approach (concepts, terminology, methods)General project evaluation activities - eg ...

Annual internal reviews external reviews

M&E details Goal level (impact)

Purpose level Results levelAppendices - eg ...

Budget Details on indicators, monitoring mechanism, reporting Gnat chart of key M&E activities over project life


22/36



A few points are worth emphasising:

1. It is important to be clear about the overall purpose and scope of theM&E system. In particular it should be clear who needs what sort ofinformation for what reasons, how extensive or minimal M&E needs to

be, and what resources are available. For example a project that has a

learning or action research focus will require a more comprehensiveM&E system than a project that is simply implementing a physicalworks programme.

2. The overall system that will be needed for M&E to be effective must beconsidered. This includes designing evaluation questions and indicatorsthat are relevant and practical, training staff in monitoring techniques,developing monitoring forms and reporting processes, establishing aninformation management system and establishing how information will

be analysed, reported and used.

3. It is particularly critical to design learning processes in which staff, beneficiaries, partners and donors participate. For example, annualreview meeting. Information from the M&E system should stimulate,inform and support this learning process.

4. The whole M&E system must be developed around the use ofinformation. If information can't be used its collection is a waste oftime. Yet, it is remarkable how much fragmented and often unusabledata is collected by projects in the name of carrying out M&E.

5. It is important to recognise the difference between regular monitoring of progress vs periodic and in-depth evaluation of some part of or the entire

programme or project. Regular progress monitoring will generally focusmore on output level indicators and the achievement of establishedmilestones or targets. Periodic in-depth evaluation examines whetheroutputs are leading to expected outcomes and impacts, explores reasonswhy and should assesses the effectiveness of the process of the

programme or project.

6. Responsibilities for M&E must be very clear and explicit in any terms ofreference, in job descriptions and be a core part of any staff performancemonitoring and appraisal system. If staff are asked to undertake M&Ework but it is not formalised as part of their core responsibilities it will

inevitably slip to the bottom of the work pile and never get done.

7. Make the M&E plan visual so everyone is aware about it and of theirresponsibilities.


23/36



3.9 Developing and Monitoring Evaluation Questions and Indicators

A good M&E plan should clearly articulate the key evaluation questions thatneed to be asked for each level of the objective hierarchy. To answer theseevaluation questions it will be necessary to identify information needs. Thenecessary information may come from specific quantitative or qualitativeindicators, general project records, generally available information or fromspecially designed evaluative or action research activities.

Traditionally a lot of emphasis has been placed on the development ofquantitative indicators as the key element in developing an M&E plan.Starting at this point tends to narrow down and straitjacket an M&E systemand reduce its usefulness particularly in relation to supporting learning. Forgood reason it is often very difficult or even impossible to develop sensiblequantitative indicators for the goal purpose and outcome levels of a

programme or project. Very often when quantitative indicators have beendeveloped for these levels they are either impractical to monitor or providerelatively useless information in terms of overall evaluation of the result.

There is no question that indicators and in particular quantitative indicators

are an important part of an M&E system and wherever practical they should be used. However, an M&E system will be far more useful if it is designedaround the broad evaluation questions rather than narrowly focusedindicators.

In thinking about evaluation questions and indicators it is important to makethe distinction between evaluation and monitoring. It will often be necessaryand helpful to have some simple indicators that show regular progresstowards a result and which are monitored regularly. Output indicators are

Visualising an M&E Plan

Q1 Q3Q2 Q4 Year 1

Q1 Q3Q2 Q4 Year 3

Q1 Q3Q2 Q4 Year 2

Develop M&Eplan withstakeholders

Preparationfor mid term

Mid termreview

Training in useof reportingsystem

Annual Reviewand Planningworkshop

Annual Reviewand Planningworkshop

PRA withparticipatingcommunities

Preparation for annual review(performance andlessons learnt)

Phase twopreparation

Key Meetings

Report Due

(Illustrative Only)


24/36



particularly useful in this regard. This is the ongoing monitoring of progressthat is required to manage a programme or project and which should showearly warning signs of problems. Evaluation is a more in depth and probingassessment of the whole situation that should explore the reasons for successor failure. This generally occurs less frequently. Different types ofinformation and indicators may be required for regular monitoring vs indepth evaluation. These two different but related aspects should be reflectedin the description of an M&E plan for a particular result.

To effectively monitor and evaluation progress towards any particular result(objective) in a programme or project the following steps will generally beappropriate:

1. Identify the key evaluation questions for each level and result in theobjective hierarchy.

2. For each question identify what information or indicators will be

required to answer the question.3. For each piece of required information or indicator establish:

The methods and frequency for gathering the information ormonitoring the indicator.

The baseline information required for comparison.

What preparation and resources are required for the data to becollected, collated and analysed, for example data collection andanalysis forms, training of staff, data base design, externalexpertise.

Who is responsible for carrying out each of the above and bywhen.

4. For each question, or a set of questions, establish what overall analysis isrequired and how the resulting knowledge will be used and what change

processes need to be in place to learn from and respond to theknowledge.

5. Decide on an overall monitoring and evaluation plan for the particularresult. For example, how often will an overall evaluation of progress be

made and what indicators or information will be used to regularlymonitor progress and how often?


25/36



3.10 Open Ended Evaluation Activities

A good evaluation system should give adequate attention to what shall betermed here open ended evaluation activities. These are all the aspects ofevaluation that complement an indicator based approach. Such open endedactivities (examples of which are given below) are necessary for thefollowing reasons:

1. There will often be unintended positive or negative results andimpacts from a project that will be missed by an evaluation that justfocuses on monitoring predetermined indicators.

2. Monitoring indicators alone often not provide an understanding of whyobjectives have or have not been met. This requires discussion andanalysis with project staff and partners.

3. Monitoring indicators alone will not lead to understanding and learning by programme or project staff and partners.

4. For complex or messy objectives it may not be possible to develop a

easily measurable indicator and the achievement of the objective mayhave to be demonstrated through more anecdotal information.

5. Monitoring indicators provide only limited capacity for evaluation ofthe success or otherwise of the process of the project.

Detailed Result M&E Plan

EvaluationQuestions

RequiredInformation

andIndicators

DataGathering

Methods,FrequencyandResponsibilit-ies

BaselineInformation

RequirementsStatus andResponsibilit-ies

RequiredForms,

Planning,Training, DataManagement,Expertise,ResourcesandResponsibilit-ies

Analysis,Reporting,

Feedback andChangeProcessesandResponsibilit-ies


26/36



Examples of Open-Ended Evaluation Activities

Annual Review and Planning Processes Monthly and/or Quarterly Review and Planning Processes Open Ended Impact Assessment PRAs External Reviews Peer Reviews Stakeholder Meetings Regular Staff Meetings Analysing and Documenting Lessons Learnt Conference Presentations and Papers Advisory Committee Functions

Independent Assessments Staff Performance Reviews Implementing Partner Performance Reviews


27/36



4 Logical Framework Approach (LFA) and ZOPP

The logical framework approach (LFA) and ZOPP have evolved from the1960s as methodologies for improving the systematic planning of

development projects. Over time, they have evolved from simply aframework for structuring project objectives to more sophisticated, processorientated, approaches for involving stakeholders in project design andmanagement.

LFA and ZOPP are based around the following programme/projectdevelopment steps:

1) Systemic and participatory analysis of the situation in which someintervention is anticipated

2) Clearly identifying the problem(s) to be addressed and identifying

the causes and effects of the problem(s). This is usually done bydeveloping a problem tree.

3) Using the situation analysis and problem identification steps toconsider intervention alternatives and to develop a logical hierarchyof activities and objectives that will enable the problems to beovercome.

4) Identifying the assumptions that underlie the logic of the objectivehierarchy ie being explicit about why it is assumed that particularlower lever activities or objectives will lead to higher level ones.Associated with this is identifying the external risks that may lead tothese assumptions not being realised and hence the project notsucceeding.

5) Establishing the indicators that will be used to verify that projectobjectives have been achieved.

6) Developing the means by which information for the indicators will be collected and analysed.

Various groups and facilitators have integrated an extensive range of participatory planning methodologies and tools with the basic LFA/ZOPP

framework and quite sophisticated planning workshops have beendeveloped. There are numerous LFA/ZOPP manuals and documents.

The objective hierarchy for LFA and ZOPP usually has the following levelsand terms:

Goal the long term objective, change of state, or improvedsituation towards which the programme or project is making acontribution.


28/36



Purpose the immediate project objective ie the observable changesin performance, behaviour or status of resources that should occur asa direct result of the programme or project.

Results (Outputs) the products, services or results that must bedelivered by the project for the purpose to be achieved.

Activities the specific tasks that must be undertaken for the resultsto be achieved

The outcomes from such a planning processes are summarised in a project planning matrix (PPM) or logframe table as illustrated below. It is importantto distinguish between the logical framework approach and the project

planning matrix. Often poorly planned projects, that in fact do not reflect anLFA approach, are summarised in such a matrix.

Different terminology is used by different donors and other groups for boththe logframe objective hierarchy and the headings for the columns in the

project planning matrix. The main terminology used by the key donors issummarised below. Its also worth remembering that the staff ofdevelopment agencies arent always themselves familiar with the correctdefinitions of some of the terms they are using. Different parts of the same

organisation may be using the same terms in different ways. Sometimes, theadoption of new terminology within these organisations takes some time toreach all of the employees.

Outputs is the most commonly used term for the level between activities and purpose, however the term results is now becoming more widely used, partlyreflecting the move towards results based management approaches an partly

because there is some confusion within the M&E terminology about themeaning of outputs. IUCN has decided to use the term result rather than

ObjectiveHierarchy

Indicators MonitoringMechanisms

Assumptionsand Risks

Activities

Goal

Purpose

Results(Outputs)

Project Planning Matrix (PPM)


29/36



output. The project planning matrix is usually only shown with one level ofresults (outputs) however it is understood that there can be several levels ofresults (ie key results and sub results) for a large and complex programme or

project.

Conventionally it has been understood that the inputs, activities and results

are within the direct control of a project, while the purpose and goal is beyond direct project control. However this is a blurred rather than clearline and depends on whether one is concerned with overall projectaccountability (design, funding, eventual impact) or the just theaccountability for project implementation. With complex projects that needto be adaptively managed even this distinction becomes blurred.

LFA and ZOPP have become widely accepted as useful and necessary toolsfor project planning, however, they do have their weaknesses that include:

focussing too much on problems rather than opportunities and

vision; being used too rigidly and leading people into a blueprint approach

to project design and implementation;

limited attention to problems of uncertainty where a learning and anadaptive approach to project design and management is required;

the tendency for poorly thought through sets of activities andobjectives to be entered into a PPM table giving the appearance of alogical framework, when in fact the key elements of the analytical

process have been skipped;

the simple logic of the LFA is often not appropriate to programmelevel planning where it may be necessary to deal with a number of

parallel or cross cutting logics.

Despite these limitations and provided due attention is given to the participation of stakeholders, and it is not used to rigidly the LFA/ZOPPapproach remains a very valuable tool for project planning and management.


30/36

Introduction to PM&E Concepts, Approach and Terms

Working Draft Version 1 March 2000

Comparison of LFA Terminology Used by Different Donor Agencies

CIDA DANIDA DFID EC FINNIDA GTZ SIDA

Goal

Goal is becomingthestandardterm at thislevel.

Goal Goal Goal OverallObjectiveOverall

Objective Overall GoalDev

O

Purpose

Purpose orImmediateOutcomeare themain

alternativesat this level

Purpose ImmediateObjective Purpose Purpose Purpose Purpose O

Results

At thislevel, thealternativesare outputsor results.

Outputs Outputs Outputs Results Results Results Resu

ActivitiesActivitiesare used byall

Activities Activities Activities Activities Activities Activities Activitie

(Source: ITAD Ltd Draft Glossary Developed for IUCN)


31/36



5 Results Based Management

Over recent years (Canadian) CIDA and USAID, in particular, have movedto what has been called a results based approach and away from any explicituse of the LFA. This development has arisen for two reasons. First becauseit was recognised that more attention needs to be given to the actual

management of programmes and projects if planned results are to beachieved. Second because there has been growing pressure from donorgovernments for donor agencies to demonstrate more explicitly the impactsof development assistance.

While RMB and logical framework approaches do have slightly differentemphasis the underlying principles are quite similar. In essence they bothattempt ensure logical project design, that results are actually achieved andthat there are mechanisms for monitoring projects and demonstrating whathas been achieved.

Part of the reason for a movement away from the logical frameworkapproach was a perception that it was too ridged and did not provide forenough flexibility in project implementation. Also the move to results basedapproaches is an attempt to link development projects more explicitly to anoverall development strategy for the donor, the country or the region. Donoragencies have become interested in showing the collective impact of theirentire portfolio of development assistance.

There is also a strong theme within the results based management ofmanaging a project to ensure higher lever results or project purpose areachieved. This reflects an explicit recognition of the need for adaptive

project management.

CIDA defines Results Based Management (RBM) as:

a management approach that centres on the establishment of a process andenvironment where individuals work together to accomplish expected results.The RBM process allows project managers to allocate or reallocate scarce

project resources based on performance information and incorporates lessonslearned into project management.

USAID defines Results Orientated Assistance (ROA) or what is also referredto as Managing for Results (MFR) as:

A grant or cooperative agreement awarded to a Development Partner toachieve results that contribute to USAIDs performance goals.

There are three principal elements of ROA:

1) a results orientated programme description

2) a performance measurement system


32/36



3) responsibility for performance

The ROA approach of USAID is designed to show how a particular projectcontributes to the overall development assistance goals that have been set byUSAID and approved by the US Congress.

The main difference between RBM/ROA and LFA/ZOPP is that RMB/ROA

places as much emphasis on management and M&E as it does on the design,while LFA/ZOPP has tended to focus more on planning and design.

The RBM/ROA approaches are specifically designed to enable projectmanagers to cope with change and uncertainty and move away from blue

print development planning. For example, USAID states:

Overly prescriptive input-related detail should be avoided, in order to preserve subsequent flexibility to adapt to changing circumstances on theground during implementation of the activity

However, some experiences with USAID would suggest that this principle is

yet to be fully integrated into the various departments and processes that dealwith project approval. At times one will find different understanding

between the planning departments and the financial management andcontracting departments of agencies, the former saying flexibility andadaptive management is fine while the latter demands much great rigidity.

In essence there is no particular conflict between LFA and results basedapproaches, and LFA can be used in a perfectly complementary way within aRBM context.

To further understand Canadian CIDAs RBM and USAIDs ROA it will behelpful to examine briefly some of the processes and terminology used byeach.

5.1 CIDA RBM Terminology

The basic framework of RBM is set out on the following page.

RBM is defined by the following characterists and processes:

stakeholder participation; defining expected results; identifying performance indicators identifying critical assumptions and performing risk analysis organisational learning performance reporting

The monitoring and evaluation aspect of RMB is referred to as aPerformance Measurement Strategy (or Plan) (PMS).


33/36



RBM uses the following terminology and definitions:

Result : a describable or measurable change in state that is derived from acause and effect relationship

Developmental result : an actual change in the state of human development

that is the logical consequence of a CIDA investment in a developingcountry, measurable at output, outcome and impact levels.

Operational result : the administrative and management product of anagency, its programs or projects.

Goal : the highest level objective that links a programme/project to a widerset of strategies being undertaken to address a specific problem.

Purpose : the second level objective that defines specifically what the programme, project or service is delivering and who are the beneficiaries.

Activities : the coordination, technical assistance and training tasks organisedand executed by the project personnel that transform inputs into results.

Inputs : the human organisational and physical resources contributed directlyor indirectly by the stakeholders of a project.

Impact (corresponds to project goal): a long term developmental result,linked to the goal or vision, that is the logical consequence of achieving acombination outputs and outcomes.

Outcome (corresponds to project purpose/component): a medium termdevelopmental result that is the logical consequence of achieving a

combination of outputs.

Output (corresponds to project activities): a short term developmental resultthat is visible, concrete and tangible and is the logical consequence of projectactivities.

Results chain : the sequence of results (or result hierarchy) from outputs tooutcomes to impacts.

Performance Indicators : qualitative or quantitative measures of projectinputs/activities (operational results) and outputs, outcomes and impact usedto monitor progress towards the achievement of expected result


34/36


35/36

Introduction to PM&E Concepts, Approach and Terms

Working Draft Version 1 March 2000

RESULTS-BASED MANAGEMENTDefining Expected Results

Results Chain

Developmental Res

An actual change in the state of human dthe logical consequence of a CIDAdeveloping coun

Program / Project Management

Operational Results

The administrative and managementproduct of an agency, its programmesor projects

Inputs Activities Outputs Outcomes

The humanorganisation andphysicalresourcescontributeddirectly by thestakeholders of aproject

The coordination,technicalassistance andtraining tasksorganised byproject personnel

A short-termdevelopmentalresult that is thelogicalconsequence ofprojectactivities

A medium-termdevelopmentalresult that is thelogicalconsequencesachieving acombination ofoutputs


36/36


5.2 USAID ROA Terminology

USAID uses very similar terminology to that defined above in relationRBM, however in the Results-Orientated Assistance Sourcebook there is noclearly laid out set of key terms and definitions.

ROA uses the following hierarchy of results (objectives):

In relation to M&E ROA uses the following terms and definitions:

Performance : effectiveness in converting inputs to outputs, outcomes andimpacts.

Performance monitoring : a process of collecting and analysing data to

measure the performance of a programme, process, or activity against itsexpected results.

Performance monitoring plan : a detailed plan for managing the collectionof data in order to monitor performance.

Evaluation : a relatively structured analytic effort undertaken selectively toanswer specific management questions regarding USAID-funded programsor activities. In contrast to performance monitoring, which provides ongoingstructured information, evaluation is occasional. Evaluation focuses on whyresults are or are not being achieved, on unintended consequences, or on

issues of interpretation, relevance, effectiveness, efficiency, impact orsustainability.

ROA Term Accountability forPerformance

Comments

Agency Goal USAID Agency Agency Objectives USAID AgencyMission/OperatingUnit StrategicObjective

Mission and/orWashington-basedOperating Unit SOTeams

It is important recognisethat these top threelevels relate to USAIDsown objectives and sitabove those of afunded project.

Intermediate Result(s) DevelopmentPartner(s)

Outcomes DevelopmentPartner(s)

Outputs DevelopmentPartner(s)

Activities, Strategies,Processes

DevelopmentPartner(s)

These levels areestablished by thedevelopment partner (in

cooperation withUSAID) and define thefunded project.

Documents

Pme Concepts Terms 00