Planning, Monitoring and Evaluating Programmes and …cmsdata.iucn.org/downloads/pme_concepts_terms_00.pdf · Planning, Monitoring and Evaluating Programmes and Projects Introduction

Planning, Monitoring and Evaluating

Programmes and Projects

Introduction to Key Concepts, Approaches and Terms

Working Draft

Version 1 – March 2000

Global Monitoring and Evaluation Initiative

Jim Woodhill IUCN M&E Facilitator for East and Southern Africa

Introduction to PM&E Concepts, Approaches and Terms

Working Draft Version 1 – March 2000 IUCN Global M&E Initiative

IUCN – The World Conservation Union

Founded in 1948, The World Conservation Union brings together States, government agencies and a diverse range of non-governmental organizations in a unique world partnership: over 900 members in all, spread across some 138 countries.

As a Union, IUCN seeks to influence, encourage and assist societies throughout the world to conserve the integrity and diversity of nature and to ensure that any use of natural resources is equitable and ecologically sustainable.

The World Conservation Union builds on the strengths of its members, networks and partners to enhance their capacity and to support global alliances to safeguard natural resources at local, regional and global levels.

The IUCN Monitoring and Evaluation (M&E) Initiative Through an approach which fosters questioning and reflection and engages stakeholders at the regional and global levels, the IUCN M&E Initiative aims to:

• develop a common understanding of M&E within IUCN

• develop a reflective culture within IUCN

• improve project/programme design and implementation through the use of methods and tools in project, systems and institutional assessments

• assess the relevance of the Union’s work against the broader picture of ecosystem and human wellbeing

• improve learning processes and reporting of lessons learned

• put an overall M&E System in place for the Union.

Publications from the M&E Initiative are available on-line on the IUCN website http://iucn.org/themes.html

Acknowledgements Written by Jim Woodhill – IUCN Monitoring and Evaluation Facilitator for East and Southern Africa.

Email – [email protected]

©: (2000) IUCN – International Union for Conservation of Nature and Natural Resources.

Reproduction of this publication for educational and non-commercial purposes is authorised without prior permission from the copyright holder provided the source is fully acknowledged. Reproduction for resale or other commercial purposes is prohibited without the prior written permission of the copyright holder.

Introduction to PM&E Concepts, Approaches and Terms


Contents

1 INTRODUCTION ........................................................................................................................................... 1

2 APPROACHES TO PLANNING MONITORING AND EVALUATION............................................... 2

3 KEY PLANNING, MONITORING AND EVALUATION CONCEPTS................................................. 3

3.1 A PARTICIPATORY LEARNING APPROACH ..................................................................................................... 3 3.2 THE MANAGEMENT CONTEXT ....................................................................................................................... 4 3.3 PROGRAMME AND PROJECT CYCLE ............................................................................................................... 6 3.4 PROGRAMME AND PROJECT LOGIC ................................................................................................................ 6 3.5 OBJECTIVE HIERARCHIES AND ASSUMPTIONS............................................................................................... 8 3.6 ACCOUNTABILITY AND CONTROL................................................................................................................ 15 3.7 KEY ASPECTS OF EVALUATION.................................................................................................................... 16 3.8 DEVELOPING AN OVERALL M&E STRATEGY/PLAN.................................................................................... 17 3.9 DEVELOPING AND MONITORING EVALUATION QUESTIONS AND INDICATORS ........................................... 19 3.10 OPEN ENDED EVALUATION ACTIVITIES .................................................................................................. 21

4 LOGICAL FRAMEWORK APPROACH (LFA) AND ZOPP................................................................ 23

5 RESULTS BASED MANAGEMENT ......................................................................................................... 27

5.1 CIDA RBM TERMINOLOGY......................................................................................................................... 28 5.2 USAID ROA TERMINOLOGY....................................................................................................................... 32

Introduction to PM&E Concepts, Approach and Terms 1


1 Introduction

This document has been developed to assist IUCN staff and partners navigate their way through the terminology that surrounds the practice of planning, monitoring and evaluation (PM&E) in environment and development programmes and projects. It should be considered a working draft and will be revised and improved on the basis of feedback from those who use it. A separate glossary of terms accompanies this document.

There is no question that planning, monitoring and evaluation are fields littered with terminology that is often unclear and which is used with different meanings by different groups. This problem can’t be solved by this document, however, what it can do is to help explain the different approaches to PM&E and how terms are used by different organisations. Unfortunately confusion around terminology often makes PM&E seem much more complex and difficult than is actually the case. In fact there are really only a handful of key concepts that need to be understood about PM&E. If these concepts are understood then it is easy to make sense of the different terminology and to translate terms between different approaches to PM&E.

Monitoring and Evaluation (M&E) is often considered as a separate function and responsibility from planning. However, if a project is poorly planned it is very difficult to monitor and evaluate. Very often M&E staff and specialists find themselves having to go back to basic planning principles before they can assist with M&E. Hence the attention in this document to planning and M&E.

Within IUCN it would simplify the difficulties of confusing terminology if a standard set terminology could be used. The Global M&E Initiatives is working towards such a set of concepts and terms. However, the reality is that IUCN will always be working with different donors who demand the use of their particular approach and terminology. This means that it will always be necessary for staff to understand the underlying concepts of PM&E and be able to translate between the approaches and terminology of different donors and partner organisations.

The confusion about PM&E can also be reduced by understanding clearly the way concepts and terms are used by different approaches and donors. Consequently the later part of this document explores a number of approaches in some detail.



2 Approaches to Planning Monitoring and Evaluation

Broadly there are three main approaches to PM&E in use by the major donor agencies:

1. The logical framework approach (LFA) which is the most common and widely used.

2. The German ZOPP, a close derivative of LFA. The acronym stands for the German equivalent of objective oriented project planning.

3. Results Based Management (RBM) or managing for results, which has become the favoured model of the Canadians and Americans in recent years.

However, even within each approach there are often differences in the use of terminology and many adaptations have been made as different groups put the approaches into practice. Further, those within agencies who should understand the approach being used are often not as clear in their understanding as would be ideal. This difficulty is compounded when agencies are in a transition from one approach to another.

However, while there are certainly differences between the approaches, the underlying principles of PM&E that they are each trying to promote are remarkably similar. In essence, they are:

1. To develop programmes and projects based on a thorough understanding of the situation in which an intervention is planned.

2. To involve stakeholders in a participatory process of programme or project design and evaluation.

3. To develop a set of clear logical objectives that can realistically be achieved within a particular timeframe and within an allocated budget and which will make a significant and sustained contribution to a higher level development objective.

4. To make explicit the cause and effect (means ends) relationships and external factors that underpin the programme or project and which must hold true if planned activities are going to lead to desired results and impacts.

5. To establish a monitoring and evaluation system, including indicators, which will show if the objectives have been achieved and provide information to support effective management and learning.



3 Key Planning, Monitoring and Evaluation Concepts

3.1 A Participatory Learning Approach

The approach to PM&E adopted by IUCN is one that emphasises the participation of stakeholders in continually learning how to improve performance. Monitoring and evaluation is seen very much as a learning process and not as an external top down ‘policing’ function. It is recognised that given the complexity of conservation and natural resource management and an extremely rapidly changing wider environment an adaptive approach to programme and project management is essential. PM&E should be seen as a process of helping people to learn how to do things better. Consequently the theory and practice of adult learning is very important to the monitoring and evaluation approach being developed within IUCN. Given the breadth of this field in itself, the glossary will not attempt to cover it in any detail. However, when using the glossary it will help to keep in mind the idea of a participatory learning approach to monitoring and evaluation.

A participatory learning approach also means that there is much more to M&E than just identifying and monitoring quantitative indicators. Learning implies understanding, analysis, questioning, being critical and trying to explain why things have worked or failed. Certainly quantitative indicators are important and can be helpful but very often they provide only a small part of the information needed for learning. Also for higher level objectives or goals, such as improving protected area management or community well-being it is just simply not possible to develop simple quantitative indicators that have any real meaning. Unfortunately there is a widespread view that developing an M&E plan for a programme or project is primarily about developing a set of such quantitative indicators. This document explicitly challenges this perception. For example, well-facilitated review meetings with staff, or the use of qualitative inquiry methods with beneficiaries, will often provide much more valuable information for learning.



3.2 The Management Context

PM&E are essential functions of good management and should enable a programme or project to achieve a high level of performance as illustrated in the box below.

It is important to see monitoring and evaluation as tools to be integrated into all aspects of programme and project management, as illustrated in the next box. The starting point is to ask; “what information is required for effective management and what sort of M&E system is required to provide it?”

Management Functions and M&E

FunctionsMonitoring

&Evaluation

StaffingStaffing

OrganisingOrganisingControllingControlling

LeadingLeading

PlanningPlanning

PM&E(learning)

Performance

PM&E - Critical Tools for Management

Management

should support

resulting in



Unfortunately M&E is often erroneously viewed as an annoying task of simply providing donors with the information they require. Certainly accountability to funding bodies is one function of an effective M&E system but it is certainly not the only or the most important function. A list of purposes for M&E is given below.

Purposes of Monitoring and Evaluation

Ensuring planned results are achieved

Improving and support management

Generating shared understanding

Generating new knowledge and support learning

Building the capacity of those involved

Motivating stakeholders

Ensuring accountability

Fostering public and political support



3.3 Programme and Project Cycle

The diagram below illustrates a generic programme or project cycle. It emphasises the importance of starting with detailed scoping, situation analysis and design stages. It also illustrates the importance of considering M&E at all stages in the cycle. Importantly it also illustrates the need for constant cycles of planning, acting, monitoring and evaluation (in other words learning) during implementation.

3.4 Programme and Project Logic

There is an overall logic to any programme or project as illustrated below. This logic can be described as follows. First, there is a situation that a group of stakeholders wish to improve – the reasons for a programme or project. This situation is defined, to a significant extent, by the problems and visions of the stakeholder groups. There may or may not be common perceptions of what the problems are or what would actually constitute an improvement, which is why participatory approaches to planning are so important. Developing a detailed and holistic situation analysis is a critical aspect of programme or project planning.

The understanding of the situation will lead to a programme or project plan. This will usually include:

The goal – a summary of what in the long term the programme or project is contributing towards – related to impact.

The Programme/Project CycleOrganisation

Mission

Financing andcontracting

Final Evaluation

Gov/DonorGoals/Policies

BeneficiaryNeeds

Plan

Act

Monitor

EvaluateImplementation

Monitoring andEvaluation

Scoping

Formulation /Design

M&EStrategy

Mobilisation andImplementation Planning

M&EPlan



The Purpose – a summary statement of overall what the programme or project should achieve given its timeframe and resources – the overall outcome.

A set of Results – the main things that must be achieved for the programme or project to realise its purpose, there may be several levels of results. Results are also referred to as outputs and outcomes.

A set of Activities – what must actually be done for the results to be realised.

A set of Inputs – the resources required for the activities to be undertaken.

The programme or project is then implemented according to this plan involving a process of inputs being used to undertake activities that lead to actual results. Of course few projects go exactly as planned and there will need to be constant cycles of planning, acting, monitoring, evaluating, re-planning and so on. In some cases it may be necessary completely revise the original plan.

The actual results should lead to a set of impacts that will improve the original situation. In most projects there will also be unanticipated impacts that may be positive or negative, which are also important to track.

Understanding this basic logic of a programme or project is the starting point for understanding PM&E.

The Situation toImprove

Problems and Visions

Plan(Goal, Purpose,

Resultsand Activities)

Inputs Activities Actual Results

Reasonsfor Programme

Project

ProgrammeProjectand

ImplementatioProcess

IMPACT

Project/Programme Make Up and Logic



3.5 Objective Hierarchies and Assumptions

Any programme or project has an objective hierarchy, as shown below. This maps out the way low level tasks or activities contribute to higher level objectives and how meeting these objectives leads to the achievement of the overall purpose or goal of a project or programme. Such a hierarchy of objectives is also referred to as the intervention logic or the narrative summary of a programme or project. It shows the cause and effect, or means – ends relationships of an intervention. Developing a clear logical objective hierarchy is fundamental to good programme and project design and essential for M&E. It is called an objective hierarchy because any level within it can be seen as being an objective. Higher level objectives are, or should be, a consequence of achieving lower level objectives. Alternatively it could be considered a results hierarchy. Although as will be shown later a distinction is sometime made between an objectives hierarchy being what is planned to be achieved and a results hierarchy being what is actually achieved.

An objective hierarchy can be thought of as like the structure of a tree, with the leaves or twigs being the detailed activities or tasks (low level objectives) and the trunk being the overall goal (highest level objective) and the branches being different intermediate level objectives. In theory there can be many levels to an objective hierarchy, larger and more complex programmes or projects require more levels than do small simple projects. In practice, most planning approaches find four or five levels adequate. For a programme one might imagine the trunk and main branches as being the programme objectives and the sub-branches twigs and leaves as being the projects that contribute to the programme objectives. In a very large project, it may be helpful to think of having sub-projects. For a complex programme

Objective Hierarchy

Purpose

Goal

KeyResult

KeyResult

SubResult

SubResult

SubResult

ACTI

V ITY

ACTI

V ITY

ACTI

V ITY

ACTI

V ITY

ACT I

VITY

ACT I

VITY

ACTI

VITY

ACTI

VITY

ACTI

VITY

ACTI

VITY

KeyResult

ACTI

V ITY

ACTI

VIT Y

ACTI

V ITY

ACTI

VIT Y

SubResult

ACTI

VITY

A CTI

VITY

SubResult

SubResult

SubResult

SubResult

ACTI

VITY

A CTI

VITY

ACTI

VITY

A CTI

VITY

A CTI

VITY

A CTI

VITY

ACTI

VITY

ACTI

VITY

Assumptions(hypotheses)



with large projects, it is conceivable that it may be necessary to have a programme, project and sub-project level of planning and each of these levels having six levels of an objective hierarchy. In the end, planning needs to be taken down to sufficient detail to enable day to day workplans for activities and tasks. A well-developed objective hierarchy makes it clear what must be done to achieve results and in reverse, along with indicators, what results are achieved from completing activities.

An objective hierarchy also has a sideways logic. The outputs, products or results from one strand of the hierarchy will often be critical inputs into another strand. The conventional representation of an objective hierarchy (as shown above) does not illustrate this sideways flow leading to the criticism that such an approach to design is too linear and restrictive. However it is equally possible to map out a project from a systems perspective showing a series of interconnecting systems that have inputs and outputs. It is important that the linear criticism is not used as an excuse for poorly developed intervention logic.

Certainly, for complex programmes it will not always be possible to arrive at a simple hierarchical logic, such as the one illustrate above, that adequately expresses all the dimensions that need to be communicated. There may need to be a number of ‘parallel logics’ or a matrix structure to the programme framework. A detailed discussion of this is beyond the scope of this document.

To assist in the planning process different levels within an objective hierarchy are given different terms (goal, purpose, outcomes, outputs, results, specific objectives, activities, etc). It is the lack of consistency in the way terms are used for the different levels in an objective hierarchy that creates much of the PM&E confusion. However, as long as the concept of different levels in a hierarchy is understood and the meaning of different terms is understood by the stakeholders within a particular context, it doesn’t really matter what they are called. Some examples of different objective hierarchies that have a different number of levels and which use different terminology are given on the next page.



Assumptions

An objective hierarchy should reflect cause and effect relationships between lower and higher level objectives. The lower level activities/objectives are the ‘means’ for achieving the ‘ends’ of the higher level objectives. There will always be assumptions about the cause - effect (means – ends) relationships in an objective hierarchy and these assumptions should be made explicit in the design of a programme or project.

There are two types of assumptions, which are often confused in the logical framework approach to planning. The first type of assumption is that related to the internal logic of the programme or project. For example a project design might be based on an assumption that by reducing poaching in an area an endangered species will be preserved. This may be a correct assumption or it may be incorrect because the main threat to the species might in fact be reduced habitat or disease.

The second type of assumption relates to the external factors or external environment that must exist for a project to succeed. For example, for a tree planting project it may be assumed that rainfall will not be significantly below average. For any project it will be assumed that there will be sufficient political stability for the project to operate effectively.

For either type of assumption it is often helpful to identify, what are referred to as, ‘killer assumptions’. These are the assumptions that if they are wrong the project will fail completely.

Examples of Different Objective Hierarchies

Goal

Activities

Outputs

Purpose

Vision

Activities

Objectives

Goal(s)

Goal

Outputs

Outcomes

Purpose

Activities

Activities

Vision

StrategicObjectives

Mission

Goals

KeyResultAreas

Results

Goal

KeyResults

Purpose

Activities

SubResults



Good project design looks very carefully at assumptions and makes them explicit. Likewise an important part of M&E is checking on the validity of original assumptions.

Sorting Out Objective Hierarchy Terminology

Objective hierarchy terminology, as already mentioned, is the source of most of the confusion around PM&E. To help sort this out it will be helpful to discuss some basic ideas about different levels in an objective hierarchy before getting too caught up in the terminology.

At the highest level of an objective hierarchy it is helpful to place the programme or project within the context of some larger human endeavour. For example an integrated conservation and development project may be contributing towards the protection of the ecological, economic and cultural values of a particular forest. However the project itself will not be able to fully realise this highest level objective or goal, it can only make a contribution. This level provides clarity about why a programme or project is being undertaken. It helps to provide a sense of vision about the future for those engaged with a programme or project. This level is commonly understood as the goal for a programme or project. Some planning frameworks call this level the vision and reserve the goal for the next level down.

Impact is generally used to refer to the extent to which a programme or project in fact does make a contribution towards the goal. However this may not occur during the life of a programme or project and where others are also contributing it may be difficult to desegregate the contributions made by one initiative from those of another. This makes impact evaluation particularly difficult, but nevertheless important.

Below this top level of an objective hierarchy is what can be considered the purpose of a programme or project. This is overall what a programme or project should achieve if it is successful. It is generally considered helpful to try and summarise the purpose as a single statement to ensure focus and clarity. For example “to develop the institutional frameworks and human and organisational capacity for sustainable forest management”. This is what a programme or project should be able to achieve within its available resources and implementation period. If planning for an organisation this level will usually be called the mission of the organisation.

Below this level are a series of major outcomes or results that need to be achieved for the purpose to be realised. These should be the actual observable changes in for example behaviour, institutions, economic circumstances or physical conditions. Again these should be achievable within the resources and timeframe of the programme or project. For example, “staff within the forest department effectively carrying out their responsibilities”. In the LFA this level is subsumed into the purpose level



and it is argued that there are outcomes at the purpose level. Other planning frameworks refer to this level as the main objectives.

Each of these major outcomes or results arise from the logical consequence of the programme or project delivering a set of products or services – outputs – which arise directly from programme or project activities. For example “forestry department staff trained in the skills required to carry out their responsibilities”.

Not all planning frameworks make this distinction between outcomes and outputs. It is this middle level of an objective hierarchy where terms and concepts are most problematic. While outcomes generally refer to the higher order changes or effects that arise from delivering outputs, which are generally considered as tangible or concrete products and services, the distinction is not always so clear cut. There is not a neat dividing line between what can be considered outputs and are outcomes. In fact rather than there being just two cause and effect steps there may be many, ie there are also higher and lower level outputs and outcomes. While a slightly artificial, the output and outcome distinction is still helpful in summarising what can be expected to result at different levels within an objective hierarchy. However, it needs to be remembered that it is a shorthand summary and like all categorisations an approximation of reality.

This ambiguity around outputs and outcomes is why IUCN finds it less confusing to talk of key-results and sub-results within the objective hierarchy. The relationship between these terms is illustrated below.

Objective and Results HierarchiesObjective Hierarchy - what isplanned to be achieved

Results Hierarchy - what isactually achieved(also called impact orhierarchy and results

Actual Overall ResultOutcomes

Actual Key ResultsOutcomes

Actual Sub ResultsOutputs

Goal

Purpose

(Planned)Key Results

(Planned)Sub Results

Activities

Impact

Outputs

Outcomes

Outcomes

Outcomes

Outputs

This is whereM&E terminology

can becomes veryconfusing

Inputs



The lowest level of an objective hierarchy is the activity level. For detailed work planning it will often be necessary to break activities down the sub-activities and/or tasks that will occur, for example over a particular year or quarter.

From an M&E perspective it is very helpful to think in terms of impacts, outcomes and outputs. However it is not necessary to limit impact questions to the goal level of the objective hierarchy or outcome questions to the key-result/outcome level of the hierarchy. This is easiest explained with an example. Take the sub-result of a large integrated conservation and development project - “sustainable livelihood activities adopted in target villages”. An activity of this sub result may the promotion of bee keeping. The real reasons for this may be to try and reduce the negative effects of wild honey collection on a forest. The impact of such an activity would relate to the extent to which a reduction of wild honey collection leads to an improvement in the ecological values of the forest. The outcomes may include the reduced level of actual wild honey collection, the level of success adoption of bee keeping and extent to which bee keeping increases household/village income. The outputs would be the number of farmers trained in bee keeping, the formation of a bee keeping group, the provision of hives or the development of a honey processing facility. In other words even down to the activity level of an objective hierarchy it is possible to examine impacts, outcomes and outputs.

In development work objective hierarchy thinking has been very much influenced by the logical framework approach which has emphasised the output level of planning and subsumed the outcome level into project purpose. The idea was to focus on the tangible products and services that a particular project management team should be directly responsible for delivering. This has problems that manifest themselves in two ways. One, the higher level results or outcomes that a project needs to achieve are not made explicit and project management becomes very activity/output driven and hence may not adapt its implementation strategy to achieve higher level results when circumstances change. Two, in trying to overcome this problem and attempting to focus on higher level achievements outputs come to be used as essentially equivalent to outcomes. These issues relate to the questions of accountability and control that will be discussed in the next section.

The following table summarises the key terms discussed above.



Term Definition Goal The longer term, high level improved situation that a

programme or projects is contributing towards. The goal helps explain why a programme or projects is being undertaken. Generally the goal can only be achieved through the combined efforts of others and a programme or project cannot be solely responsible for the goal being realised. Sometimes the term goal is used to refer to the highest level direct results of a programme or project (ie it is used interchangeably with what in this table is defined as the purpose). Vision is sometimes used in place off goal. Example – The ecological, economic and cultural values of XXX forest protected for current and future generations.

Impact The extent to which a programme or project, or some part of it, actually makes a contribution towards the goal. Impact is concerned with intended and unintended and positive and negative contributions.

Purpose Overall what a programme or project, within the timeframe and resources available, should achieve. Example – XXX forest being used and managed in a sustainable way.

Outcome The observable changes in, for example, behaviour, institutions, economic circumstances or physical conditions that need to result from a programme or project in order for it to realise its purpose and make a contribution to the goal. Result or key result is also used interchangeably with outcome. Example – Forestry officers carrying out their responsibilities competently and effectively.

Output The direct services or products that must be delivered for the outcomes to realised. Result or sub result is also used interchangeably with output. Example – Forestry officers trained to develop collaborative management agreements with local communities.

Result Used to refer generally to both outputs and outcomes. There can be lower (sub) and higher (key) level results. Example – as for outcome and output.

Objective Used generally to refer to anything that should be achieved by a programme or project. Example – as for purpose, outcome output depending on use.

Activity Specific actions that need to be undertaken for outputs to be produced or outcomes/results/objectives to be realised Example – Training workshop on collaborative management conducted.

Sub Activity / Tasks

A detailed breakdown of activities to the level required for yearly/quarterly/weekly/daily workplanning.



3.6 Accountability and Control

As you move up an objective hierarchy it becomes increasingly difficult to directly control external factors that influence the achievement of results or the goal of a programme or project. For example, a goal of a project might be to improve the capacity of a government department responsible for conservation. Part of this project might involve training. The project can directly control the hiring of a training venue, the preparation of materials, the provision of a trainer, and the notification of potential participants. It has less control over whether potential participants will actually attend and considerably less control, if any, over whether the skills the participants learn will actually be used back in the organisational setting.

The issue of accountability and control relates very closely to the notion of assumptions. For example assumptions will have been made that the participants would use there skills back in the organisational setting otherwise there would be not rationale for the project (or at least that particular project activity).

There are two dimensions of accountability and control that are very important to distinguish. The first relates to what a programme or project management team should be accountable for if they have been given a plan to implement. In this case they can really only be held accountable for carrying out the activities and for the products or services that flow directly from these activities - ie the outputs. The second dimension relates to the overall accountability of a programme or project, which rests with those who design, fund or have overall management responsibility. At this level there should be accountability for the higher level results (outcomes) and the purpose, even though the achievement of these is dependent on actions by others over which there may be no direction control by the programme or

Withinprojectcontrol

Beyondprojectcontrol

Goal

Activities

Purpose

Key Results(Outcomes)

Sub Results(Outputs)

Tasks

… what overallthe project canreasonably beaccountable forachieving

Ends … what the projectis contributingtowards

MeansAdapted from Materials Developed by ITAD

The Limits of Control and Accountability

… what iswithin thedirectmanagementcontrol of aproject



project. However, the programme or project has been established on assumptions that these actions by others will occur and the programme and should be held accountable for the assumptions within the design.

Traditional logical framework approaches tend to emphasise the former and suggest that the purpose of a project is outside the control of a management team. Results based management approaches tend to emphasise managing to achieve the higher level objectives (results).

3.7 Key Aspects of Evaluation

In developing any monitoring and evaluation system there are five aspects of evaluation to consider as illustrated below. If you can provide information on each of these you will be able to judge the overall performance of a programme or project.

Relevance - Was/is the programme or project a good idea given the situation to improve? Was the logic of the intervention logic correct? Why or Why Not?

Effectiveness - Have the planned results been achieved? Why or Why Not

Efficiency - Have resources been used in the best possible way? Why or Why Not?

Impact - To what extent has the programme or project contributed towards its longer term goals? Why or Why Not? Have there been any unanticipated positive or negative consequences of the project? Why did they arise?

Sustainability - Will there be continued positive impacts as a result of the programme or project once it has finished? Why or Why Not?

The Situation to Improve

Problems and Visions

Plan(Goal, Purpose,

Results and Activities)

Inputs Activities Actual Results

4. IMPACT

Key Aspects of Evaluation

3. Efficiency

1. Relevance

2. Effectiveness

5. Sustainability



3.8 Developing an Overall M&E Strategy/Plan

To effectively monitor and evaluate any programme or project it is necessary to develop an overall M&E strategy or plan. A common failing for many projects is that the only reference to M&E is the list of indicators and monitoring mechanisms in the logical framework matrix table. This just does not provide enough information to guide the actual implementation of a M&E system. The boxes below illustrate the process for developing and the general content for an M&E plan.

Steps for Developing an M&E Plan1. Establish use and scope of M&E system2. Check project objectives and logic3. Establish overall evaluation requirements and questions4. Establish requirements for regular monitoring of implementation

and progress towards desired results5. Test overall M&E strategy with potential users and refine 3 and 45. Establish the information and indicators needed for 3 and 46. Develop and test regular data gathering / monitoring mechanisms7. Design open-ended and/pr periodic evaluation activities8. Design information management system9. Design a learning and feedback process10. Decide how to evaluate the evaluation

- key evaluation questions- focussing questions for learning lessons- indicators and monitoring mechanisms- open-ended evaluation activities- participation and responsibilities

Contents for an Overall Project M&E PlanPurpose and scopeOverview of approach (concepts, terminology, methods)General project evaluation activities - eg ...☺ Annual internal reviews☺ external reviews

M&E details☺ Goal level (impact)☺ Purpose level☺ Results level

Appendices - eg ...☺ Budget☺ Details on indicators, monitoring mechanism, reporting☺ Gnat chart of key M&E activities over project life



A few points are worth emphasising:

1. It is important to be clear about the overall purpose and scope of the M&E system. In particular it should be clear who needs what sort of information for what reasons, how extensive or minimal M&E needs to be, and what resources are available. For example a project that has a learning or action research focus will require a more comprehensive M&E system than a project that is simply implementing a physical works programme.

2. The overall system that will be needed for M&E to be effective must be considered. This includes designing evaluation questions and indicators that are relevant and practical, training staff in monitoring techniques, developing monitoring forms and reporting processes, establishing an information management system and establishing how information will be analysed, reported and used.

3. It is particularly critical to design learning processes in which staff, beneficiaries, partners and donors participate. For example, annual review meeting. Information from the M&E system should stimulate, inform and support this learning process.

4. The whole M&E system must be developed around the use of information. If information can't be used it’s collection is a waste of time. Yet, it is remarkable how much fragmented and often unusable data is collected by projects in the name of carrying out M&E.

5. It is important to recognise the difference between regular monitoring of progress vs periodic and in-depth evaluation of some part of or the entire programme or project. Regular progress monitoring will generally focus more on output level indicators and the achievement of established milestones or targets. Periodic in-depth evaluation examines whether outputs are leading to expected outcomes and impacts, explores reasons why and should assesses the effectiveness of the process of the programme or project.

6. Responsibilities for M&E must be very clear and explicit in any terms of reference, in job descriptions and be a core part of any staff performance monitoring and appraisal system. If staff are asked to undertake M&E work but it is not formalised as part of their core responsibilities it will inevitably slip to the bottom of the work pile and never get done.

7. Make the M&E plan visual so everyone is aware about it and of their responsibilities.



3.9 Developing and Monitoring Evaluation Questions and Indicators

A good M&E plan should clearly articulate the key evaluation questions that need to be asked for each level of the objective hierarchy. To answer these evaluation questions it will be necessary to identify information needs. The necessary information may come from specific quantitative or qualitative indicators, general project records, generally available information or from specially designed evaluative or action research activities.

Traditionally a lot of emphasis has been placed on the development of quantitative indicators as the key element in developing an M&E plan. Starting at this point tends to narrow down and straitjacket an M&E system and reduce its usefulness particularly in relation to supporting learning. For good reason it is often very difficult or even impossible to develop sensible quantitative indicators for the goal purpose and outcome levels of a programme or project. Very often when quantitative indicators have been developed for these levels they are either impractical to monitor or provide relatively useless information in terms of overall evaluation of the result.

There is no question that indicators and in particular quantitative indicators are an important part of an M&E system and wherever practical they should be used. However, an M&E system will be far more useful if it is designed around the broad evaluation questions rather than narrowly focused indicators.

In thinking about evaluation questions and indicators it is important to make the distinction between evaluation and monitoring. It will often be necessary and helpful to have some simple indicators that show regular progress towards a result and which are monitored regularly. Output indicators are

Visualising an M&E Plan

Q1 Q3Q2 Q4Year 1

Q1 Q3Q2 Q4Year 3

Q1 Q3Q2 Q4Year 2

Develop M&Eplan withstakeholders

Preparationfor mid term

Mid termreview

Training in useof reportingsystem

Annual Reviewand Planningworkshop

Annual Reviewand Planningworkshop

PRA withparticipatingcommunities

Preparation forannual review(performance andlessons learnt)

Phase twopreparation

Key Meetings

Report Due

(Illustrative Only)



particularly useful in this regard. This is the ongoing monitoring of progress that is required to manage a programme or project and which should show early warning signs of problems. Evaluation is a more in depth and probing assessment of the whole situation that should explore the reasons for success or failure. This generally occurs less frequently. Different types of information and indicators may be required for regular monitoring vs in depth evaluation. These two different but related aspects should be reflected in the description of an M&E plan for a particular result.

To effectively monitor and evaluation progress towards any particular result (objective) in a programme or project the following steps will generally be appropriate:

1. Identify the key evaluation questions for each level and result in the objective hierarchy.

2. For each question identify what information or indicators will be required to answer the question.

3. For each piece of required information or indicator establish:

• The methods and frequency for gathering the information or monitoring the indicator.

• The baseline information required for comparison.

• What preparation and resources are required for the data to be collected, collated and analysed, for example data collection and analysis forms, training of staff, data base design, external expertise.

• Who is responsible for carrying out each of the above and by when.

4. For each question, or a set of questions, establish what overall analysis is required and how the resulting knowledge will be used and what change processes need to be in place to learn from and respond to the knowledge.

5. Decide on an overall monitoring and evaluation plan for the particular result. For example, how often will an overall evaluation of progress be made and what indicators or information will be used to regularly monitor progress and how often?



3.10 Open Ended Evaluation Activities

A good evaluation system should give adequate attention to what shall be termed here open ended evaluation activities. These are all the aspects of evaluation that complement an indicator based approach. Such open ended activities (examples of which are given below) are necessary for the following reasons:

1. There will often be unintended positive or negative results and impacts from a project that will be missed by an evaluation that just focuses on monitoring predetermined indicators.

2. Monitoring indicators alone often not provide an understanding of why objectives have or have not been met. This requires discussion and analysis with project staff and partners.

3. Monitoring indicators alone will not lead to understanding and learning by programme or project staff and partners.

4. For complex or messy objectives it may not be possible to develop a easily measurable indicator and the achievement of the objective may have to be demonstrated through more anecdotal information.

5. Monitoring indicators provide only limited capacity for evaluation of the success or otherwise of the process of the project.

Detailed Result M&E PlanEvaluationQuestions

RequiredInformationandIndicators

DataGatheringMethods,FrequencyandResponsibilit-ies

BaselineInformationRequirementsStatus andResponsibilit-ies

RequiredForms,Planning,Training, DataManagement,Expertise,ResourcesandResponsibilit-ies

Analysis,Reporting,Feedback andChangeProcessesandResponsibilit-ies



Examples of Open-Ended Evaluation Activities

• Annual Review and Planning Processes • Monthly and/or Quarterly Review and Planning Processes • Open Ended Impact Assessment • PRA’s • External Reviews • Peer Reviews • Stakeholder Meetings • Regular Staff Meetings • Analysing and Documenting Lessons Learnt • Conference Presentations and Papers • Advisory Committee Functions • Independent Assessments • Staff Performance Reviews • Implementing Partner Performance Reviews



4 Logical Framework Approach (LFA) and ZOPP

The logical framework approach (LFA) and ZOPP have evolved from the 1960’s as methodologies for improving the systematic planning of development projects. Over time, they have evolved from simply a framework for structuring project objectives to more sophisticated, process orientated, approaches for involving stakeholders in project design and management.

LFA and ZOPP are based around the following programme/project development steps:

1) Systemic and participatory analysis of the situation in which some intervention is anticipated

2) Clearly identifying the problem(s) to be addressed and identifying the causes and effects of the problem(s). This is usually done by developing a problem tree.

3) Using the situation analysis and problem identification steps to consider intervention alternatives and to develop a logical hierarchy of activities and objectives that will enable the problems to be overcome.

4) Identifying the assumptions that underlie the logic of the objective hierarchy ie being explicit about why it is assumed that particular lower lever activities or objectives will lead to higher level ones. Associated with this is identifying the external risks that may lead to these assumptions not being realised and hence the project not succeeding.

5) Establishing the indicators that will be used to verify that project objectives have been achieved.

6) Developing the means by which information for the indicators will be collected and analysed.

Various groups and facilitators have integrated an extensive range of participatory planning methodologies and tools with the basic LFA/ZOPP framework and quite sophisticated planning workshops have been developed. There are numerous LFA/ZOPP manuals and documents.

The objective hierarchy for LFA and ZOPP usually has the following levels and terms:

• Goal – the long term objective, change of state, or improved situation towards which the programme or project is making a contribution.



• Purpose – the immediate project objective ie the observable changes in performance, behaviour or status of resources that should occur as a direct result of the programme or project.

• Results (Outputs) – the products, services or results that must be delivered by the project for the purpose to be achieved.

• Activities – the specific tasks that must be undertaken for the results to be achieved

The outcomes from such a planning processes are summarised in a project planning matrix (PPM) or logframe table as illustrated below. It is important to distinguish between the logical framework approach and the project planning matrix. Often poorly planned projects, that in fact do not reflect an LFA approach, are summarised in such a matrix.

Different terminology is used by different donors and other groups for both the logframe objective hierarchy and the headings for the columns in the project planning matrix. The main terminology used by the key donors is summarised below. It’s also worth remembering that the staff of development agencies aren’t always themselves familiar with the correct definitions of some of the terms they are using. Different parts of the same organisation may be using the same terms in different ways. Sometimes, the adoption of new terminology within these organisations takes some time to reach all of the employees.

Outputs is the most commonly used term for the level between activities and purpose, however the term results is now becoming more widely used, partly reflecting the move towards results based management approaches an partly because there is some confusion within the M&E terminology about the meaning of outputs. IUCN has decided to use the term result rather than

ObjectiveHierarchy

Indicators MonitoringMechanisms

Assumptionsand Risks

Activities

Goal

Purpose

Results(Outputs)

Project Planning Matrix (PPM)



output. The project planning matrix is usually only shown with one level of results (outputs) however it is understood that there can be several levels of results (ie key results and sub results) for a large and complex programme or project.

Conventionally it has been understood that the inputs, activities and results are within the direct control of a project, while the purpose and goal is beyond direct project control. However this is a blurred rather than clear line and depends on whether one is concerned with overall project accountability (design, funding, eventual impact) or the just the accountability for project implementation. With complex projects that need to be adaptively managed even this distinction becomes blurred.

LFA and ZOPP have become widely accepted as useful and necessary tools for project planning, however, they do have their weaknesses that include:

• focussing too much on problems rather than opportunities and vision;

• being used too rigidly and leading people into a ‘blueprint’ approach to project design and implementation;

• limited attention to problems of uncertainty where a learning and an adaptive approach to project design and management is required;

• the tendency for poorly thought through sets of activities and objectives to be entered into a PPM table giving the appearance of a logical framework, when in fact the key elements of the analytical process have been skipped;

• the simple logic of the LFA is often not appropriate to programme level planning where it may be necessary to deal with a number of parallel or cross cutting logics.

Despite these limitations and provided due attention is given to the participation of stakeholders, and it is not used to rigidly the LFA/ZOPP approach remains a very valuable tool for project planning and management.



Comparison of LFA Terminology Used by Different Donor Agencies

CIDA DANIDA DFID EC FINNIDA GTZ SIDA World Bank UNDP

Goal

Goal is becoming the standard term at this level.

Goal Goal Goal Overall Objective

Overall Objective Overall Goal Development

Objective

Country Assistance Strategy-

related Goal

Development Objective

Purpose

Purpose or Immediate Outcome are the main alternatives at this level

Purpose Immediate Objective Purpose Purpose Purpose Purpose Project

Objective

Project Development

Objective

Immediate Objective

Results

At this level, the alternatives are outputs or results.

Outputs Outputs Outputs Results Results Results Results Outputs Outputs

Activities Activities are used by all

Activities Activities Activities Activities Activities Activities Activities Activities Activities

(Source: ITAD Ltd Draft Glossary Developed for IUCN)



5 Results Based Management

Over recent years (Canadian) CIDA and USAID, in particular, have moved to what has been called a results based approach and away from any explicit use of the LFA. This development has arisen for two reasons. First because it was recognised that more attention needs to be given to the actual management of programmes and projects if planned results are to be achieved. Second because there has been growing pressure from donor governments for donor agencies to demonstrate more explicitly the impacts of development assistance.

While RMB and logical framework approaches do have slightly different emphasis the underlying principles are quite similar. In essence they both attempt ensure logical project design, that results are actually achieved and that there are mechanisms for monitoring projects and demonstrating what has been achieved.

Part of the reason for a movement away from the logical framework approach was a perception that it was too ridged and did not provide for enough flexibility in project implementation. Also the move to results based approaches is an attempt to link development projects more explicitly to an overall development strategy for the donor, the country or the region. Donor agencies have become interested in showing the collective impact of their entire portfolio of development assistance.

There is also a strong theme within the results based management of managing a project to ensure higher lever results or project purpose are achieved. This reflects an explicit recognition of the need for adaptive project management.

CIDA defines Results Based Management (RBM) as:

“a management approach that centres on the establishment of a process and environment where individuals work together to accomplish expected results. The RBM process allows project managers to allocate or reallocate scarce project resources based on performance information and incorporates lessons learned into project management.”

USAID defines Results Orientated Assistance (ROA) or what is also referred to as Managing for Results (MFR) as:

“A grant or cooperative agreement awarded to a Development Partner to achieve results that contribute to USAID’s performance goals.”

There are three principal elements of ROA:

1) a results orientated programme description

2) a performance measurement system



3) responsibility for performance

The ROA approach of USAID is designed to show how a particular project contributes to the overall development assistance goals that have been set by USAID and approved by the US Congress.

The main difference between RBM/ROA and LFA/ZOPP is that RMB/ROA places as much emphasis on management and M&E as it does on the design, while LFA/ZOPP has tended to focus more on planning and design.

The RBM/ROA approaches are specifically designed to enable project managers to cope with change and uncertainty and move away from ‘blue print’ development planning. For example, USAID states:

“Overly prescriptive input-related detail should be avoided, in order to preserve subsequent flexibility to adapt to changing circumstances “on the ground” during implementation of the activity”

However, some experiences with USAID would suggest that this principle is yet to be fully integrated into the various departments and processes that deal with project approval. At times one will find different understanding between the planning departments and the financial management and contracting departments of agencies, the former saying flexibility and adaptive management is fine while the latter demands much great rigidity.

In essence there is no particular conflict between LFA and results based approaches, and LFA can be used in a perfectly complementary way within a RBM context.

To further understand Canadian CIDA’s RBM and USAID’s ROA it will be helpful to examine briefly some of the processes and terminology used by each.

5.1 CIDA RBM Terminology

The basic framework of RBM is set out on the following page.

RBM is defined by the following characterists and processes:

• stakeholder participation;

• defining expected results;

• identifying performance indicators

• identifying critical assumptions and performing risk analysis

• organisational learning

• performance reporting

The monitoring and evaluation aspect of RMB is referred to as a Performance Measurement Strategy (or Plan) (PMS).



RBM uses the following terminology and definitions:

Result: a describable or measurable change in state that is derived from a cause and effect relationship

Developmental result: an actual change in the state of human development that is the logical consequence of a CIDA investment in a developing country, measurable at output, outcome and impact levels.

Operational result: the administrative and management product of an agency, its programs or projects.

Goal: the highest level objective that links a programme/project to a wider set of strategies being undertaken to address a specific problem.

Purpose: the second level objective that defines specifically what the programme, project or service is delivering and who are the beneficiaries.

Activities: the coordination, technical assistance and training tasks organised and executed by the project personnel that transform inputs into results.

Inputs: the human organisational and physical resources contributed directly or indirectly by the stakeholders of a project.

Impact (corresponds to project goal): a long term developmental result, linked to the goal or vision, that is the logical consequence of achieving a combination outputs and outcomes.

Outcome (corresponds to project purpose/component): a medium term developmental result that is the logical consequence of achieving a combination of outputs.

Output (corresponds to project activities): a short term developmental result that is visible, concrete and tangible and is the logical consequence of project activities.

Results chain: the sequence of results (or result hierarchy) from outputs to outcomes to impacts.

Performance Indicators: qualitative or quantitative measures of project inputs/activities (operational results) and outputs, outcomes and impact used to monitor progress towards the achievement of expected result



RESULTS-BASED MANAGEMENT Defining Expected Results

Results Chain

Developmental Results An actual change in the state of human development that is

the logical consequence of a CIDA investment in a developing country

Program / Project Management

Operational Results The administrative and management

product of an agency, its programmes or projects

Inputs Activities Outputs Outcomes Impact

The human organisation and physical resources contributed directly by the stakeholders of a project

The coordination, technical assistance and training tasks organised by project personnel

A short-term developmental result that is the logical consequence of project activities

A medium-term developmental result that is the logical consequences of achieving a combination of outputs

A long-term developmental result that is the logical consequence of achieving a combination of outputs and outcomes



5.2 USAID ROA Terminology

USAID uses very similar terminology to that defined above in relation RBM, however in the Results-Orientated Assistance Sourcebook there is no clearly laid out set of key terms and definitions.

ROA uses the following hierarchy of results (objectives):

In relation to M&E ROA uses the following terms and definitions:

Performance: effectiveness in converting inputs to outputs, outcomes and impacts.

Performance monitoring: a process of collecting and analysing data to measure the performance of a programme, process, or activity against its expected results.

Performance monitoring plan: a detailed plan for managing the collection of data in order to monitor performance.

Evaluation: a relatively structured analytic effort undertaken selectively to answer specific management questions regarding USAID-funded programs or activities. In contrast to performance monitoring, which provides ongoing structured information, evaluation is occasional. Evaluation focuses on why results are or are not being achieved, on unintended consequences, or on issues of interpretation, relevance, effectiveness, efficiency, impact or sustainability.

ROA Term Accountability for Performance

Comments

Agency Goal USAID Agency Agency Objectives USAID Agency Mission/Operating Unit Strategic Objective

Mission and/or Washington-based Operating Unit – SO Teams

It is important recognise that these top three levels relate to USAID’s own objectives and sit above those of a funded project.

Intermediate Result(s) Development Partner(s)

Outcomes Development Partner(s)

Outputs Development Partner(s)

Activities, Strategies, Processes

Development Partner(s)

These levels are established by the development partner (in cooperation with USAID) and define the funded project.

Documents

Planning, Monitoring and Evaluating Programmes and …cmsdata.iucn.org/downloads/pme_concepts_terms_00.pdf · Planning, Monitoring and Evaluating Programmes and Projects Introduction