Inception Report 1

Inception Report for

Developing An Approach Paper as a Key Step Towards Producing DAC Guidance for Evaluating Conflict Prevention & Peacebuilding Activities

Submitted to the DAC Networks on Development Evaluation and on Conflict, Peace and Development Cooperation by

Collaborative for Development Action (CDA), June 2006 (See also Steering Group Comments in Annex 4, July 2006)

I. INTRODUCTION The field of conflict prevention and peacebuilding (CPPB) is undergoing rapid growth and change. This is, therefore, an opportune time for the DAC Networks to develop guidance on the issues surrounding the evaluation of CPPB activities. As recently as five years ago, attempts to define CPPB evaluation methodologies would have been too narrowly focused for today’s realities. Starting ten years ago but accelerating greatly in the past three to five years, the peacebuilding field has expanded rapidly, both in terms of the numbers and types of actors and in terms of the range of activities that are considered as peace practice. These days, participants in CPPB conferences include experts in the security sector, rule of law, human rights, DDR, humanitarian assistance and development cooperation, as well as peace and conflict resolution specialists. Such participants also range from grassroots activists to international diplomats. These new specialists also operate from a broad range of institutions, including private companies and contracting agencies, local NGOs and INGOs, UN agencies with various mandates, multilateral institutions, and a widening array of donor government departments, including, at a minimum, units addressing foreign policy, development and defense. The expansion of actors and programming approaches reflect significant differences that have also emerged in the types of conflicts where international efforts are undertaken. As often noted, many current conflicts take place within, rather than between, nations and are fought among multiple sub-groups. Command structures of armed groups are often loose and volatile, exercising less control over foot soldiers and their actions than formal armies. As a result of these trends, a great deal more CPPB programming is now focused at the grassroots level and among civil society actors than was previously the case. In the early 1990s, the preponderance of peace work focused on high level negotiations, often including representatives of governments in conflict with each other. Today, multi-level, dispersed and often unconnected peace programmes proliferate simultaneously and involve many sectors of conflicting societies. These added dimensions of peace processes are sometimes helpful and sometimes not—but have not been assessed thoroughly. The “peacebuilding palette” presented in the Utstein Report clearly reflects the expansion of what is considered peace programming in recent years.

Inception Report 2

These realities pose several challenges to the evaluation of CPPB activities. The first challenge is uncertainty regarding the boundaries of CPPB work. The many new actors with multiple perceptions, different histories and varying mandates use the same terms but with divergent meanings. There is no widely-accepted agreement regarding what is, and is not, a CPPB activity. A second challenge has to do with multiple assumptions, implicit theories and “fuzzy” goals. Many of today’s variety of CPPB activities are based on unclear or unproven assumptions that reflect implicit theories of change, or theories of how peace is built. Multiple actors implementing programmes in the same area, each employing his/her own assumptions/theories can complicate efforts to test assumptions. A fundamental role for evaluation is to test assumptions in order to determine which are valid in the context. Directly linked to the issue of assumptions and theories is the question of goals. Again, the full range of actors is pursuing a plethora of activities based on (sometimes conflicting!) goals. In CDA’s experience, the evidence suggests that, in many instances, the goals themselves are neither clear nor connected logically (through assumptions and theory) to a strategy for achieving desired change. Attempts to evaluate effectiveness and to determine how to account for change (or the lack of it!) are encumbered by the nexus of assumptions, theories and goals among multiple actors. And a third challenge concerns varying emphasis on programme processes as opposed to programmatic impacts. Some actors or programmes assert that their positive contribution is assured simply by the initiation of a process (of dialogue, reconciliation, relationship building, negotiation, etc.). Others undervalue such processes, looking only for clear impacts of activities on larger social systems. Guidance for evaluation should recognize and attempt to address this tension in the field. CDA welcomes these challenges and the opportunity to work with the DAC Networks on disentangling these threads and sorting out what the accumulating experience and evidence can tell us about how to address these challenges. II. CDA’S METHODOLOGY In all of our work, CDA is committed to evidence-based learning. Our approach is always to gather broad experience and, by comparing and analyzing it, to identify common themes, issues and lessons across a range of contexts. We do this in collaboration with colleagues who are themselves experienced in the fields where we work. As we develop an approach paper for guidance on CPPB evaluation, we also intend to be experience- and evidence-driven. Given the timing and budget of the DAC Networks’ TORs, we will focus on gathering experience through three basic mechanisms:

Focused interviews Document review and analysis Learning from CDA’s projects

Inception Report 3

Focused Interviews The TORs specify that we should conduct 15-20 in-depth interviews (see Annex 4 references to additional interviews). In preparation for the writing of this Inception Report, we have done four selected interviews. Our selection for the additional interviewees (after the Inception Report) will be based on a) identification of categories of people that represent a range of perspectives on the issues raised by the TORs and b) consultations with colleagues about individuals within these categories who are available and would be interested in contributing to the DAC learning. We also hope that the DAC Networks will provide feedback to our selection when receiving this Inception Report and make additional suggestions.

The five categories of people we have identified as important to interview are:

1. Members of the DAC steering group (for this effort) and selected members of the Evaluation and CPDC networks (government and multilateral donors): From this group, we seek both advice on issues that each member feels should be addressed in the approach paper, and we also hope to discuss the full range of issues and experiences that we will include in all other interviews.

2. Other donor representatives (beyond the DAC networks): From this group, we will invite especially ideas on a) unclear/unresolved issues that need further attention; b) the use and usefulness of CPPB evaluations; and c) how their donor and governmental CPPB policies affect their work and its effectiveness.

3. Implementers/Practitioners (NGOs): With this group, we will focus on questions that have to do with effectiveness, strategy and substance of programming and how to assess the outcomes and impacts of these.

4. Evaluation Specialists (beyond DAC networks, private and/or within NGOs): Here we will concentrate on their assessments of the state of the art of CPPB evaluations, with particular attention to how CPPB evaluations differ from evaluations in other fields and their ideas of the most appropriate (or inappropriate) methodologies for CPPB evaluations and why.

5. Academics/Others: There is a small group (see attached list of potential interviewees) of others who are close to the fields of CPPB and who are good observers of how things progress in the field as a whole. With this group, we will explore all the questions and others that they introduce.

The individuals we have identified to interview are included in Annex 1. The types of questions we have developed as guides to our interviews are also attached in Annex 2. Note that these questions do not constitute a “questionnaire.” Rather, these are meant to guide an in-depth conversation. The purpose of these interviews, cumulatively, is to survey the field of opinion and experience regarding CPPB evaluations and the most salient and pressing issues to be

Inception Report 4

dealt with in these evaluations. Also, we will use these as one basis for gaining a sense of which methodologies have proven most (or least) useful. Review of Documents We will not duplicate or repeat the document reviews done for the Utstein Report and by FAFO and others, but will build on their helpful findings. To a great extent, the TORs have already identified the areas, highlighted in both previous reports, that deserve additional sorting and clarifying (see Annex 5, ToRs). We are identifying additional documents to review, because they appear to be closely related to these remaining questions posed in the TORs and will provide another source of evidence for addressing these issues. Most of these have been produced in recent months. The categories of documents we are reviewing include:

1. Conflict sensitivity literature, including PCIAs and other selected conflict

analysis tools and reflections on their usefulness. 2. Recent conflict prevention and peace building literature, particularly those

items that analyze the boundaries of the fields and evidence of varying approaches. These will be helpful in clarifying definitional issues.

3. Evaluation literature, including a. Most significant change literature b. Goal free evaluation literature c. Focus-structured comparisons d. Theory-based evaluation literature

4. Literature on how to assess when circumstances are improving (e.g., CSIS in Iraq).

5. Policy statements of donors and multilaterals, and some NGOs regarding CPPB principles and approaches.

6. Meta-evaluation literature especially those of DAC donors not yet covered by Utstein or FAFO.

7. Other evaluations, assessments and case studies named in our interviews as especially useful/instructive.

The overall purpose of these literature reviews will be to give us a solid awareness of the recorded state of the art of evaluations, allow us to review the broader set of potential approaches to CPPB evaluations and how methodologies from other fields are, or are not, relevant and useful for CPPB. Further, we will be able through some of these materials to clarify implicit and explicit assumptions and theories of change that guide CPPB policies, programmes and evaluations. Learning from CDA’s Projects We will review and include experience and evidence gathered through other CDA projects and evaluations. The set of case studies just completed for our Steps Towards Conflict Prevention project will provide evidence regarding effective approaches to prevention and how they can be assessed. The case studies and consultation reports from

Inception Report 5

Do No Harm are relevant and instructive with regard to analyzing and assessing conflict sensitivity approaches, particularly for programmes in humanitarian and development assistance.

The Reflecting on Peace Practice project documents also contain a great deal of relevant evidence about definitions, boundaries, processes, assumptions, etc. of CPPB activities and how these have, or have not, been found to contribute to effective peacebuilding and/or conflict prevention. In addition, in recent months, RPP has focused directly on developing alternative approaches to conflict analysis, on identifying and testing impact indicators, on tracing cumulative effectiveness of multiple CPPB activities and on understanding micro-macro CPPB relationships. Although the efforts to push learning in these four areas are not yet complete, the experience that has already been gathered will add evidence to our exploration of key issues for the DAC approach paper. III. FEASIBILITY OF THE TERMS OF REFERENCE Priority Issues Drawing on the issues highlighted in the TORs, preliminary inputs from the pre-Inception Report interviews, and an initial document review, we have identified the following areas for priority focus in the Approach Paper:

1. How CPPB Evaluations May Differ from Other Fields. We will examine how the particular characteristics of CPPB programming affect the relevance, or not, of various evaluation methodologies developed for other areas of programming. So far, several characteristics have been identified as setting CPPB programming and evaluations apart: The essentially political nature of CPPB programming, (which must address

political relations directly, whereas, while other types of programming have political implications, these can be secondary to their primary mandates and purposes);

The complex and often violent context of CPPB work, which can limit access; The explicit biases that are to be expected from potential informants; The dearth of baseline data relevant to CPPB activities; The longer time frames of much CPPB work.

In this section, we will also address the relevance of the DAC Evaluation Criteria and the usefulness of the Humanitarian Assistance Guidance as a model for CPPB guidance. 2. Expectations of CPPB Evaluations: We will review the evidence and opinions

regarding the reasonable expectations of CPPB evaluations. This line of inquiry will include an exploration of evaluation standards (data collection techniques, pre- and post-testing/analysis, control groups, etc.), and whether it is reasonable to attempt to assess impacts (as opposed to outputs and proximate outcomes). This review will also examine the issue of timing: a) how soon one can see CPPB

Inception Report 6

results, and b) how do the phases and types of conflict affect our ability to recognize impacts with reasonable certainty. We will also address the reasonable expectations of any given CPPB activity, within its context and in relation to broader strategic objectives regarding Peace Writ Large.

3. Levels of Intervention: We will examine the evidence regarding evaluations at

different levels of societal interventions and the relationships among these levels. Included here will be consideration of the distinctions and uses of project evaluations vis-à-vis context evaluations—that is, the differences between evaluations focused on the impacts and effectiveness of individual programmatic efforts and those that assess the cumulative impacts of multiple programmes in the same conflict zone. We will also explore assessment efforts at the community, national, regional and/or international levels, looking for ways to provide guidance on whether and how evaluations at these levels require different approaches.

4. Context Analysis and Indicators: The CDA team will gather evidence regarding

the importance of conflict analysis, as it affects both programme and policy effectiveness and evaluations of programmes and policies. Preliminary evidence suggests that some form of context analysis is necessary for developing clear goals and strategies—and for identifying appropriate indicators for monitoring and evaluation. We will also be gathering experience with CPPB indicators, in order to determine if there are some that have potential universality, as opposed to those that will always be context-specific.

5. Assumptions and Theories of Change: We will clarify the range of assumptions

and theories of change that underlie the various types of CPPB programming. Evidence to date shows that many CPPB activities (both programmes and policies) are based on unexpressed or unexamined assumptions and theories. We will suggest ways to make assumptions and theories more explicit, as a critical step in good evaluation.

6. Relationships with Host Governments: We will examine the relationship of

CPPB activities to interactions with governments. Development principles (including, for example, DAC Guidance and the Paris Declaration) place great emphasis on working with and through local authorities. However, the experience with CPPB activities suggests that governments are often important parties to conflict, requiring, therefore, somewhat different approaches to working with host governments. Future guidance on CPPB work and CPPB evaluations must address the range of approaches to governmental relations.

7. CPPB Evaluation Methodologies: Finally, the examinations and clarifications of

all of the topics described above will inform our review of a range of evaluation methodologies as we attempt to discern which approaches, or parts of approaches, could be best adapted to assess impacts and outcomes of CPPB programmes. We will include specific examples of evaluations that have proved particularly useful.

Inception Report 7

Realistic Limits of the Terms of Reference Categorising CPPB Activities for Evaluation Guidance The TORs call for an Annex addressing issues for evaluating CPPB programming in a range of sectors. This effort would require several steps:

1. Clarification of the overall domain of CPPB—its boundaries, limits, what activities/programmes/interventions fall in and outside of it;

2. A more comprehensive identification of the various sub-categories within CPPB work (such as security sector reform, DDRR, reconciliation, etc.) and the activities generally associated with those sub-categories; and

3. Identification of key issues and approaches to addressing challenges in evaluating programmes in each sub-category, based on a thorough examination of the literature and engagement of sector specialists.

Under the current TORs, we will address #1 above fully, and make significant progress towards #2. With regard to #3, we will provide an overall framework that will be useful for developing further specific procedures for evaluating specialized programmes in the sub-categories. In addition, in a few sample areas where CDA already has considerable knowledge, we will offer proposed evaluation methods and considerations. Further, we will suggest how in-depth elaboration of considerations for evaluation in the sub-categories can be addressed in subsequent efforts beyond the scope of this TOR. Cross-Cutting Issues We recognize the importance of the cross-cutting issues noted in the TORs (including, gender, rights-based programming, HIV/AIDS, and others). However, we realize that to cover all of these issues adequately, we would need to read the literature and interview knowledgeable individuals in each of these issue areas. We estimate that within the current plan for fulfilling the TORs, we will not have time to address the cross-cutting “lenses” adequately. However, as we conduct the overview work outlined above, we will seek information to address these topics, and we commit to reporting on all that we do learn about these and, additionally, to providing a suggested plan for gaining greater depth in subsequent processes. IV. WORK PLAN AND BUDGET See Appendix 3.

ANNEX 1: POTENTIAL INTERVIEWEES: TRACKING TABLE (This list takes into account comments from the SG, see Annex 4) Category/Possibles Initial Contact Made Completed Evaluation Specialists Michael Woolcock/JFKSch Harvard Niels Dabelstein/Denmark Koenraad van Brabant/WSP Catherine Barnes/UK Nick Mabey/e3g.org-UK Uwe Kievelitz/Germany Tony Vaux Mark Keen * Jon Hanssen-Bauer [NORAD] Arne Strand [NORAD]

Michael Lund* Cheyanne Church

DAC Steering Group Henrik Mungenast/SIDA

Beate Bull/ NORAD Julia Compton/DFID Cristina Hoyos/DEZA

Jennifer Stuttle/DFID* Bjorn Holmberg/SIDA* Agnete Eriksen/ NORAD

Other DAC Stein-Erik Holjen/NORAD Krishna Kumar/USAID Ciru Mwaura/DFID* Steve Ainsworth/DFID* Simon Arthy/DFID Mark Segal/DFID* James Fennell/DFID Rob Wilkinson/DFID* Tim Heath/DFID Debi Duncan/DFID

Patrick Merrienne/DFID* Jill Tirnauer/USAID/CMM Ruairi O’Connell/DFID

Private Donors Steve Pittam or Juliet Prager/Joseph Rountree Trust/UK Carnegie Corporation

Melanie Greenberg/former Hewlett Fdn

Multilaterals Peter Bachtelor + Kathleen Cravero/UNDP Ian Bannon/Colin Scott/WB

Connie Peck/UNITAR Gay Rosenblum-Kumar/UN-DPA

Ron Parker + team/WB Andrew Russell/UNDP Cyprus Jaco Cilliers/UNDP Cyprus

Implementers/Practitioners Gavin Preuss/Oxfam Hizkias Assefa (Kenya) David Moussa Ntambara (Rwanda) Vasu Gounden/ACCORD (So. Africa) Frederic Kama-Kama Tutu (PTN) NPI-Africa Rachel Goldwyn /CARE* Alliance for Peacebuilding (Boston grp) Kosovo RPP group? Tyrol Ferdinands/Sri Lanka Andres Serbin/Argentina Augusto Miclat/Philippines

John Paul Lederach Liz McClintock

Dan Smith/IA Tale Steen-Johnson* Mohammed Abu-Nimer/AU Howard Wolpe/WWIC Pamela Aall/USIP Anton Baare Emmanuel Bombande/WANEP

Academics/Others John Erikson Peter Wallenstein/Uppsala Peter Uvin/Tufts

Eileen Babbitt/Tufts

Jonathan Goodhand Mohammed Abu Nimer/AU

* Some SG priority suggestions ** Interviews completed before Inception Report.

ANNEX 2: INDICATIVE INTERVIEW QUESTIONS 1. Have you ever read a really good evaluation of a CPPB programme? What made it good?

What was most helpful about it? 2. What is hardest to evaluate in CPPB programming? What have we still not found a way to

assess? Why would that be important? What are the key open questions and challenges regarding the evaluation of CPPB programmes?

3. What progress has been made—can we point to anything that we now “know” about

evaluating CPPB? 4. What have proven to be useful indicators for monitoring progress? In what context, for what

kinds of programs…? What indicators have proven unhelpful? 5. What are the purposes of CPPB evaluations? How do people actually use them (or not)? Do

people follow-up to implement recommendations, make programmatic changes, etc.? [Explore accountability vs. learning purposes.]

6. Which evaluation methodologies seem to be most helpful for CPPB activities (classic results

models; theory-based; policy evaluations, etc.)? For which types of programs? 7. What are the boundaries of CPPB? What is/is not included? What are the useful sub-

categories and what activities fall within those? 8. Who else should we be talking with? 9. (For donors/government representatives): What are your policies regarding CPPB? How do

they influence your work and the choices you make for funding? 10. What are the key factors/issues that should be addressed in a DAC guidance document on

CPPB evaluation?

ANNEX 3: CDA WORK PLAN /BUDGET Workplan Time estimates: Mary B. Anderson, Executive Director and Chief of Project 28 days Diana Chigas, Co-Director, Reflecting on Peace Practice Project 15 days Peter Woodrow, Co-Director, Reflecting on Peace Practice Project 15 days Marshall Wallace, Director, Do No Harm Project 4 days Tarah Farman, Project Associate, RPP and DNH Projects 10 days Total 72 days • CDA will submit the Inception Report to the DAC Networks by May 29, 2006. 1. The CDA group will meet together to outline and discuss the issues raised in the DAC TORs

and notes from comments on CDA’s previous proposal in order to outline and focus the Inception Report.

1 day 2. CDA will undertake focused aspects of the literature review and some focused interviews to

a) assess more carefully the range of ideas and issues identified by the CPPB field as critical or troublesome for evaluation and b) to ensure our later interviews and literature reviews are well focused.

5-6 days 3. Mary B. Anderson (MBA) will write the Inception Report; others will review.

3 days for MBA, ½ day each for all other staff for a total of 5 days If given go-ahead by the DAC Networks to proceed: 4. Diana Chigas (DC) will read and review additional ephemera and other recent evaluation

reports and analyses, to update background for our work with items produced since the Utstein and FAFO reports.

2-4 days 5. Mary B. Anderson will collect and review additional pertinent donor policy papers.

2-3 days 6. Peter Woodrow (PJW) will interview relevant people in CPPB, in Evaluation groups and

among donors. 3-4 days

7. Marshall Wallace (MW) will interview selected personnel from humanitarian NGOs and possibly donor groups that have added CPPB mandates to their work.

2-3 days 8. On the basis of this background from interviews and literature review, and CDA learning

from previous evaluations, RPP and DNH, will write the Approach Paper. Writing and editing will occur during July and August. CDA will submit a draft of the Approach Paper to DAC by end of August 2006

MBA 11-14 days PJW and DC 5-9 days

9. Revisions and additions on basis of DAC Network and other expert comments through first

half of September 4 days

10. CDA will help plan DAC workshop where Approach Paper will be reviewed. 2 days

11. Workshop travel and participation. 4 days

12. Final revisions/rewrite of Approach Paper based on workshop feedback/suggestions. 3 days

13. Tarah Farman, will be involved throughout in gathering materials for review, setting up interviews, keeping track of discussions, providing back-up to writing and layout expertise, etc.

10 days Budget Mary B. Anderson (28 days at 700/day) 19,600.00 Diana Chigas and Peter Woodrow (15 days each at 600/day) 18,000.00 Marshall Wallace (4 days at 600/day) 2,400.00 Tarah Farman (10 days at 400/day) 4,000.00

Staff Fees $44,000.00 Estimated interview costs* 1,000.00 Financial management, accounting 1,500.00 *Direct costs associated with interviewing process. PJW and MW will hold some of the interviews in person when they are traveling near locations where there are people we wish to interview.

Direct Costs $2,500.00 Subtotal $46,500.00 Overhead 15% 6,957.00 Includes costs apportioned to ongoing office expenses such as rent, insurance, equipment, etc. Total USD $53,457.00 Total in Euros (exchange rate of .82825) €44,276.00

ANNEX 4. SUMMARY OF COMMENTS FROM CPDC-EVALUATION NETWORKS STEERING GROUP (SG) MEMBERS ON INCEPTION REPORT AND CONFERENCE CALL WITH CDA AND THE SG (6 JULY 2006)

This Inception Report (IR) was well received, and the following points came up in a discussion between Steering Group members (NORAD, DFID, SDC and SIDA) and Mary B. Anderson, Head of Collaborative for Development Action (CDA). These notes are designed to enhance and possibly expand on some areas of the IR.

1. Expectations about how far the "approach paper" will go toward guidance: The Steering Group had decided that before developing full guidance an "Approach Paper" would be needed that goes a relatively long way toward producing guidance. This should be useful as an in-depth background paper to support a short evaluation guidance publication that will be developed based on this paper and a consultative workshop. Note: CDA team should consider "approach paper" as a working title only for now and decide on the most relevant term/title as it develops, as some were not completely satisfied with the term itself.

2. Audience: It was agreed that the “approach paper” (and the guidance) would be aimed at donor agency staff primarily (mainly field, also headquarters), as this is the DAC's comparative advantage. However, as is the case for other DAC guidance, it should be useful for a wider audience working on evaluations of CPPB activities.

3. Outputs and outline: Mary B. Anderson pointed to Section III in the inception report on "Priority Issues" which covers the top seven issues that CDA intends to explore and will form the basis of the broad outline for the paper. Action: It was agreed that specific recommendations for CPPB evaluation guidance would be added to that list. Mary B. Anderson noted that outlines evolve in the writing process, so this was not a strait-jacket.

4. Developing more in-depth information on the areas of the CPPB domain [(Approach paper annex), see "Realistic Limits of the Terms of Reference"]: As stated in the IR, CDA will provide an overview of the CPPB domain and an overall framework that will be useful for developing further specific procedures for evaluating specialized programmes in the sub-categories of CPPB. In addition, CDA will offer proposed evaluation methods and considerations in a few sample areas where the team already has considerable knowledge. CDA will also provide in-depth elaboration of considerations for evaluation in the sub-categories which can be addressed in subsequent work. The group discussed the question of how to address sub-sectors and areas where CDA has less experience (e.g. SSR, DDR). Many of these are ‘state-level’ interventions that may require a contrasting approach to evaluation to areas such as dialogue where CDA has most experience. It was agreed that two contrasting examples (dialogue and SSR) would be used as case studies in the approach paper to highlight the different issues involved. It was suggested that a small experts' workshop (day and a half with about 20 participants) could be held to conduct two parallel working groups on these topics that would allow us to draw out approaches, see what is tailored to certain areas, what applies to different ones, and seek to identify gaps. This would be an effective tool to complement work that CDA can provide in a few sample areas, as noted above. It is important to note that this workshop would substitute for work on dialogue in the first draft approach paper which is an area where CDA does have good experience and knowledge. By putting this topic into the workshop, the group decided that CDA would not

try to handle it before the workshop. CDA could rather take up training, or organizing joint projects instead. Action: Look into logistics of a small complementary experts' workshop on "evaluation methods in dialogue and SSR activities" (about 20 experts – 10 for each) to help identify evaluation methods in these fields. This would also support the CPDC work on M&E for security system reform. CDA could provide support (CDA to cost, once the Secretariat has specified how much of the logistics/planning can be done by SG members/the Secretariat). (Note: DFID offer to host this at the Global Facilitation Network for SSR, Birmingham, UK to combine with existing plans to do work on evaluation of SSR activities).

5. Interviews, literature and methodology On interviews, the SG felt it was important to be sure that approaches tried by partner countries were reflected and that those on the receiving end of programmes provide information from their perspectives. Mary B. Anderson pointed out that interviews with Southern partners were already being conducted (Ghana, Kosovo, Cyprus), including with a number of people form conflict zones who have been on the recipient side of CPPB work and evaluations. This came up in early interviews (as outlined in the Inception Report), and CDA immediately began to pursue it. Another point was raised that, since donors are all moving toward the Paris Declaration with different types of support, it is important to talk to a range of different types of donors/actors (State and non-State actors, e.g., NGOs and INGOs). The contrast between ‘state building programmes’ under a country-led approach (where evaluation may well be through a joint donor-government process) and individual project approaches was particularly important to cover. Mary B. Anderson noted that there is in the overall interviews, however, a time when one sees diminishing returns. As more interviews are done, the repetition becomes greater across a number of issues, a caution to the SG about wanting to spend too much time on additional conversations. On methodology, it was also suggested that a table could be developed in reflecting on usage that sets out the kinds of analysis/evaluations being used for which types of activities, if this could be helpful for guidance purposes. CDA noted that they do not plan to review all the evaluation methodologies available but are exploring what people have seen to be useful and what they have not found helpful. Action: First, the Steering Group has submitted further ideas for interviews with contacts and the priority of these (see updated interview table below). Given this, CDA will cost some additional interview days. The key issue is to ensure balanced coverage, particularly with partner country practitioners.

Draft Timeline

July: Finalise any amendments to the contract with CDA (e.g., more interviews/small workshop support) and circulate Inception Report to Network members. August: First draft approach paper complete by end-August to circulate to members for comment by (11 September). Decide on date of main workshop event, “Evaluating Conflict Prevention and Peace building Activities”. September/Early October: Possible small experts' workshop on different evaluation methods in dialogue and SSR (Note: DIFD offer to have this hosted by the Global Facilitation Network for SSR).

16-17 October - CPDC Network meeting: CPDC Task Team on Evaluation to meet in margins of CPDC for update and input on work. October or November: Could hold main workshop event, “Evaluating Conflict Prevention and Peace building Activities” to review full draft. 16-17 November 2006: Evaluation Network meeting. First quarter 2007: Goal for developing draft Guidance based on Steering Group and Members of CPDC and the Evaluation Networks thoughts on steps needed to develop the “approach paper” into draft guidance, particularly to cover sub-sectors/areas not in that piece. Piloting the draft guidance to begin and be conducted in 2007-8.

CONSULTANT TERMS OF REFERENCE Developing Guidance for Evaluating Conflict Prevention and Peace Building Activities:

An Approach Paper

DAC Network on Development Evaluation and

DAC Network on Conflict, Peace and Development Cooperation

Introduction The DAC Network on Conflict, Peace and Development Cooperation (CPDC) and the DAC Network on Development Evaluation are collaborating on the development of guidance for evaluating conflict prevention and peace building (CPPB) activities, an area where only minimal guidance already exists. The two Networks have agreed to commission a consultant to develop an approach paper that provides in-depth exploration of the options for practical guidance for CPPB at programming and policy levels.

The approach paper will require tasks such as a literature review of key DAC and donor agency policies and guidance on evaluation and on conflict prevention and peace building (CPPB) and in-depth interviews with a select number of key informants. The approach paper will explore and define possible ways forward, highlight the main methodological issues and recommend next steps for developing guidance on evaluating conflict prevention and peace building activities.

This will build on work previously carried out by the DAC Evaluation Network, by the CPDC and by the Utstein Group1. The consultants should review and utilise relevant DAC, donor and other publications, including the DAC guidance documents on conflict prevention and security system reform as well as the recently developed issues briefs on CPPB (www.oecd.org/dac/conflict) and the guidance documents available on the DAC Evaluation Network website (www.oecd.org/dac/evaluationnetwork), along with a draft report which seeks to review substantive and methodological lessons from past evaluations of CPPB activities.

Questions and Challenges: Towards Developing Guidance for Evaluating Conflict Prevention and Peace Building Activities In developing the approach paper, the style and approach of the Evaluation Network publication Guidance for Evaluating Humanitarian Assistance in Complex Emergencies (www.oecd.org/dataoecd/9/50/2667294.pdf) should be considered as a possible loose framework/model for developing CPPB guidance in this field. The consultants will, therefore, first undertake a detailed analytic review of the humanitarian guidance, as well as other guidance developed by DAC member agencies, and determine if any areas of this text or others could be transferable to future CPPB guidance. New areas and issues will also need to be added The Humanitarian Guidance should be used only as a basic model on which to build rather than as a fixed formula or constraining framework.

The following sections provide a working outline of possible questions to be addressed in the approach paper for CPPB guidance. In addition to the outline and issues raised below, it will also be important to look at related cross-cutting questions such as: i) the need for assessing gender sensitivity of CPPB programmes; ii) the need to assess whether a rights perspective has been mainstreamed/included in the CPPB programmes; iii) a reference to violence in evaluations, and how the programme evaluated contributed to reduce violence/violent conflict (referred to in section 4 below); and iv) some consideration of if/how HIV and AIDS prevention is covered in CPPB evaluations, as this is becoming an increasing problem in conflict-affected

1 The Utstein study was a review of the peace building experience of Germany, Netherlands, Norway and the UK.

countries and regions, especially in Africa. Please note that this outline should be seen as a working guide not as a straightjacket. The consultants are encouraged to take the scope and flexibility necessary for creative thinking in developing thematic areas and ideas in the approach paper and for proposing content for future guidance. 1) An Overview of CPPB Activities and Current Evaluation Practices: the Value Added of Guidance An introduction could provide a broad but concise overview of the conflict prevention and peace building environment, using available policy documents. The introduction should cover questions of why guidance is needed in this area and outline the key audiences for guidance and their evaluation needs.

Questions to be addressed may include: What, in brief, is the operational meaning of Conflict Prevention and Peace Building activities? What are the characteristics of modern-day conflict environments? What kind of conflict prevention and peace building activities are most commonly undertaken and by which agencies? What do we mean by evaluation of these activities? What is the need for and purpose of undertaking evaluations of conflict prevention and peace building activities? What is the present situation with regards frequency, rigour and robustness of conflict prevention and peace building evaluations (Note: See work already been done on lessons learned on evaluating conflict and building peace for these DAC Networks.)? What is the purpose and value added of the new guidance? What would be a suitable methodology and approach for developing CPPB guidance?

2) Analysing and Developing Guidance for CPPB Activities: Differences from Traditional Evaluations

The approach paper should look at how guidance might outline the main generic differences involved in evaluating on the one hand CPPB activities and on the other hand both ‘conventional’ development programmes and humanitarian aid interventions. Differences to be explored in this scene-setting overview could include, but not be limited to, the following:

The particular difficulties involved in defining CPPB - for example it can be difficult to determine when conflict ends or peace begins. The consultants are not required to devise methodologies to answer such questions but rather to acknowledge that in any given context debates may exist about whether the environment is one of ‘peace’ or ‘conflict’. It will be helpful to look at how agencies shape their programmes differently according to the context in a conflict-affected or prone country.

The particular importance of the political nature of conflict prevention and peace building and of its ideological context or country-specific situations as they relate to the donor community, government agencies, NGOs, national and regional actors, etc.

The essence of not working 'around' but 'in' and 'on' conflict situations should be considered. The latter is the main approach of CPPB activities and, therefore, the study shall address approaches taken by different donors in terms of being proactive and working 'on' conflict.

The ‘high stakes’ nature of CPPB activities, e.g. misconstrued projects/programmes can potentially lead to an increased risk of conflict. The consequent value of a ‘Do No Harm’ approach and conflict sensitivity by working 'in' and not 'around' conflict situations.

The importance of policy coherence – whole-of-government approaches to issues involving diplomacy, defence, trade policy, etc. at the donor and partner country levels – due to the interconnected nature of causational factors.

The cross-border, regional, sub-regional and international context of many conflicts. The question of applying the principles of the Paris Declaration in a fragile state or in a

context where key stakeholders may be protagonists of the conflict. How do principles of

ownership and alignment apply in this context and what are the challenges? The problem of vested interests (military, political, other) who may lose out from the

success of a project/programme. The need for rigorous stakeholder analysis? The particular importance of looking at the effect of an individual project/programme on

the wider ‘peace writ large’ within any given context, e.g. the difficulties of demonstrating the links between micro projects and macro 'peace writ large', noted below as well.

The particular importance of UN and NGO actors in conflict environments. The data deficit and particular difficulties in establishing good baseline information. With

the possible absence of ‘hard data’ in conflict situations, what other methods can be used to qualify/quantify other than data sources?

Lessons learned should also be reflected in the analysis, and DAC guidance and Issues Briefs could be helpful in that endeavour, see www.oecd.org/dac/conflict .

Participatory approaches to evaluation of CPPB activities and the perceptions of stakeholders, in particular the poor, are also critical in these contexts.

3) Options for Guidance on Programme Design and Management Based on the above overview of the context, the approach paper should consider recommended steps for CPPB practitioners to build steps into project/programme design and management which would facilitate their evaluation. Issues to be covered would include, but not be limited to the importance of:

Articulating and clarifying objectives at the planning stage; Working according to the principles of results based management; Developing robust baselines and monitoring systems; and Carefully considering the different needs of the intended end-users/audiences.

4) Recommendations for Practical Guidance on Evaluation

This section of the approach paper should consider some of the core questions related to providing guidance for undertaking evaluations of conflict prevention and peace building activities and underline the key point that evaluations in this field should be designed in a conflict-sensitive manner, with a special emphasis on the obstacles and possible incentives to achieving this. In addition to areas covered in the humanitarian guidance, additional issues could include:

What kinds of evaluation and research methods are generally more suitable in conflict environments – e.g. qualitative versus quantitative or mixed methods approaches? How can the design of an evaluation methodology be informed by the political economy and/or by conflict analysis tools? What are the advantages or disadvantages of using theory-based evaluation in evaluating CPPB activities?

What is the particular value of policy evaluations in conflict environments? What are the difficulties of demonstrating the links between micro projects and macro ‘peace writ large’? How could we look at evaluating all CPPB activities within one conflict or national context?

What is the role of conflict analysis when evaluating the relevance of CPPB activities? How to make use of other conflict analysis tools and Post-conflict Impact Assessments (PCIA)?

The importance of triangulation in a context where stakeholders may have firm ideological standpoints and/or whose perceptions may be affected by trauma or perceived danger.

The importance of protecting the security and safety of the evaluation team, while addressing the need for evaluations to take account of work done in potentially more

insecure rural areas. The ethical responsibility to protect the security and anonymity of evaluation informants. How can evaluators make the links between the micro- and macro levels, and to what

extent can any existing national policy frameworks such as Poverty Reduction Strategies (PRS) be used to frame the approach?

Are joint evaluations more suitable in a context where multiple programmes are affecting the same ‘peace writ large’?

How can steps be taken towards addressing the ‘strategic deficit’ in CPPB evaluations, as identified by the Utstein report - how can evaluations assess the relevance of donor project/programmes to country-specific and broader policies?

How should the timing of evaluation be planned in a conflict prevention or peace building environment?

How can a lack of baseline information be addressed by the evaluators? How applicable are the five standard DAC criteria to a conflict environment? Do they

need modifying or adapting to the conflict context and are any additional criteria needed?

5) Annex: Categorising CPPB Activities for Evaluation Guidance

The approach paper for developing guidance on undertaking evaluations in conflict prone and/or conflict affected settings should be supplemented by a short discussion piece (5-10 pages) which categorises the domain of Conflict Prevention and Peace building (CPPB) for evaluation purposes and serves as an annex to the main approach paper.

It is recognised that conflict prevention and peace building encompasses a number of distinct policy areas (for example, from security and development, to governance, to state building, to peacekeeping operations and support for truth and reconciliation), together with a wide variety of activities. Given such a broad domain, it is anticipated that - while guidance could provide generic advice cross the conflict contexts - more specific advice/direction will also be required within each CPPB category. The purpose of this annex is to discuss a breakdown of the CPPB domain and to differentiate each category from the point of view of how evaluations could be undertaken. The annex should also provide preliminary proposals on specific key issues/questions that would be pertinent for an evaluation team for each category within the CPPB domain of activities.

There have been a number of categorisations of CPPB, including the ‘palette’ concept contained in the Utstein study. However, none have attempted to categorise CPPB for evaluation purposes. Issues that may be pertinent to defining the categories relate to the type of actors involved, level of local ownership, political/conflict environmental factors and the scope of the activities being undertaken (whether focused on regional, national local actors/institutions). Developing a clear picture of CPPB activities as practiced by most agencies will be informed by the literature review and interview process.

Overview of Activities 1) Develop an inception report that would provide: a presentation and justification of the methodology chosen to approach the task; the feasibility of the terms of Reference - its questions, scope, time, budget, and foreseen methodology; and deviations from ToR with justifications for these.

2) To underpin the work, undertake a focussed literature review of key DAC and donor agency policies/guidance on evaluation on CPPB, with particular attention to the useful information to be drawn from the DAC guidance on Humanitarian Assistance.

3) Conduct in-depth interviews with a select number of key informants (15-20) by telephone, including members of the Networks on Evaluation and Conflict, Peace and Development Co-operation. The list of informants will be agreed with the DAC Secretariat.

4) Review and map the breadth of activities conducted in the field of CPPB to be included in the Annex with a view to categorising the various areas for the purpose of future evaluation.

5) Conduct analytical work and conceptual thinking to develop and write an approach paper covering preliminary recommendations for guidance for evaluation of CPPB activities and policies.

6) Present and discuss the findings of the draft approach paper at a workshop in Oslo planned for the 3rd quarter of 2006. The consultants should also prepare break-out group discussion topics based on questions raised in the approach paper and provide inputs to the agenda.

7) Finalise and submit a final approach paper that is of publishable quality in view of the discussion and feedback at the workshop.

Budget, Inputs and Competencies

The consultants should develop a detailed budget proposal and suggest in detail the number of necessary person days in order to complete the tasks, also indicating any other individuals who may support this work (with relevant CVs) if working in a team. At least eight to ten weeks of full-time work is envisaged, followed by intermittent work through end-2006. The detailed budget proposal, and information on the full team, if relevant, should be included in an annex to the inception report.

The consultant(s) will need strong experience of donor practices in development cooperation and a background in: (1) Conflict prevention / Peace Building / Security Programming; (2) Conflict assessment/Conflict Analysis; (3) Evaluation methodology and implementation; and (4) Communications and writing skills. Timing and Outputs The consultancy is expected to begin in May 2006. The consultant(s) will first be asked to provide an inception report (8-10 pages) for the approach paper within three to four weeks of the signature of the contract. The inception report will then be reviewed and considered for approval by the Steering Group. Once agreement has been reach on the inception report, the consultant(s) will be asked to begin work on the approach paper. A first draft of the approach paper (approximately 30 pages + 5-10 page annex) should be provided for preliminary review and feedback by the DAC Secretariat within six to eight weeks of approval of the inception report. As noted, under "Activities", the consultants will be asked to undertake a literature review to explore the options for guidance in this area, as well as interviews with a select number of key informants and the other activities described above for the development of the approach paper.

The full draft approach paper (approximately 30 pages + 5-10 page annex) is due for submission within two to four weeks after the approval of the draft approach paper. The draft approach paper, which should provide information on the scope, audiences and preliminary findings for guidance in this field, will be presented and reviewed at a workshop envisaged for the 3rd quarter of 2006. Therefore, a draft list of topics for break-out group discussions at the workshop, based on questions raised in the approach paper, should also be provided with the full draft approach paper. This work may involve some wider input to the workshop agenda.

A final revised version of the approach paper - that is re-worked in view of the outcomes of the workshop and of publishable quality - should be submitted within two to three weeks of the completion of the workshop.

Conduct of the Work The consultants shall report to the OECD Secretariat (CPDC and the Evaluation Network), and the point of contact will be Lisa Williams (lisa.wiliams@oecd.org). Drafts of the approach paper will be shared with the members of the two Networks for direction, feedback and input.