Prepared by Iigo Retolaza and Elena Dez Pinto Democratic Dialogue Project
EVALUATING MULTI-STAKEHOLDER DIALOGUES: A COMPARATIVE ANALYSIS
A background paper prepared for the
Generative Reflection Workshop: Assessing the Impact of Democratic Dialogues
Carter Center, Atlanta, January 24-25 2007
May God keep us from single vision
and Newtons sleep
1.- PAPER BACKGROUND
The Carter Center and the United Nations Development Program convened a workshop to reflect and learn about evaluation and assessment of dialogue processes, by exploring the subject from diverse approaches and practical experiences in Latin America, the Caribbean and other regions. The workshop intends to generate relevant knowledge on this field in order to provide practitioners, donors, and social and political leaders with elements to analyze and improve the depth, quality and impact of dialogue interventions. In particular, to discuss a set of useful indicators to measure results and impact from dialogue processes and lay the foundation for developing an effective impact assessment methodology. This paper constitutes an input for this workshop.
2.- AIM AND STRUCTURE OF THE DOCUMENT
This paper aims to serve as an input to stimulate further reflection on the field of evaluation by making a comparative analysis of three dialogue evaluations: Jamaican Civic Dialogue, Destino Colombia and the Argentinian Dialogue. The approach and methods used by each of the evaluation teams will be analysed drawing attention to: i) the theoretical approach applied, ii) the actors responsible for the evaluation, iii) the production of knowledge and learning, iv) the tools used, v) the outcomes achieved, vi) the societal learning and change process, vii) role of donors and lenders.
The final section synthesises some of the main findings and invites for further reflection regarding: i) monitoring and evaluation mechanisms for effective impact and assessment, ii) democratic dialogue as a societal learning and change process, iii) managing complexity in multi-stakeholder dialogue processes, iv) democratic governance and multi-stakeholder public dialogue, and v) long-term donor support.
3.- BRIEF INTRODUCTION TO THE COUNTRY CASES
The three cases have a basic common ground: all of them were supported in one way or another by an international organization. Nevertheless, the context in which these experiences occurred was clearly different. The Colombian dialogue took place in the midst of a longstanding armed conflict1 while in Jamaica the country was facing low economic output, high levels of crime, violence and social alienation and staggeringly high debt to GDP ratiosi. In contrast, the Argentinean dialogue was launched as a consequence of a deep institutional crisis in which democratic institutions which were unable to tackle ever increasing and deep structural socio-economic degradation.2 Table 1 summarizes basic aspects of these dialogue processes.
1 Diez Pinto E., De Len A., 2000, Destino Colombia: 1997-2000. A treasure to be revealed, UNDP: New York, 2000.
2 UNDP, 2004, Evaluacin del Dilogo Argentino, UNDP: not published
Table I: Basic Aspects of Dialogue Processes
Destino Colombia Jamaican Civic Dialogue Argentinian Dialogue
Purpose3 Shared vision; dealing with conflict
Shared vision; capacity building; personal development/leadership
Shared vision; strategy/action planning; decision making; consensus building
Time span4 and dialogue phases
March 1996-July 2000 3 phases: 1st phase: assembling the promoter and dialogue groups, dialogue design and funding, sharing international experiences 2nd phase: scenario design 3rd phase: scenario national dissemination
July 2002-January 2004 4 phases: 1st phase: pre-project investigation and design 2nd phase: scenario design 3rd phase: initial engagement phase 4rd phase: deep engagement
January 2002-October 2004 3 phases: 1st phase: launching of dialogue and setting the agenda among key actors 2nd phase: enhanced dialogue table, support to the Presidential electoral process 3rd phase: sectoral tables
Conveners Five recognized national leaders UNDP Government with Catholic
church and UNDP support
Businessmen, guerrilla, paramilitary, academics, social activists, NGO leaders, journalists, union leaders, senators, military,
Military, police, government, youth, church, social leaders, businessmen, judiciary
Business associations, NGOs, government, political parties, grassroots organizations, bank associations, university, religious organizations
Methodology Civic scenario Civic scenario Sectoral Roundtables
The three dialogues basically pursued building a shared vision of a common problem and all were multi-stakeholder, that is, assembled a microcosm of the country society. The Jamaica and Argentina dialogues lasted two years while Colombia lasted four. The dialogues of Colombia and Jamaica used civic scenario methodology. Participants of these dialogues were influential leaders and individuals but did not come to the dialogue as formal representatives of their organizations. In Argentina, a sectoral roundtable methodology was used. Sectoral action plans were defined through a series of round tables in order to establish and implement new public policies.
4. - COMPARATIVE ANALYSIS OF DIALOGUE EVALUATIONS
In this section the approach and methods used by each of the evaluation teams will be analysed and compared, drawing attention to: i) the theoretical approach applied, ii) the actors responsible for the evaluation, iii) the production of knowledge and learning, iv) the tools used, v) the outcomes achieved, vi) the societal learning and change process, vii) role of donors and lenders.
In reading this analysis two observations must be kept in mind. First, evaluations occurred at different time frames. The Jamaican and Colombian exercises were undertaken long after the respective dialogues were over (1-2 years later), while the Argentinean assessment started at the concluding stages of the third and final dialogue phases (Sectoral Round Tables). This difference in time frames, in turn, accounts to different outputs and outcomes of the dialogue processes.
Second, none of the three dialogues had any sort of pre-established monitoring system at place. No indicators were defined before the dialogue started and no formal entity was in charge of implementing a learning-oriented monitoring system.
3 Characterization of purpose after Pioneers of Change, 2006, Mapping dialogue. A research project profiling dialogue tools and
processes for social change, GTZ: South Africa 4 The time spam includes pre-dialogue, dialogue and dissemination phases.
4.1. Evaluation theoretical approach
The three evaluating exercises had some basic common ground as well as some peculiarities regarding the theoretical approached used. All three cases were i) critical and reflective-learning oriented, ii) context-based, iii) relational and inclusive, in terms of how knowledge and learning were produced, iv) dialectic, v) qualitative, and vi) propositional in their final recommendations.
The theoretical approach used in the Jamaican case was based on Sustained Dialogues theory of change and on the World Banks Social Capital Implementation Framework. The Argentinian case was grounded on evaluation as an integral and continuous process, a constructivist approach, and on participatory and dynamic approaches. The Colombian case was based on the transformative potential of dialogue and orders change, and the use of the learning history methodology. These approaches are summarized in Table II.
Table II: Theoretical approaches used in evaluating dialogues
COUNTRY CASE THEORETICAL APPROACH
Jamaica Sustained Dialogues Theory of Change World Banks Social Capital Implementation Framework
Argentina Evaluation as an integral and continuous process Constructivist approach Participatory and dynamic
Colombia Transformative potential of dialogue Third order change5 Learning history
4.2. Who evaluated the process and why?
In every evaluation, a team was assembled and lead by an international expert accompanied, by national colleagues, and hired by donors (UNDP and others to a lesser degree). This was done to assure a high quality of the evaluation process and final product, and objectivity and balance. In particular, national consultants were involved for having a deeper and broader understanding of the local historical, social, economic and political context where the dialogue took place, for knowing the relations between actors and for having access to local leaders.
4.3. How was learning and knowledge produced?
Although the three cases followed a relational and participatory approach to evaluation, participants interviewed during the evaluation were approached mainly as information givers and as process and content analysts. Interviewees opinions were somehow framed by indicators and criteria determined beforehand by the evaluation team. The Argentinean case used a more interactive approach as it engaged participants at an early stage of the evaluation. A group of participants were brought together to help the evaluation team define areas and indicators to be assessed.
An epistemological contradiction may be found here as the three evaluation teams acknowledged learning and knowledge as a negotiated process among significant actors, and yet, the teams initially set the ground rules for further generation of learning and knowledge. By doing so, the evaluators
5 Pruitt and Waddel