User Guide

Measuring Achievements of Partnership-based ProgramPrepared by:IKAT US Component 2 Team

Table of Contents

Some Basic Things 4

References 72

Using Experimental Method 66

Quasi-experimental Method 69

Other Evaluation Approach? 71

Re-thinking Our M&E Practice 5 A. Changing the M&E Approach and Mindset 6 B. Donor: Let’s Get Better Together 9 Strengthening M&E:Insights for Donor 11 C. Partner Organization: Building the Learning Culture! 17 Strengthening M&E: 13 Insights for Organization 20 D. M&E for Partnership Program 27

M&E: Weaving Learning Into Change 33 A. Essentials of Monitoring, Evaluation & Learning 34 B. M&E: Some Tools, Methods & Approaches 35 Most Signi�cant Change 36 Outcome Mapping 39 Theory of Change 44 C. Hybrid Approaches 49

Measuring the “Immeasurable” 53 A. Utilizing Good Governance Indicators 54 B. Tackling the Evaluation Challenge 60 C. Methodological Challenges in Evaluating Governance Intervention 63 D. Rigorous Evaluation of Governance Intervention: Is it Possible? 66

Chapter 2

Chapter 3

Chapter 1

List of Picture

Picture 1. Change in the Approach & Practice of Monitoring and Evaluation

List of Box

Box 1. IKAT US Program Approach in Linking Partners & Manage KnowledgeBox 2. Evaluation Approach in Oxfam GBBox 3. AWID’s Strategic InitiativeBox 4. Experiences of IKAT US’s Lead Organizations in Initiating PartnershipBox 5. POWER: Showing Leadership & High Quality Training based on NeedsBox 6. Illustrative Use of MSC for ASIA CALLINGBox 7. Outcome Mapping in ACCESS Phase IIBox 8. Illustrative Use of ToC in IKAT US Partnership ProgramBox 9. Experience of HIVOS: Building Southern Organizational Capacity Through Theory of ChangeBox 10. Women’s Learning Partnership (WLP) M&E FrameworkBox 11. AWID (Association for Women’s Rights in Development) ME FrameworkBox 12. Olken’s Study on Corruption in Road Infrastructure Projects in IndonesiaBox 13. The Evaluation of Impact of international Election Observers on Election Quality in IndonesiaBox 14. The e�ect of information on the performance of local government o�cials on people’s participation during decentralization in the Philippines

List of Tables

Table 1. Users and Uses of Governance IndicatorsTable 2. List of Governance Indicators and Online ResourcesTable 3. Four Quasi-experimental Methods

3

Some Basic Things…About this Guide

This Guide will not talk much about de�nition of monitoring & evaluation nor types of monitoring of evaluation in great details. There are many good monitoring & evalua-tion guidelines out there that serve this purpose. The goal of this user guide is to extract monitoring and evaluation lessons learned from the common practice and share it to all of you as a part of learning process.

Nevertheless, it is important to lay down some basic principles before we move forward in this document:

How do we de�ne Monitoring & Evaluation?

Although the term “monitoring and evaluation” tends to be run together as if it is only one integrated process, monitoring and evaluation are, in fact, two distinct sets of organizational activities, related but not identical.

4

Source: Designing for Results: Integrating ME in Con�ict Transformation Program, Search for Common Ground, p.96

What is it? Ongoing collection and analysis of data on progress toward results, changes in the context, strategies and implementation

Reviewing what has happened and why, and determining relevance, e�ciency, e�ectiveness, impact, etc.

Inform day-to-day decision-making

Accountability and reporting

Strengthen future programming

Provide evidence of success

Deepen our understanding of how and why things work

Program Sta� and/or Partners and/or Participants External consultant, sta�, participants or combination of these groups

At design stage Core decisions taken at design stage and re�ned prior to implementation

Why do it?

Who does it?

When to plan?

Throughout the program – periodically, frequently or continuously

Mid-term (formative)Completion (summative)After completion (impact)

When to implement?

Level Monitoring Evaluation

Re-thinking Our M&E Practice

Chapter 1

A. Changing the M&E Approach and Mindset

Limited time

Lack of technical capacity

“Getting something wrong is not a crime. Failing to learn from past mistakes because you are not monitoring and evaluating is.”

Source: www.civicus.org/new/media/Monitoring%20and%20Evaluation.pdf

No one will argue about the importance of monitoring and evaluation. However, many get to really feel and experience the true bene�t of doing monitoring & evalua-tion.

So the question remains: why Monitoring & Evaluation fail to deliver its bene�t for development purposes?

There can be several reasons forwarded in answering this question:

These reasons are basically true, but there is actually other fundamental cause if we trace further. It goes beyond the internal issue of one’s organization such as limited capacity and limited resources. And it has to do with the way the overall development sector - aid agencies and non-pro�t organizations - view M&E itself:

• M&E is perceived as a mean to justify the development aid expenditures and thus, focus limitedly on logical framework activities and �nancial reporting, such as financial audits, field visits, mid-term review and end of project evaluation.

• M&E is treated as an “add-on” to the project implementation, a mechanical operation separate from daily program operations and done separately by an M&E O�cer or M&E Unit. Investment to M&E is less prioritized and given by very limited M&E activities.

6

Limited resources

M&E is less-priority

• M&E is also perceived as a measurement of performance and success. Good news is preferred over the bad ones, leading to a level of M&E that gives minimum assurance of effectiveness without raising difficult questions.

• We want to achieve outcome but we want a short cut. M&E is dominated by questions of measuring outcomes because we want to achieve the outcome level. However little investment is made in developing a proper project/program interven-tion (design) with proper project period that enables the achievement of outcome. As a result, there is often a weak link between the project’s output and outcome. Further-more, little investment is made also to encourage and facilitate a robust evaluation method to really produce lessons learned.

7

Added with the fact that organization has limited resources and capacity to carry out M&E, these have led to bad practices of M&E. This phenomena might satisfy the requirement of donors, but have little or no real benefit for the organization, intended beneficiaries and even the donors themselves because only little learning can be captured from it. A change of approach and mindset is needed not only within the internal organization but also the donors so that M&E could be useful!

8

Source: Designing for Results: Integrating ME in Con�ict Transformation Program, Search for Common Ground, p.96

Picture 1. Change in the Approach & Practice of Monitoring and Evaluation

B. Donors: Let’s Get Better Together!

9

The key principle is to Get Better Together. Donors/aid agencies have to acknowl-edge that most partner organizations have limited resources & capacity to achieve

unrealistic targets. Thus, more active role of donors/ aid agencies is imperative. Commitment and support, not only in terms of funds, are needed. Pursuing real

partnership and linking partners to knowledge and other resources are some inno-vative ways to move forward.

One of the biggest issues the donors/aid agencies here and often becomes a chal-lenge is in managing relations between donors/aid agencies and its partner organizations (grantees). Even though these organizations are called as ‘partners’ but in reality, they are still trapped in the donor-grantee relations - where donors avoid to invest in strengthening grantee’s or partner’s management and infrastruc-ture whereas on the other hand partners are afraid to speak the truth because it might risk their funding. They are trapped by a mere compliance to donors’ required reports in which substantially don’t lead to performance improvement Without a well planned M&E framework and execution, reports are least useful other than just to ful�ll the formal accountability requirement. Partners then spend time and resources on measurement that doesn’t do any good, instead of measurement that fosters learning.

Establishing international standards for methodological rigor, ethical practice and e�cient management processes in M&E is another critical challenge. Key issues include how donors/aid agencies should oversee evaluations being outsourced to consultants, how to coordinate evaluations of joint donor program e�ectively and further on, how to feed the result of evaluations to the development com-munities, particularly their CSOs/NGOs partners.

This means – that merely designing an M&E system is not enough: donors need to commit to supporting M&E at all stages of implementation, which include in the plan-ning to the implementation and post implementation phase: selecting appropriate indicators, establishing baselines, collecting quality data and reporting and using �ndings e�ectively. Badly designed and managed evaluations can do more harm than good: e.g. misleading results can undermine the e�ective channeling of resources for poverty reduction. Therefore, learning – feeding the results of evaluation and link-ing knowledge resources that the donors/aid agencies have is also crucial.

10

Box 1. IKAT US Program Approach in Linking Partners & Manage KnowledgE

IKAT US 1 partnership network was established in 2010 to promote partnerships among Indonesian, U.S. and Southeast Asian CSOs to expand democracy experiences and expertise in the region. The three-year program aims to increase the capability of Indonesian CSOs to cascade their expertise and experiences inside, outside Indonesia and across South East Asian (SEA) countries.

The medium term goal of the program is to improve democratic development, gov-ernance, and respect for human rights in the Southeast Asia region, while the long-term goal of the program is to sustain South-South partnerships between Indonesian CSOs and their counterparts throughout the region to advance human rights nd democracy.

Currently, IKAT US Component 1 partnership consists of 50 organizations from 7 SEA countries (Annex 1). Mainly all six partnerships identify themselves as ABA-ROLI, RWI, PPMN, POWER, R4D and AGENDA. All lead partners are based in Indonesia and formed a local partnership with local organizations but at the same time, through IKAT US program, each partnership expands their partnership to Timor Leste, Malaysia, Cam-bodia, The Philippines, Laos, Burma and Thailand

USAID then initiated the IKAT-US Component 2, aiming to provide support to the members of the IKAT-US Component 1 partnerships, in order to enhance their capac-ity to implement regional programs. Component 2 links the partnerships and draws lessons learned from the partnership program implementation to facilitate learning & manage knowledge among partners. It hosts annual conferences that bring together the Indonesian, US and Southeast Asian partner CSOs from all partnerships to share experiences and lessons learned as well as coordinate their program activities and assist the US Government with evaluating the IKAT-US initiative as a whole and seek ways to strengthen these regional partnerships.

In the next page, we will take a further look at 7 Insights for Donors adapted from intensive research and analysis by AWID (Association for Women’s Rights in Develop-ment). Between 2009 and mid-2010, AWID has been engaged in an intensive research into the challenges of monitoring & evaluating the progress of women’s rights work. The consultant team adapted and synthesized AWID principles according to the lesson learnt from other resources and �eld experiences.

11

1. Make M&E a learning partnership, not a performance test

2. Approaches that include multiple M&E frameworks/tools/methods are more e�ective

3. Balanced quantitative & qualitative assessment techniques

4. Seek contribution to impactful changes instead of achievement in numbers It is more important to seek contribution to impactful changes rather than claiming big achievements by using statistical numbers.

5. Make M&E systems �exible and adaptable

6. Invest resources in developing M&E capacity

7. Invest in the creation of baselines

Strengthening M&E: Seven Insights for Donors 1

1: Batlilawa, Srilatha, “Strengthening Monitoring & Evaluation for Women’s Rights: Twelve Insights for Donors” for

1. Make M&E a learning partnership, not a performance test.

3. Balance of quantitative and qualitative assessment techniques.

2. Approaches that include multiple M&E frameworks/tools methods are more e�ective.

12

The first and most important overarching lesson from AWID’s research and conversa-tions with both donors and women’s organizations, and other recent work, is that assessment is best achieved when it is approached as a learning avenue for both donors and grantees. We need to change the paradigm from “proving that you did what you were supposed to” to a more collaborative paradigm “let’s learn together on how to alter the deeply embedded causes of these issues/problems”.

Of course, donors themselves are under pressure to show results and thus, are forced to adopt stringent, rigid or overly quantitative M&E. But if we do not change the current practice and adopt a more collaborative approach to assess the results and impacts, engage grantee partners in M&E design and/or subsidizing the capacity building of the grantee partners, then the work is less likely to be sustainable due to unrealistic expectations and lesser impact since the program has been “projectized”.

In the spirit of learning, we also have to not only focus on our performance or the positive impact of our intervention, but also to the negative impacts. Current assess-ment tools do not capture many negative. Designing instruments that enable to pick up these negative e�ects are vital, since they may radically alter the assessment of a project’s “success” or “failure”, by placing achievements against a more realistic yard-stick.

No single M&E framework can capture all aspects of the change, impact or results of a women’s rights/ empowerment of project or strategy – in short, one size does not �t all! Expected changes in democracy & governance as well as policies are in�uenced by complex power relations, embedded in multiple social, cultural, economic and political structures and institutions. Thus, a single M&E instrument – such as logical framework, theory of change, outcome mapping – can assess the some dimensions of the change process and indicate impacts, but not all. So while one instrument cannot tell us the whole story, strategic and intelligent combinations can bring us much closer to a more comprehensive understanding of the change process, its gains and limitations.

There is a widespread belief that donors prefer quantitative – or “hard” – evidence of results, rather than “soft” data that tends to appear “anecdotal” and hence not rigor-ous. In reality, though, the most complete picture of positive change – or of reversals – emerges when quantitative and qualitative tools of assessment are combined.

13

Qualitative method could produce explanation on the contributing factors behind the targeted numbers gained with quantitative method. An overemphasis on quantitative data actually undermines our ability to understand how changes happen, and there-fore, of how to make it happen more e�ectively.

Indicators that spells numbers such as 100 women received training on improvement livelihoods, 200 sex workers increased knowledge on HIV/AIDS, and so on. This approach could be more complete if there are tracking system which will produce sample of stories from the bene�ciaries on enabling or supporting factors that could make the training results more sustain and how they use the knowledge in their daily practice.

Example: IKAT US-Kemitraan Partnership Framework Vs M&E FrameworkIKAT US-Kemitraan program basically use two frameworks to see the progress on progress of each partnership with the assumption that “better quality of partnership will lead to better results”. M&E Framework to measure achievements and progress for each partnership and Partnership Framework to assess depth and quality of the part-nership. The Partnership Framework contains three main phases of the partnership development from pre-partnership, development and Partnership phase. All indica-tors used are more on qualitative aspects and use survey method using sample of all the partners. With this combined approach, deep and sharp lesson learned reports could be produced and next round of programming could get more feedbacks on the achievements as well as the pitfalls in designing the next intervention. (Source: www.kemitraan/or/id/krc)

In order to balance quantitative and qualitative evidence, it is important to give due weight to participatory tools and methods – such as bene�ciaries’ narratives of change – rather than privileging so-called “objective” evidence alone. This is because no one can as accurately assess change in their situation as the bene�ciaries themselves as the target of our intervention.

Far from being merely anecdotal, these narratives and other participatory tools are often the most sensitive indicators of project impact. While it is true that these meth-ods can be manipulated to present a rosy picture of achievement, the solution is not to dismiss them. In fact, even “hard” data can be manipulated to hide less �attering truths by highlighting some data and burying others – as in the case of South Asian NGO who highlighted survey �ndings that their micro-credit program had increased women’s income by over 30% - but hid the fact that school drop-out rate among these women’s daughters was equally high!

4. Seek contribution to impactful changes instead of achievement in numbers

14

It is vital for both donors and organizations to recognize that macro-level changes – such as more e�ective public spending and service delivery – occur over longer peri-ods of time, when a number of stakeholders buys in to the change, and when a variety of cultural, political and economic force coalesce. No single project can work on all these fronts, much less in a three or �ve-year time frame. What we can measure, how-ever are interim changes within speci�c stakeholder groups. It is therefore important for donor and grantee representatives to discuss, at the outset of a project, what can be most realistically measured and reported within the time frame of the project. This preempts grantees from making exaggerated claims and donors from feeling disap-pointed with the results.

Rigid approaches that insist on the initially set out targets / indicators / outputs / outcomes are not always useful for multi-dimensional interventions - they reflect the limited understanding of those imposing them, rather than the contextual realities in which such interventions are actually situated. But this is not a peculiarity of social change work – even major corporations and businesses have been forced to revise their targets and indicators when macro- economic and market realities change!

A simple example is that of quantitative targets that might have been set at the begin-ning of the project, but the process of implementation shows that these were over-estimated – or under-estimated. The M&E indicators may quickly have to be revised, under these circumstances, to reflect the level of response a project was able to do. A timely re-negotiation also ensures that indicators are not changed arbitrarily because the earlier ones reflected negatively on the organization’s work, but because external factors have forced changes in intervention’s trajectory.

The attribution-based approaches like the logical framework are perfectly acceptable when tracking performance or implementation of project activities, but do not work well when we are trying to understand how the change process produces intended – and unintended - results. They are also not designed to capture the interim goals that must be achieved in order to reach final intended outcomes.

Logical Framework results tend to jump right to measuring the overall goal of a program which are achievable only in the longer term, and consequently, in measuring macro-indicators (like poverty rates, literacy rates, mortality rates, or other indicators) that go beyond the scope of a single program’s sphere of influence or impact.

So in developing the right “mix” of M&E instruments, it is advisable to ensure the use of approaches which strive to track the organization’s contribution to these shifts, rather than those that seek to claim the entire credit for the change, which is hard, if not impossible, to prove.

5. Make M&E system �exible, adaptable and suitable to the nature of the issues

15

The point is that M&E systems must be agile and �exible, since evidence shows that even the most carefully-chosen approaches and measures may have to be changed during midstream if the ground-realities shift radically in the course of the project implementation.

The world of development’s interventions, including in democratic and governance projects, is populated by organizations with diverse and complex architectures – networks, local-to-national-to-regional-to-global structures, coalitions and other factors.

These entities often gain donor support because of their architecture, which gives them a greater reach, bandwidth, and impact at multiple levels of policy and activism. Yet, when it comes to monitoring their work, assessing their results, or evaluating their impact, they are compelled to use M&E frameworks and tools that were designed for far simpler, grassroots-based, direct-action or service-delivery organizations. These tools ask questions like “how many bene�ciaries have you reached / benefitted?” which are almost impossible for multi-layered structures to answer.

Anti Corruption, democratic, human rights, and more contextual based issues are more complex. For an instance, these type programs generally involve the secretariat of a large, geographically dispersed network or coalition, with multiple organizations as members, is typically the recipient or pass-through agency for funds. They cannot answer this question without rolling it through all the layers of its structure. And even when they provide the answer to the question, it tells us nothing about the value that has been added by supporting this type of structure, rather than a simpler one. But formations with multi-layered architectures are in fact aiming not only to scale up, but to deepen the impact of the work of their individual members by fostering their knowl-edge, capacity and strategies, and through value-adding approaches that a single organization may not have the ability, resources, or reach, to undertake.

Complex governance or democratic based programs require M&E approaches that combine the assessment of (i) their e�ectiveness as networks (such as the Wilson-Grau/Nunez) or partnership, (ii) their advocacy impact (through tools and recommen-ndations such as o�ered by Patton and Klugman), and (iii) a judicious combinationof the more conventional frameworks for assessing local- or member-level results.

If M&E is intended to increase operational e�ectiveness and contribute to learning, it cannot be treated as an afterthought. Yet sometimes, complex M&E requirements are attached to projects after they are approved, assuming this is something that every organization should be able to deliver. This fails to take into account the fact that many frameworks and methods require a certain level of skill or training – not to mention sta� time - for their e�ective use.

6. Invest resources in developing M&E capacity.

16

As mentioned, above, this is not always the case – but more problematically, grantees find that their M&E needs are not seen as part of the project cost, or built into their budgets. Donor expectations on how lean an organization should look and results or changes to be achieved within a short period of time is somewhat unrealistic. These organizations are on the front lines and doing the heavy lifting while they rarely have enough money for program, much less operations. So, success, if not survival, depends highly on fundraising e�orts. Thus, donors and aid agencies have to acknowledge that to achieve their expected results, they have to invest in building capacity of their grantee organizations to overcome these technical capacity issue. The time has come to integrate M&E into project design much more consciously, and to allocate serious resources for this purpose – this investment, after all, yields rich dividends in terms of both results and learning that can be leveraged for future work by both grantee and donor.

Many organizations find that the M&E frameworks and approaches required by some donors demand resources that are beyond their capacity, such as the abilities of their sta�, the time-intensiveness of their implementation, or the need to bring in external expertise that they cannot always access or a�ord due to resource constraints, loca-tion, or other factors. Often, the complexity and amount of data required is excessive, and does not necessarily provide a more comprehensive picture of implementation or impact – for e�ective M&E, sometimes less is more, if the approach includes fewer but more sensitive and intelligent indicators. In addition, in organizations with multi-layered structures, capacity to manage sophisticated M&E tools and data require-ements will vary across the network. All this implies a need for grant-seekers to assess their organizational M&E capacity in transparent and non-threatening ways, without feeling exposed or fearing loss of funding. Overall, balancing M&E needs with capac-ities require both donor and grantee to interrogate di�erent M&E frameworks and tools through the capacity lens during the grant negotiation process itself, identify fewer but more sensitive indicators of progress and change, and create approaches that are feasible for the grantee organization to use, given their particular capacity profile

It is in this context that generating periodic baseline becomes a powerful tool in accurately assessing project achievements – and directions for the next phase of work. When a clear situational analysis is generated at the outset of a project, organi-zations can be more accurately place the changes that have occurred in the course of their works – both positive and negative – against the baseline, identify what worked and what did not, and refine their strategies accordingly. They can make much stronger cases as well, for continued investment in their works. Donors can in turn leverage this kind of evidence to raise their case to their governments or other contributors, for why such work needs support. Together, donors and grantees can advocate more convincingly to the world at large. For all these reasons, baselines are highly important and worthy of investing time, money, and people resources.

7. Invest in the creation of baselines.

C. Partner Organization: Building the Learning Culture!

Organizations (grantees) confront the M&E challenges from a di�erent vantage point. They rarely have enough money for programs, much less operations, so priorities are always given to somewhere else. In many ways, M&E has become about compliance and a means – if not a checklist – to satisfy the donor requirement in the most mini-mum level. Moreover, they are often undermanaged, in one part because unrestricted funding for organizational expertise is hard to come by and in other parts because of their own inclinations and abilities. Many are trapped in a kind of “starvation cycle,” which begins with unrealistic donor expectations about how lean the organization should look, and continue as nonpro�ts by cutting overhead to the bone to meet those expectations—gutting the organization, but further encouraging those unreal-istic expectations.

So from the grantee’s perspective, the donor-grantee trap can feel inevitable. Grantees don’t have the means to “resource it right” on their own. They may want to pursue genuine donor partnerships – but they are in a position where they have less bargaining power. As for getting better together, that also takes investment - not to mention discipline, rigor and a willingness to appear vulnerable by identify-ing performance shortfalls and strategic gaps. It seems far safer to comply with donors’ existing reporting requirements and declare victory whenever possible.

One of the most di�cult challenges is related to the resilience towards M&E. M&E is often received as a measurement of our performance. Therefore, resistance is common – it is not always easy to receive feedback. Engagement of M&E Unit and Project Management Unit (PMU) should be built and strengthened since the begin-ning. It is crucial, since the planning process that both M&E and PMU sit together to discuss about the project designs and ways to measure it, ensuring that both units’ sta� participated in the process. The focus of M&E should also be about solution and improvement instead of solely on who is not doing what.

Overall, from the organization side, strong leadership’s support and commitment are important to change the M&E practice internally. Leadership has to be able to encour-age culture of learning and determine su�cient investment to enable the learning culture to occur even under limitation of resources and time. Finding ways to optimize the utilization of internal resources and external resources through collaboration might be one way.

Here are three organizations, which don’t have the resources of the World Bank or Gates Foundation, yet each one has demonstrated the willingness to question their assumptions and try to meet the evaluation challenge in di�erent ways :

17

2

2: Savedo�, William, “Impact Evaluations Everywhere: What a Small NGO to do?”, for Center for Global Development, 17 Sept 2012, accessed at: http://international.cgdev.org/blog/impact-evaluations-everywhere-what%E2%80%99s-small-ngo-do

18

Box 2. Evaluation Approach in Oxfam GB

Oxfam GB is a relatively large NGO that has an evaluation approach that would be recognizable to a large development bank or government agency, yet on a scale that tries to balance precision with sta� time and costs. Oxfam GB randomly picks about 40 projects a year and tries to infer impact using di�erent methods. When projects involve a�ecting the lives of many people, they compare project bene�ciaries with a comparison group. When the projects are related to in�uencing policy or empower-ment, they use process-tracing. If projects are randomly chosen, it helps Oxfam GB avoid cherry-picking the most promising projects and therefore allows them to say something more general about the performance of their portfolio. Randomly choos-ing projects also increases the probability of discovering things that sta� might not have expected. The value and use of the studies will depend, as always, on their qual-ity – particularly the degree to which their conclusions are credible. Are the counter-factuals appropriate? Is the causal chain clearly described and can alternative expla-nations be discarded?

Source:Savedo�, William, “Impact Evaluations Everywhere: What a Small NGO to do?”

19

Box 3. AWID’s Strategic Initiative

The Association for Women's Rights in Development (AWID) is an international, femi-nist, membership organization committed to achieving gender equality, sustainable development and women's human rights. A dynamic network of women and men around the world, AWID members are researchers, academics, students, educators, activists, business people, policy-makers, development practitioners, funders, and more. AWID’s mission is to strengthen the voice, impact and in�uence of women’s rights advocates, organizations and movements internationally to e�ectively advance the rights of women.

AWID’s work is structured through multi-year programs known as Strategic Initiatives. Each strategic initiative includes a range of activities from membership consultations and surveys, primary research and dialogues with policy makers (including targeted advocacy) to capacity building institutes, regional networking and information dissemination. In addition, AWID works to ensure that the speci�c priorities and voices of young women are strongly represented in all their initiatives.

Recognizing that movements without resources cannot be sustained, this initiative undertakes research and advocacy in order to signi�cantly increase the amount and quality of funding to support women’s rights work. The initiative was primarily developed to tackle the urgent need of women’s organizations and movements to access more resources on better terms and to transform their relationship to funding – moving beyond a culture of “scarcity” to see funding and resource mobilization as a critical aspect of their political agendas and key for building strong feminist move-ments. At the same time, the initiative also works to improve the ability of women’s organizations to use these funds in ways that are strategic, bold and e�ective.

Source: www.awid.org

It is in this context that through 2009 and 2010, AWID initiated multiple action research projects to study the challenges faced by both women’s organizations and their donors in e�ectively monitoring and evaluating women’s rights work, and to enhance our collective capacity to assess the in�uence and impact of such work. An in-depth quantitative and qualitative study of the experiences and challenges faced by 37 out of the 45 organizations that received the Dutch Government’s MDG3 Fund grants in 2008. The study led to the following insights that are drawn from Learning More from the MDG3 Fund Experience (Batliwala and Pittman) . It is important to take a look at these insights since it can also be applied to the works that our organizations do!

3

3: Batlilawa, Srilatha, “Strengthening Monitoring & Evaluation for Women’s Rights: Thirteen Insights for Women’s Organizations” for AWID, 2011.

20

Strengthening M&E: Seven Insights for Organizations 4

4 : Slightly adapted from “Strengthening Monitoring & Evaluation for Women’s Rights: Thirteen Insights for Women’s Organizations” by Srilatha Batlilawa for AWID, 2011

1. Make M&E key ingredient in our learning & accountability (1&4)

2. One size does not �t all (3, 5)

3. Prioritize approaches that assess our contribution to impactful change instead of achievement in numbers (4 & 5)

4. Make M&E system �exible and adaptable (7,8,9)

5. Negotiate M&E systems & result indicators with donors (10 & 11)

6. Develop M&E capacity (2 & 13)

7. Create baselines

21

AWID’s research and conversations with both donors and women’s organizations reveal that M&E is most useful and relevant when it is approached as a learning process, rather than a reporting or fundraising requirement. Solid, comprehensive and rigorous assessment of our e�ectiveness is a critical expression of our account-ability to our constituency, and to our longer-term mission. We often engage in this learning process subconsciously or informally. In our internal meetings and planning processes, or in conversations with each other, for instance, we are constantly assess-ing the progress of our work. We frequently share valuable insights about what we are learning, about what we have achieved, the setbacks we have su�ered; we identify who and what is behind both the challenges to our work and the progress we have made; and we analyze why we think change has—or has not—happened. This is exactly what monitoring and evaluation means. The task is thus to transform this inter-nalized habit of analysis and learning into more systematic and articulated forms that can be shared with others—not just the donors who may require the information, but others who could learn from our experiences and insights. Even more, we have to apply this learning more consciously in reviewing our practices and strategies, and shaping new interventions. The challenge, therefore, is to make M&E a central part of the way we learn and strengthen our work at every level, a vibrant expression of organizational and individual learning and growth, and a critical contribution to the collective learning worldwide. “M&E should become part of our knowledge management system”.

To address this learning goal, we have to also acknowledge that M&E system should not only track positive impact as an indication of our performance. M&E systems that allow space for us to track and document negative changes, resistance and backlash are essential. For example, measuring the result of a�rmative action program for women in parliament in terms of increased number of women in parliament might only tell one part of the story. Whether having more women in parliament lead to better development policies or gender-sensitive policies is another issue or even, whether the a�rmative actions basically further the view that women can only sit in parliament due to the quota. Nonetheless, most assessment tools are not designed to track or capture these negative impacts. Designing instruments that pick up these negative e�ects and reactions are vital, since it can radically alter the assessment of a project’s “success” or “failure”, by placing our achievements in a more realistic context. In fact, in many cases, negative reactions or reversals are actually evidence of positive impact. Similarly, we need approaches that give due value to processes that success-fully hold on to past gains that “hold the line”, such as preventing the repeal of a law entitling women’s access to abortion, or protection from domestic violence. Holding the line, in this context, is a success story, not evidence of a failure to move forward. The sign of positive impact might actually simply be that “things haven’t gotten worse”.

1. Make M&E a key ingredient of our learning and accountability.

22

Therefore, the �nal and possibly most important principle emerging from our M&E system is that if we make the time, e�ort, and resources available to design and implement the best M&E system possible for tracking and assessing our work – make it a part of knowledge management, chances are that it will also serve the needs of other stakeholders to whom we may be accountable— our governments, donors or the public that we seek to serve. It can actually be useful to other stake-holders in helping them design a better program or intervention.

AWID’s research has shown clearly that no single M&E framework can capture all aspects of change, impact, or results of women’s rights / empowerment project or strategy. No single tool or method can respond to all of our learning needs, since each has been designed to track or capture speci�c dimensions of change or operational e�ectiveness but not others. In any M&E instrument—whether it is the logical frame-work, theory of change, outcome mapping, or gender impact analysis—it only assesses a particular set of dimensions, but not all. Consequently, a comprehensive assessment requires the application of multi frameworks, methods, and tools, working together in a complementary fashion. In fact, many women’s organizations are already doing this: over half (51%) of the women’s organizations in AWID’s recent study of MDG3 Fund grantees use more than four M&E approaches and tools, or elements from several, to document their progress and impact. This is quite logical given that the nature of gender and social power relations are complex and that organizations oper-ate in di�erent social, cultural, economic and political contexts. Our study also found that “organizations that used more than four M&E approaches, experienced a slightly higher level of satisfaction with their M&E system (and conversely, lower levels of dissatisfaction) than those that use one or two methods.” (Batliwala and Pittman, np).

Just as we need to consider combining multiple approaches, we also need to combine both quantitative and qualitative M&E tools and their results in an appropriate balance. The experience of most women’s rights activists and organizations is that quantitative, or “hard”, evidence of results is taken more seriously than “soft” data like stories of change, which are treated as anecdotal and lacking in rigor. There is also a tendency to believe that our work can only be assessed qualitatively, and that our processes of change are too complex or subtle to be measured in numbers. While it is true that no one can as accurately assess change as the communities bene�ciaries who are the subjects and agents of a change process, we should not see these as substi-tutes for, but complements to, harder assessment methods. In fact, an overemphasis on qualitative information often limits our ability to demonstrate that our work is making a di�erence, especially with audiences like government policy makers or the donor community.

2. One size does not �t all.

23

For example, we could combine surveys which generate quantitative data on changes in women’s political participation, mobility, income, awareness of rights, literacy, health-seeking behavior, and changes in male attitudes, with qualitative methods like narratives of individual and collective struggles, stories of change, and focus group discussions that describe how change happened. In fact, quantitative data often validates and provide nuances for qualitative evidence, and vice-versa. The quantitative-qualitative balance can be achieved by organizations regardless of their size, location, or context, especially when combined with the “less is more” approach (see point 7). Building our stories of change by combining quantitative data and quali-tative evidence can help us make our case far more e�ectively.

Another example is IKAT US-Kemitraan Partnership Framework Vs M&E Framework refer to the previous section on “Insights for Donors). IKAT US-Kemitraan program basi-cally use two frameworks to see the progress on progress of each partnership with the assumption that “better quality of partnership will lead to better results”.So, while an M&E instrument cannot tell us the whole story, strategic combinations can bring us much closer to a more comprehensive understanding of the change process, including its strengths and limitations.

We are often tempted to claim credit for all the changes that occur in a development process, or feel pressured to do so by the struggle to secure funding for our work. And sometimes we are reluctant or too modest to take credit for our contribution to change, fearing it will be seen as exaggerated or self-promoting. Some M&E frame-works - such as the logical framework or Results Based Management—are in fact designed to attribute results to our interventions in a simplistic way. But in reality, such approaches are more appropriate for tracking performance or implementation of project activities. They do not work as well when we are trying to understand how the change process produced results, both intended and unintended. What is more, such “attribution-seeking” approaches are not designed to capture the interim steps that must be achieved in order to reach the �nal intended outcomes. For example, a height-ened awareness of domestic violence as a crime is a necessary �rst step to reducing such violence. This results in jumping straight to measuring the overall goal of a program or intervention —which we know can only be achieved in the longer-term (e.g., reduction or elimination of domestic violence)—and in making exaggerated claims of attribution that can rarely be supported. Worse, both having held the line, or reversals and backlashes will also be placed at our door in a negative way. This is why contribution-based approaches – such as Outcome Mapping, Most Signi�cant Change or Theory of Change, - should be important components of our M&E systems, since they allow us to make more realistic, but modest, claims about our role in the change process.

3. Prioritize approaches that assess our contribution to impactful change instead of achievement in numbers

24

Rigid approaches to reporting on targets / indicators / outputs / outcomes that were planned at the start of a change intervention are often not useful for our line of work. When we plan an intervention and design the M&E system to monitor and assess its results, we are engaged in intelligent guesswork rather than infallible certainties. The broader context can a�ect planned interventions, which requires the organization/activists to adapt to the new circumstances. There is always an element of unpredictability in women’s rights work, so that the best laid plans can go wrong, often for reasons beyond our control. On the other hand, things can also go right, or proceed much faster, than we had anticipated. This lack of predictability is not a peculi-arity of social change work - even major corporations and businesses have been forced to revise their targets and indicators when macro-economic and market realities change! Under these circumstances, M&E indicators may quickly have to be revised, to re�ect what the organization was able to do in response. As such, M&E systems must be agile and �exible, since evidence shows that even the most carefully-chosen approaches and measures may have to be changed midstream if the ground-realities shift radically in the course of project implementation (Batliwala and Pittman, np). Of course this �exibility should not be misused —to hide, for instance, our own mistakes or strategic errors. It should be applied only when it is clear that the trajectory of our change intervention has been altered by external factors beyond our control or as a result of new information that suggests a change in course.

The world of development intervention, including in democratic and governance project, is populated by organizations with diverse and complex architectures – networks, local-to-national-to-regional-to-global structures, coalitions, etc. But most M&E frameworks and tools at our disposal were designed for far simpler, grassroots-based, direct-action or service-delivery organizations. These tools ask questions like “How many bene�ciaries have you reached? How many bene�ciaries have bene�tted?” which are often quite inappropriate for complex, multi-layered, multi-locational stru-ctures. Networks, coalitions, and transnational organizations have created complex structures that enable them to work in multiple countries, at multiple levels (local, national and global) and with multiple organizational members or partners. Their complex architecture also often enables them to deepen the impact of the work of their individual members by bringing in expertise from other locations or levels of the structure. This results in strengthening knowledge, capacity and strategies of all their members / units, and in stronger collective advocacy, multi-centric research studies, and other activities that a single organization may not have the ability, resources or reach to undertake. These organizational structures, therefore, require more complex M&E systems. They may need to combine, for instance, the assessment of their e�e-ctiveness as networks (such as the Wilson-Grau/Nunez framework), their advocacy impact (through tools and recommendations o�ered by Patton and Klugman), as well as a judicious combination of the more conventional frameworks for assessing unit- or member-level results. This is yet another example of why multiple assessment tools are essential to build e�ective M&E systems

4. Make M&E system �exible and adaptable

25

Many women’s organizations in the AWID’s study �nd that the M&E frameworks and approaches required by some donors demand resources that are beyond their capac-ity, such as the abilities of their sta�, the time required for their implementation, or the need for external expertise to which they may not have access. They may also be di�-cult to use for groups working in politically unstable or risky contexts of violence and con�ict. Sometimes, the complexity and amount of data required is excessive, and does not necessarily give a better picture of implementation or impact. This usually happens because M&E requirements are not prioritized in the grant-negotiation process, but treated as an afterthought or add-on by both donor and grantee, result-ing in a nasty shock when the reality of what is involved becomes clear.

Furthermore, we have to remember that macro-level changes, such as lower violence rates, can only be reliably assessed in the long term, and only after accounting for the role of multiple actors involved in the change process, as well as the in�uence of a variety of cultural, political, and economic forces. A single project or organization cannot work on all these factors, much less in a three- or �ve-year timeframe. What we can measure, instead, are the interim changes within speci�c stakeholder groups and our contribution to enabling that change. Therefore, indicators of results should also be realistic and it is important for both the organization and donors to have the same understanding on this.

But this can be pre-empted if we seize the initiative in designing a sound M&E frame-work and indicators for our proposed work—based on quantitative-qualitative balance, “less is more”, and other principles o�ered here—but which are tailored to our contextual realities and organizational capacity, and which demonstrate a serious approach to tracking and evaluating our work. This also ensures that the basis for negotiating the assessment of our work is created by us, rather than others less experienced or expert in the strategies and contexts of our work. Understanding and negotiating M&E expectations at the outset of a project or funding cycle is a useful strategy to avoid tensions and misunderstandings at a later stage.

The research shows that many of AWID’s organization members reported that it has become harder than ever to mobilize resources for their work, particularly when their strategies are seen hard to monitor or their results di�cult to measure. Similarly, this is also experienced by many other organizations. So there is a need to generate more rigorous and convincing data about the e�ectiveness and impact of our strategies. However, the research indicates that assessing our work e�ectively often demands the use of tools and methods that require skills and capacities that may not be available within our organizations, or which it may feel strange to our cultures and traditions of learning. So, when setting up an M&E system either trying to rede�ne or re-design an existing one, it is important to assess the types of sta� capacities, time and other resources it requires, and to develop a mix of tools and methods that are feasible within these constraints.

5. Negotiate M&E systems & indicators of results with donors

6. Develop & Invest in M&E Capacity and Programs

26

In fact we are often launching projects and change strategies without a clear vision of the change we seek, the values and politics that inform the vision, and without a strong diagnosis of the problems we are attempting to address. Baselines can greatly advance this clarity by providing the basis for a clear situational analysis at the outset. This in turn enables us to more accurately place the changes that have occurred in the course of our work—both positive and negative—against this baseline, identify what worked and what did not, and re�ne our strategies accordingly. For women’s rights work, for example, there are many examples of baseline studies conducted by NGOs and women’s organizations assessing the state of human rights, gender relations, violence against women, etc., in many parts of world. The evidence generated through these has helped to monitor and assess the impacts of their interventions in more convincing and accurate ways by generating concrete evidence of where things were at the outset of the project. This has enabled them to better learn about the impact of their change interventions over time, but also to make a much stronger case to exter-nal audiences about what worked—and what didn’t. It is not that only large, well-funded organizations can conduct baseline studies—they can be done even with limited resources through interesting combinations of participatory methods, secon-dary data (gleaned, for example, through o�cial statistics or census data, or surveys and studies done by other organizations), as well as conventional “objective” methods like surveys. Our ability to create baselines, and place our progress along selected indic-ators within these, also enhances the ability of our supporters (including our donors) to make a stronger case to the larger world for why such work needs support

7. Create baselines. Generating periodic baseline data is a powerful but underutilized tool in accurately assessing our role and achievements in the change process—and in locating the most strategic directions for the next phase of our work

D. M&E for Partnership Program

27

5

5: http://partnershipsresourcecentre.org/knowledge-platform/learning-modules/module-2#entry

Who Evaluates what?

How to Elauate?

Who Evaluates Whom?

Partnership raises critical questions about the extent to which collaboration actually adds value in terms of both process and outcomes and how judgments are made (Atkinson, 2005). Evaluating partnerships is di�cult for various reasons such as long timescales for achieving impact, di�erent perspectives on what success means, the complexity and variability of partnership interventions and the di�erent context within which partnerships work. Neither e�ciency of the partnership process itself, nor its e�ectiveness in addressing set goals is easy.

The key point in building and implementing M&E for Partnership is in investing to build a strong partnership. Thus, open and honest communication matters, between donors/aid agencies and the grantees, and among partner organizations, themselves. Articulating and agreeing on shared goals, and how those goals will be achieved, is another essential element. With adequate strategic clarity, both donor and grantee can thoughtfully address the realities of what resources it takes to achieve the goals—including overhead and a reasonable timeframe for results to be achieved. Strategy should drive cost, not the other way around.But, it takes trust and mutual commitment to identify what is and isn’t working, to alter strategies, and to get better together.

So, how do we make partnership works? What should we do to make M&E for partner-ship works?

#1 Select potential partner carefully

28

6: Quoted from interview with BIGS (one of the partner of Building Bridges Partnership) on December 18, 2002 and PGRI (Lead Organization of POWER) on December 20, 2012 – both are a part of IKAT US Program

First it is essential for both donors and grantees to pick potential partners carefully. Foundations refer to this process as “due diligence”: evaluating a potential grantee in much the same way, as an investor would analyze a venture capital opportunity. For donors, investing time up front in carefully selecting grantees will reduce the risk of subsequent problems. And for grantees, it is important to understand your donor’s characteristic and approach and be prepared for engagement with the donor. And for grantees, it’s important to understand that sometimes it is better to walk away from a donor than to get trapped in an ine�ective and burdensome relationship.In partnership program, the lead organization also has to select its partners carefully. Both donors/aid agencies and organization have to invest time to assess and under-stand: Social, political and cultural context in the area which the potential partner organization work

Strengths & weaknesses of the potential partner organization

BIGS’Strategy to initiate a partnership at regional level in Building Bridges – IKAT US Partnership Program:

Ideally, the process should be started with Country Scan Study (Initial Mapping) �rst to understand the political dynamic and CSOs map in each country. However, BB did not do that and R4D used strictly CFP (Call for Proposal) approach that they had done previously in other programs as well. CFP was announced in public CSOs network in regional and national level. The plus point from CFP process: more neutral.BIGS considered that CFP process is not enough; it is better to conduct Country Scan study �rst. BIGS had previously become the subject of Country Scan Study for IBP (International Budget Partnership); a process using qualitative instruments (FGDs). This process allows more understanding of the country context and can prevent issues such as what BB expe-rienced in Cambodia (In Cambodia, the CSO has to obtain permission letter from related Ministry to conduct research). Aside from that, the Country Scan will also inform a sense of time frame in relation to dates of signi�cance in the relevant country that can in�uence activity implementation, based on BIGS experience However, the downside is that Country Scan Study requires more time and more fund.

Box 4. Experiences of IKAT US’s Lead Organizations in Initiating Partnership 6

29

7: Many adjustments of system, legal framework, �nancial management were required. For example, the partner in Malaysia is not used to manage budget with dual-currency.

Kemitraan’s Strategy to initiate a partnership at regional level in POWER – IKAT US Partnership Program:

The names of organizations included in the POWER partnership originated from information of individual who has worked in the country instead of resulted from Mapping of Women Organization or Needs Assessment. After the program has been approved and initiated, during the implementation, we realized that there are other organizations that perhaps are better to be included in the partnership (more experience, acknowledgement in their respective country).

From the proposal submitted, actually Kemitraan could �rstly conduct capacity assess-ment of potential partners in making a proper proposal and in capacity building. How-ever, because it will take longer time to initiate the partnership, especially each organiza-tion has di�erent capacity in program and �nancial management , the Partnership did not pursue deeper due diligence.

Kemitraan admitted that there is no budget to be allocated for a new partner and they cannot increase the budget platform. Finally, they keep involving the organization by inviting them to the workshop and include them in the partnership‘s network.

7

#2 Launch the partnership earlier

30

8: “Successful Partnership: A Guide” by OECD LEED Forum on Partnership and Local Governance

M&E capacity building and implementation are likely to be most e�ective if incorpo-rated from the inception of program design. This way both partners can develop the program and M&E components, and the partner providing M&E assistance has an opportunity to provide input into both program and M&E design.

The partnerships discussed here were not formed until after the programs were designed. M&E capacity building is still possible at this stage; ideally, however, the partnerships would have begun earlier so that the M&E goals could have informed program design.

The following elements need to be taken into account :

a. The partners must be involved in the entire exercise, including the M&E frame work development

•Partners must be able, among themselves, to set the results to be reached and the impacts and outcome to be sought

•Partners must agree on the key indicators that will best demonstrate the relevance and e�ciency of their actions and the long-term impact of the partnership

•Partners must participate in developing the monitoring & evaluation strategy that includes various methods (e.g. collection of statistical data, satisfaction surveys, etc.)

•Partners must identify monitoring procedures and be involved in developing data collection tools

b. M&E process must be implemented at the beginning of the partnership

• Establish initial baseline as quickly as possible and compile key data regularly and on-going basis

• Budget must be provided to accommodate proper M&E from the start

• Establish a monitoring mechanism friendly to the user;it should be constructed in such a way as to facilitate the work of the partners appointed to follow up the activi-ties and procedures.

8

#3 Learn about our partners

31

It is important early in the partnership to understand some key characteristics about the program partners and the context in which they work. Be open to their ideas and constraints. This process paves the way to building partners' M&E capacity, and convincing the partners that M&E will improve their programs without unduly reduc-ing their already limited resources.

Furthermore, we need to recognize that in order to be able to achieve the shared goal, each partner will need to be supported to overcome their organization’s weaknesses. This is also the key for sustainability that is often demanded in each project/program.

In M&E for example, specialist must be engaged and responsible for some of the M&E stages, particularly in evaluation:

• Specialist must lend their expertise to the partners and ensure that there are shared understanding on evaluation methods• They must guide the partners through the process and verify and validate their choice of key indicators or follow-up methods• Partners have to come into an agreement to compile necessary data & informa tion

This approach helps partners familiarize themselves with the evaluation methods and develop a certain expertise in the �eld, guaranteeing both the credibility of the process and the results obtained and validated by the external specialists.

The partnerships may choose among various methods of evaluation that best suits their case, both in terms of membership and in terms of scope and objectives. The use of an external evaluator combined with the partnership’s involvement in the proce-dure shall ensure demonstrable better results.

Many projects are now introducing a start-up phase where a more intensive process of consultation leads to adjustments to project design after funding is secured. Some-times, however, partners feel they are presented with a fait accompli; they are invited to make suggestions, but only within the con�nes of the project budget, sta�ng structures and strategies that have already been de�ned.

32

Box 5. POWER: Showing Leadership & High Quality Training based on Needs

POWER is a partnership of seven civil organizations from 5 (�ve) di�erent countries. This partnership is led by Kemitraan as the lead partner to manage six regional part-ners in initiating changes in each of their respective country. With generous support from USAID, POWER share, learn and improve their capacity together. Six regional partners are National Democratic Institute (NDI), Koalisi Perempuan Indonesia (KPI-Indonesia), Persatuan Kesedaran Komuniti Selangor (Empower-Malaysia), Caucus Feto iha Politica (Timor Leste), The Cambodian Center for Human Rights (CCHR-Cambodia) and the Center for Popular Empowerment (CPE-Philippines).

POWER applied series of activities to improve partner’s capacities in advocacy. Through such approach, the mutuality has been de�ned among partners due to concrete exchange of assistance among themselves. Within two years of partnership, multi-mentoring mechanism has been e�ectively performed. in terms of program-matic as well as organizational capacity building. These are some examples: Kemitraan-NDI-CCHR had carried out training on casework/constituency outreach for elected women as well as speci�c training on facilitator skills for CCHR sta�; Kemitraan-NDI-Empower-Caucus in joint-training for women candidates; Kemitraan-CPE-Caucus in joint-training for media person on gender sensitive training; and Kemitraan-KPI-Caucus in joint-training for women candidates on leadership training.

Another enabling condition created by POWER to increase the sense of belonging towards their new network identity is by trying to obtain engagement and commit-ment from all partner members by accommodating partner's inputs.

Through various activities such as above, the likelihood to impact at national level is expanding as each partner has enhanced capacity to run stronger advocacy program in each country. The e�ort to expand awareness was done through many approaches among others shown by the success of Empower to launch their online database for women in parliament and CCHR to launch online interactive map on women candi-dates for and women elected in the Commune/Sangkat election. In Cambodia, the database became the �rst information media access for public database of women candidate and elected women, while Empower became the �rst group ever to present the result of women in parliament via interactive map in Malaysia.

Source: Lessons Learned Report IV – IKAT US Partnership

M&E: Weaving Learning into Change

Chapter 2

A. Essentials of Monitoring, Evaluation & Learning

As we have concluded in Chapter 1, the key of a meaningful M&E is learning. Then the next question is Which M&E approach to be chosen to facilitate the learning? In the following section, we will explore several M&E tools & approaches. But before we move further, there are several things that we have to remember:

1) It takes a learning culture within an organization or program to take up the chal-lenge of customizing and implementing di�erent PME approaches that are relevant for a speci�c context. In several cases we observed that a crucial element of a learning culture is the presence of a group of people who have the motivation, the courage, and the mandate to address PME challenges in their organizations or program by introducing PME approaches that are new to their organization. The cases further show how support from higher management and trustful relationships can nurture such learning culture.

34

2) Each M&E tool & approach has its own strengths and weaknesses. Organizations and its partners might have to use a mix of M&E approaches to complement each other and address the complex context of their project or program as well as the learning needs. In a partnership program, we need to also consider which approach that can address the situation and needs of the partners. Agreement among part-ners on which approach or mix of approaches to be used is essential in pursuing M&E in a partnership program.

3) Sharing:Learning through collaboration. Collaboration is a fashionable buzzword – and for a good reason. Undeniably, we simply have to work with a wide range of people and organizations to be able to contribute towards real and sustainable results. A single organization hardly has enough resources and skill to achieve intended impact. But collaborating is di�cult, it is hard enough for people inside an organization to collaborate, let alone get two or more organizations to work together. This is where learning together has a valuable role to play. There is now enough expe-rience to show that if organizations or communities learn together, sharing their prac-tices and experiences, their re�ections and learning, then this starts to lay foundation for collaboration. It helps people learn about each other and understand each other before they decide to work together.

B. M&E: Some Tools, Methods & Approaches

35

Please do remember that this list is not an exhaustive list of M&E approaches. There are other M&E approaches developed out there and it will be good to assess them one by one to see whether it �ts with our organizational or partnership architecture. In this chapter we will try to look at several of the approaches, its strengths and weaknesses and how it can be applied to our work. The logical framework approach will no longer be discussed since many organizations are already familiar with the approach.

Most Signi�cant Change

36

What:

Most Signi�cant Change (MSC) is a method of participatory evaluation that involves the collection of signi�cant change stories at di�erent levels of the intervention (for example project sta�, change agents, intervention participants) and collectively deciding on the most signi�cant change stories based on selected themes (called domains). The domains re�ect broad categories, such as change in capacity to take action, or a change in participation in an activity, as well as lessons learned.

Main steps to design an MSC process :

• A good means of identifying unexpected changes, both positive or negative• Enables monitoring of initiatives that do not have pre-de�ned outcomes against which to evaluate• It is a participatory form of monitoring that requires no special professional skills • It encourages analysis as well as data collec tion because people have to explain why they believe one change is more important than another• It can build staff capacity in analyzing data and conceptualizing impact

This is where you should introduce the MSC process to all the stakeholders and clarify its purpose. It is important to also consider who are the best people involved in the project to collect stories from.

Domains are broad and often fuzzy categories of possible changes. For example, participants in the MSC could be asked to look for signi�cant changes in four domains: - Changes in the quality of people’s lives- Changes in the nature of people’s participation in sustainability activities- Any other changes…

1. Starting

2. Establishing 'domains of change'

• MSC is not meant to be used as a stand-alone methodology• Generally not as useful in situations when the implementation processes and outcomes are straightforward and the causal paths connect ing inputs and outputs are clear• Outcomes that are recorded through signifi cant change stories may be positively biased. MSC also raises the important issue of voice and power associated with who participates in the story selection process

Strengths Limitations

9:Referring to “Storytelling” http://evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=61&Itemid=157

9

37

The central part of MSC is an open question to participants, such as:

‘Looking back over the last month, what do you think was the most signi�cant change in the participants practice of sustainability in their home life?’The question has six parts:

‘Looking back over the last month…’ – It refers to a speci�c time period.

‘…what do you think was...’ – It asks respondents to exercise their own judgment.

‘…the most signi�cant…’ – It asks respondents to be selective, not to try to comment on everything, but to focus in and report on one thing.

‘…change…’ – It asks respondents to be more selective, to report a change rather than static aspects of the situation or something that was present in the previous reporting period.

‘…in the participants practice of sustainability …’ – It asks respondents to be even more selective, not to report just any change but a change in the participants understanding of sustainability. This tag describes a domain of change and can be modi�ed to �t other domains of change (for example, participants understanding of sustainability in the commu-nity).

‘…in their home life?’ – Like the �rst part of the sentence, this establishes some boundaries. In this particular case we are not asking about people’s actions in the community or what others are doing in Australia, but focusing about people’s home life.

Selecting stories usually involves a hierarchical process, where the lower levels select signi�cant stories for the upper levels to review, and select the most signi�cant ones from the lower levels

Recommended Use of MSC:

Community development & empowerment as well as behavior change intervention Exploring the impact of an intervention both intended and unintended impacts Distill outstanding experience or initiatives and lessons learn

Remember that MSC is not meant to be used as a stand-alone methodology. MSC also is not suitable for interventions that require quantitative measurement.

MSC will also be a very useful tool in assessing the impact of collaboration in a partnership program: what are the impacts of partnership programto the organizations within the partnership? As elaborated in the previous chapter that the current world of development intervention is populated with organizations with diverse and complex architectures, including in forms of collaboration or partnership. And therefore, the M&E system also needs to cater the question of partnership’s e�ectiveness aside from their overall impact. However, most do not have a speci�c pre-determined partnership framework. MSC can be used to draw the experiences and learning of each member of partnership in order to �nd ways to strengthen it.

4. Reviewing stories and selecting most signi�cant change story

It is really important to feed back your selected SC stories and the reasons for the choice to the relevant people

5. Feedback

38

Box 6. Illustrative Use of MSC for ASIA CALLING

ASIA CALLING: Telling the Untold Stories

Telling the unheard stories is the core mission of the Asia Calling Network (ACN). After two years expansion, ACN has become one of the most extensive independent media networks in Asia working to promote democratic values across the region. With a 3 year program of support from USAID Indonesia, since April 2011, ACN has expanded its outreach by diversifying channels and media, including radio broadcasting, televi-sion, online and print media. Under the IKAT-US partnership program, the Indonesian Association for Media Development or Perhimpunan Pengembangan Media Nusan-tara (PPMN), as the leading partner, works together with KBR68H and Tempo TV in Indonesia, Media Development Investment Fund, Malaysiakini, 100 FM Chiang Mai University (CMU)–Thailand, Democratic Voice of Burma (DVB) and Cambodian Centre for Independent Media (CCIM) Cambodia. PPMN distributes its program in Indonesia, Thailand, Cambodia, Burma, Singapore, Malaysia, Philippines and Timor Leste.

Apart from the geographical outreach that has expanded beyond the program’s targets, the program content of ACN has had made signi�cant impacts on bene�ciar-ies as reported in the midterm evaluation result. Views from Asia Calling correspond-ents who have been trained by ACN felt that training provided by Asia Calling actually improved their journalistic skills. The evaluator stated that IKAT-US funding for train-ing at Asia Calling is being leveraged in multiple ways. There is no better journalism training anywhere in the world than that o�ered by Asia Calling and KBR68H.

In terms of programmatic content, the most interesting outcomes beyond the Asia Calling program have been the concrete impacts of some stories. A signi�cant number of Asia Calling’s features have led either to additional media coverage or to political action and increased community awareness in many countries.

Clarence Chua of Malaysia, a country where mainstream media outlets are controlled by parties close to the ruling coalition -- reports that his stories have been picked up by other independent and even international media. Meanwhile Khortieth Him of Cambodia reported signi�cant impact in response to a story on education for Muslim girls:

A story that had “real National” impact was the story on Young Muslim girls who have less chance to enter school. After having this story aired, the source who was a Parlia-ment member and was interviewed by Asia Calling took action to reduce the obsta-cles to young Muslim girls going to school. It was because the teacher did not allow them to wear Hijab in class and �nally the government o�cially allowed Muslim girls wearing Hijab in class and as a result many Muslim girls now have come back to school.

Stories of changes starting from how journalism training brings changes to the media network, how media networking brings changes among the members of the network and how the “untold stories” broadcasted by the media network a�ect the lives of people just like the previous story.

Source: Lessons Learned Report IV, IKAT US Partnership

Outcome Mapping

39

What:

Outcome Mapping (OM) focuses on measuring changes in the behavior of the people with whom a development initiative works most closely. The goal is to identify which stakeholders contribute to shifting behaviors, actions, activities, beliefs, policies, etc.—focusing on contribution, rather than attribution, to change. Outcome mapping limits its concern to those results – or “outcomes” – that fall strictly within the program’s sphere of in�uence. It considers only those activities where the program can claim its contribution to a direct e�ect.

Three main terms in OM:

• Behavioral change: Outcomes are defined as changes in the behavior, relation-ships, activities, or actions of the people, groups, and organizations with whom a program works directly. These outcomes can be logically linked to a program’s activi-ties, although they are not necessarily directly caused by them.• Boundary partners: Those individuals, groups, and organizations with whom the program interacts directly and with whom the program anticipates opportunities for in�uence. Most activities will involve multiple outcomes because they have multi-ple boundary partners.• Contributions: By using Outcome Mapping, a program is not claiming the achievement of development impacts; rather, the focus is on its contributions to outcomes. These outcomes, in turn, enhance the possibility of development impacts – but the relationship is not necessarily a direct one of cause and e�ect.

• OM provides a focus on people and institu tional transformation that is often lacking in techniques which emphasize the delivery of outputs as indicators of achievement.• Milestones that indicate a possible process, not final indicators; these indicate a path of change that makes it possible to assess the development in short time period and therefore to assess / change / adapt strategies within a short time.

• OM is not intended for a technical evaluation to assess the relevance of the programming area or an evaluation of the cost-e�ectiveness of one approach compared to another• OMis only concerned with the change processes which occurred in those whom the program directly interacts with

Strengths Limitations

40

Three general stages of Outcome Mapping :

In this stage, the project establish a consensus on what changes in the macro level that it will help to bring about and then plan the strategies to be used to achieve the changes. Organi-zation should clearly express the long-term downstream impacts that they want to achieve, bearing in mind that the project will not achieve the goal single-handedly.

It helps to answer four questions: - Why? (What is the vision to which the program wants to contribute?); - Who? (Who are the program’s boundary partners ?); - What? (What are the tangible changes that are being sought?); - How? (How will the program contribute to the change process among its bound ary partners?)

This stage provides a framework for the on-going monitoring of a project’s actions and the boundary partners’ progress toward the achievement of outcomes. It is based largely on systematized self-assessment. It provides the following data collection tools for elements identi�ed in the Intentional Design stage: an Outcome Journal’ (to track impact against progress markers); a Strategy Journal’ (that seeks to test and adapt the programs strategy in ever changing circumstances); and a ‘Performance Journal’ (that logs organizational practices and gauges the need for improvements).

It helps the project identi�es evaluation priorities (more in-depth review of progress) and develop an evaluation plan that makes good use of resources and provides strategic bene�t to the project

1. Design stage (Intentional Design Stage)

2. Monitoring stage (Outcome and Performance Monitoring)

3. Evaluation planning

10: “Outcome Mapping: A Basic Introduction” by Research to Action, http://www.researchtoaction.org/2012/01/outcome-mapping-a-basic-introduction 11 : Boundary partners are individuals, groups or organizations with which the program interacts directly and which the program hopes to in�uence

10

11

41

Recommended Use of OM:

For interventions that the target outcomes are in the forms ofchanges in attitude, knowledge, practices and behavior at stakeholder level (boundary partners).

Based on the indicative assessment by OutcomeMapping Learning Community from September 2011 – March 2012, OM is dominantly used in Governance Sector.

Potential use of OM in IKAT US Partnership:

Building Bridges. Its main intervention is to develop partner civil society organizations’ (CSO) research and advocacy skills. OM can be useful because it enables the partnership to measure changes in the knowledge & practice of its regional partners in using PETS, CRC and Social Audits in each distinct national context as well as to measure changes in attitude of its advocacy targets.

Human Rights in ASEAN. OM tool can be used to map outcomes related to changes in capacity of Indonesian lawyers and legal CSOs to serve as regional human rights advocacy resources and changes among boundary partners due to regional advocacy strategy pursued by this intervention.

AGENDA.OM will be a great tool to see changes in knowledge and practice of DPOs (Disabled People’s Organizations) and CSOs (Civil Society Organizations) within AGENDA’s outreach and intervention in relation to the issue of political rights of people with disabilities as well as the outcome in capacity building for electoral monitoring bodies. Furthermore, it is also useful to track changes in the lobbying to in�uence ASEAN policy-making.

Similarly, OM can also be applied to intervention with strong capacity building and advocacy element, such as POWER as well as for exchange of knowledge in the case of South-East Asia Partnership for Better Governance in Extractive Industries.

Remember!OM is meant to be �exible, complimentary approach to traditional M&E methods. OM is particularly e�ective when used from the planning stage, as it helps a program to focus on supporting speci�c changes in its partners. With some adaptations, its various elements and tools can be used separately or in conjunction with other processes (for example, a SWOT, a situational analysis or an LFA).

42

Box 7. Outcome Mapping in ACCESS Phase II

ACCESS Phase II has the following strategic directions:

• Strengthening engagement between civil society and government.• A focus on empowering citizens’ participation for democratization, from the grassroots to village, sub-district, district and national levels.• Scaling up impact through working across the systemic issues of district wide governance and through contiguous geographic expansion.

ACCESS acknowledges that local actors – government and citizens - are the driving force for better local democratic governance. Governance is a complex adaptive system involving multiple actors and factors (of which ACCESS Phase II is one) so that sustainable system change evolves from continually improving behavior changes, including interactions with others. As people experience positive results in working together, they are motivated to continue and scale up the benefits they have created. This reinforces development of norms for a more inclusive, equitable and transparent society and leads to self-directed local democracy.

ACCESS Phase II brings together interested actors, particularly those traditionally excluded, to work out how to best set and respond to their own development priori-ties, drawing on local capacities and assets. Outcome Mapping is well suited to this approach with its systems and actor-centered orientation. It is interpretive and non-linear and recognizes that change is influenced by many external factors and is not always predictable. By encouraging collective sense making and continuous learning for improvement, OM supports development of new understanding and places local actors as the central agents of change rather than as implementers or recipients

The OM’s approach provides rationale for selection of CSOs as Boundary Partners since they are the organizations best positioned to be influenced directly by the Program and in turn have high potential to influence others (Ultimate Beneficiaries). With ACCESS support, these Partners together with ultimate beneficiaries develop a Vision, Outcome Challenge (anticipated behaviors of different actors in system by the end of the Program), Progress Markers (sequence of desired behavior changes for themselves) and Strategy Maps (how they will support their partners to make change). Six monthly reflections with partners and stakeholders create shared under-standing about change processes, provide data on the contributions of different actors including ACCESS and identify what strategies need to be adjusted to strengthen performance.

In Indonesia, there are two development programs ‐ VECO Indonesia and ACCESS Phase II ‐ using Outcome Mapping (OM) as the basis for their intentional design and respective monitoring process.

43

Since ACCESS is only the second organization to implement OM in Indonesia (and the �rst to do it in a large scale). Its process has been a trial-learning case. Nevertheless, the overall ACCESS’ CSO partners and government partners have found the OM approach useful in providing a clear rationale for determining who should work with whom and tools for planning and monitoring through progress markers.

However, it is also the case that building awareness and capacity to use OM requires time, ongoing reiteration of key ideas and expert technical assistance should be done continuously. This is partially because it is conceptually challenging for CSOs to adjust to using OM when they are used to the log frame approach and there is a lot of new terminology (in English) that is di�cult for non-English speakers to grasp easily. In response to this learning, ACCESS has trained provincial ACCESS sta� and local facilita-tors in using OM to provide ongoing support for CSO partners and developed an Indonesian Outcome Mapping which �ts more with local linguistic and cultural contexts of Indonesia, provides Indonesian terminology and explains di�erences between OM and the log frame approach.

Theory of Change

44

What:

A Theory of Change (TOC) is a tool for developing solutions to complex social prob-lems. A basic TOC explains how a group of early and intermediate accomplishments sets the stage for producing long-range results. A more complete TOC articulates the assumptions about the process through which change will occur and speci�es the ways in which all of the required early and intermediate outcomes related to achiev-ing the desired long-term change will be brought about and documented as they occur. (Adapted from Anderson, A. (2005). The community builder’s approach to theory of change: A practical guide to theory and development. New York: The Aspen Institute Roundtable on Community Change)

Generally, a theory of change includes:

• A logic model/results chain;• The assumptions, risks and, in some cases, the mechanisms associated with each link in the logic model/results chain;• The external factors that may influence the expected results; and• Any empirical evidence supporting the assumptions, risks and external factors.

Theories of change are referred to by a variety of names, including program theories, impacts pathways, and pathways of change.

• Providing means to reach agreement among stakeholders on just how the intervention is expected to contribute to its intended aims. In more complicated situations, it can help clarify the intended contribution by the various subcomponents of a broader intervention.• Helps identify not only which results should be monitored, but also which other factors should be followed so that the intervention can be on track and its progress monitored.• Identifying where research may be needed to better understand how the intervention works.

• They do not necessarily provide a quantitative measure of the size of the contribution an intervention is making.• More than one theory of change may emerge. If multiple theories of change emerge and are strongly held, they may have to be tested against the evidence to see which theory best reflects reality• While context is accounted for in the Theory of Change, potential unexpected consequences are not accounted.

Strengths Limitations

45

Main Steps to Create a Theory of Change :

Start to design a simple map of the preconditions required to bring about the long-term goal. The mapping process helps stakeholders to visualize and prioritize their goals as well as specify what they expect to change and for which outcomes they want to be held account-able for.

A more detailed stage of the mapping process. Building upon the initial framework, we continue to map backwardly until we have a framework that tells the story we think is appro-priate for the purposes of planning.

Because this work is challenging and most social change programs or broader initiatives have a lot of moving parts, change frameworks usually go through many revisions. Outcomes are added, moved and deleted until a map eventually emerges that tells a story the group can agree on.

Identify the interventions that your initiative will perform to create these preconditions.

1) Identifying long-term goals

2) Backward mapping and connecting outcomes

3) Identify interventions

The Indicators stage is when details are added to the changed framework. This stage focuses on how to measure the implementation and e�ectiveness of the initiative. By collecting data on each outcome, the initiative can identify what it is or isn’t happening and �nd out why.

Remember that each indicator has four parts: population, target, threshold and timeline.

4) Developing indicators

5) Narrative of ToC Write a narrative that can be used to summarize the various moving parts in your theory.

12: Quoted from www.theoryofchange.org

12

46

Recommended use of ToC:

Theory of Change (ToC), as its name, focus on changes. Di�erent type of changes can be mapped out – changes in attitudes, knowledge, awareness, skills, behavior, health, �nancial status, social & economic conditions, visibility of issues, com-munity norms and changes in partnerships.

Theory of change is increasingly used in research, advocacy and policy change outcome as well as other projects seeking to achieve social change. An increasing number of research donors are now recommending that researchers should develop a theory of change (ToC) to help think through how they can do and communicate the results of the research in a way that will maximize the value of the research for policy and practice.

However there are few pre-conditions for optimal use of ToC approaches: i) It is suitable for more open & learning organization, and ii) programs that are at early stage in the grant process. It is found that it was harder for an organization to think openly about its theory of change once it was already well into a project and had set up many of its systems.

Box 8. Illustrative Use of ToC in IKAT US Partnership Program

As expressed by various reviews, ToC is great for complex program seeking to achieve social changes. Thus, for complex-regional partnership program, such as IKAT US Partnership, ToC will be a great tool!

However, ToC should also be developed at the beginning of grant process involv-ing the partners,where each partnership will develop their own distinct theory of change! And this process should be strongly facilitated since many partners might not be familiar with the approach.

ToC will enable us to map out:

1) Changes that each intervention seeks to achieve

• Changes in human rights policy (in ASEAN level) or mechanism.• Changes in knowledge or skills of legal advocates and CSOs in advocating for human rights.

• Changes in knowledge and awareness of relevant stakeholders on Convention on the Rights of Persons with Disabilities (CRPD) leading to changes in policies or regulation in relation to political rights of people with disabilities.• Pathway of change: from access to knowledge (Regional Election Access Index) to better advocacy and policy regarding disability access to elections.

StrengtheningASEAN’s Human Rights System

AGENDA

47

• Changes in capacity and skills of Southeast Asian media and regionally-based journalists leading to better quality media coverage with strong emphasis on democracy, good govern ance and human rights issues. • Changes from visibility of issues to awareness and knowledge of audiences in the region on democracy, good governance and human rights issues leading to promotion of such values.

Asia Calling

• Changes in capacity of regional partners in using tools such as PETS (Public Expenditure Tracking System), CRC (Citizen Report Cards) among regional partners.• Changes in budgeting and service delivery of government counterparts.• Pathway to change: from access to knowledge (online Social Accountability Atlas) to improved performance of CSO-led transparency and accountability projects in Southeast Asia.

Building Bridges

• Changes in knowledge and capacity of CSOs in Cambodia, Malaysia, Philippines and Vietnam that lead to the adoption and implementation of EITI (Extractive Industries Transparency Initiative) by their respective government.• Changes in attitude and policy of ASEAN to endorse transpar ency and accountability in managing extractiveindustry resources, including but not limited to EITI.

• Pathway to change: from research (knowledge) to capacity building of women leaders and women’s political caucuses leading to improvement of women’s political representation.• Changes in legal framework for elections that increase women’s political representation.

The partnership becomes more strategic, increased sharing of resources, foster learning that leads to better project or program performance

Better Governance in Extrac-tive Industries

IKAT US Partnership

POWER

2. Changes in Partnership

48

Box 9. Experience of HIVOS: Building Southern Organizational Capacity Through Theory of Change

Dutch funder, Hivos, became interested in theory of change as a way to support its 800 southern partners to focus more on outcomes than activities and to improve their planning, learning, monitoring and evaluation. Hivos did not prescribe a rigid format for partners to present information to them, but they noted that many have to use logframe for other donors and rarely �nd them helpful for their own monitoring – and even less for learning.

So Hivos tried a �exible approach to results-oriented planning, monitoring and evalu-ation, based on indicators developed by their partners. Hivos found, though, that the quality of planning, reporting and learning did not improve as much as they expected. They therefore decided to explore further theory of change – which they had already re�ected on in an internal policy paper in 2004.

Since 2007, Hivos therefore carried out a number of workshops with partners in di�er-ent regions, facilitated by consultants. The workshops went well and most partici-pants were initially very positive about the approach. But Hivos found that very few managed to continue the process, even where they were o�ered consultancy and funding support to do so. It was not always clear what held them back. Some partners complained that it was just a way of imposing yet another kind of logic model on them. Others feared that it would mean extra work as other donors continue to demand logframe.

For that, Hivos reached to a wall. They still thought theory of change could be useful in helping partners improve the e�ectiveness of their strategies and their learning, as well as dealing with donor demands. But the capacity building workshops were not working in the long-term. Hivos also recognized that to integrate theory of change thinking in organizational practice needs very good facilitators who can adapt to the process of di�erent kinds of organizations.

To take the process further, Hivos engaged a small group of consultants – from CDRA in South Africa, Wageningen University in the Netherlands and their Latin American consultant. They began an action learning process with partners in Southern Africa and South America to explore the value of applying theory of change thinking in di�erent ways and in di�erent contexts. In South America, Hivos invited partners to apply to be part of a longer term learning process – and received a very enthusiastic response. Some partners in Bolivia are developing a group theory of change together, as well as re�ecting on their own organizational theory.

As the processes in the regions develop, Hivos hopes that the groups will gradually expand to incorporate more partners and more consultants, generating interest locally, rather than imposed by them as the funder. Hivos is also thinking about setting up a web-based resource base.

Source: Theory of Change Review”, Comic Relief , September 2011

C. Hybrid Approaches

As emphasized in the previous chapter, there is no such thing as “One Size Fits All”. Approaches that include multiple M&E frameworks/tools methods are more e�ective, as AWID’s study pointed out. Remember that each M&E approach & tool have its own strength and limitations, this is the reason why some organizations have initiative to develop a hybrid approach, hoping that it can serve their needs better despite the limitations. The key is to basically look at the available tools & approaches and decide which one or which hybrid approach that our organization or our partnership want to pursue, the one that we think might serve our purpose the best, and be committed to pursue, implement and review the approach comprehensively and optimally.

49

Box 10. Women’s Learning Partnership (WLP) M&E Framework

The Women’s Learning Partnership (WLP) has developed a model of M&E that draws from participatory Theory of Change approaches, engaging diverse stakeholders. As an international women’s rights network with over 20 autonomous partners from across the Global South, the organization needed a coherent model for M&E that could be used in many di�erent contexts, from well-developed organizations in stable political contexts to burgeoning NGOs in unstable contexts. WLP settled on a M&E strategy that was adaptable by context drawing from Theory of Change and Outcome Mapping. Speci�cally, the WLP Theory of Change is built on the following assump-tions: • Change happens at different levels - The individual, organizational, movement, societal, conceptual and at policy levels.• Change is political and is about - Transformation of power.• Effective and sustainable change is - Participatory, horizontal and inclusive - Based on a shared vision - Based on critical re�ection, learning and is responsive and �exible - Based on living core values and beliefs.• Change is the result of many different actions and contributions - WLP International is part of a larger partnership and movement. - WLP International believes it is the most e�ective when consolidating and strengthening the capacities of others.

13: Rakhee Goyal and Alexandra Pittman. 2009. Measuring Change: Evaluating Social Change in Diverse Contexts and2007. “The Women’s Learning Partnership’s Theory of Change.” WLP

13

50

Focus of what??? Attention? Work? Action? is placed on outlining the key obstacles to achieving intended long-term and shorter-term outcomes. This strategic contextual assessment helps WLP identi�es the main external risks that threaten progress towards their overall goals so they can best strategize and adapt their programming based on any unique constraint they encounter. For example on how the longer term outcomes were mapped in relation to the broader obstacle analyses, please see the chart below.

After identifying the main obstacles to achieving long-term outcomes, WLP speci�ed their main strategies designed to reach longer-term outcomes, along with shorter interim goals. WLP detailed three primary strategies to achieve their intended outcomes: 1) advocacy and movement building; 2) curriculum development and training, particularly through its Leading to Choices (LTC) program; and 3) organiza-tional capacity building. In order to evaluate their strategies and particularly those related to programming, they have developed �exible and adaptable assessment processes. Unique to the WLP model is the attention on the cultural speci�city of evaluation frameworks. WLP encourages “�exibility and suitability for adaptation to di�erent cultural contexts.” WLP also encourages a range of data collection methods depending on context and needs of the partner organization and the population being targeted. While many of the WLP partners prefer typical interview or survey approaches to measuring change, others such as partners in Jordan, prefer to gather oral testimony since in their culture oral traditions of passing down social and historical narratives are a commonplace. As such each partner organization adapts the shared Partnership-level Theory of Change in order to meet its own speci�c and unique needs.

• Societies are governed by gender equitable norms and policies.• Strong moderate feminist voices that are grounded in human rights shape public opinion.• Strong progressive civil societies exist with women exercising in�uence in key decision-making positions.• Pathway of change: from access to knowledge (Regional Election Access Index) to better advocacy and policy regarding disability access to elections.

• Culture of patriarchy.• Political use of religion.• Weak civil society, authoritarian governments.

Long-term outcomes: Key obstacles:

51

14: “AWID’s Evaluation Strategy: Approach, Framework, and Mechanisms. (2006 – 2010)”, accessed at: http://awidme.pbworks.com/w/page/36200176/AWID%20ME%20Framework

14Box 11. AWID (Association for Women’s Rights in Development) ME Framework

AWID integrates an Outcome Mapping and Theory of Change approach. AWID recognizes that the promotion of social justice is essentially about changing how people relate to each other and their environment. For this reason it is helpful for AWID to track changes by stakeholder groups (or boundary partners) as well as making explicit the pathways through which they see changes in social justice occur-ring. Example of How AWID Monitors and Evaluates its Progress AWID has used an outcome mapping system to assess changes in each of their Strate-gic Initiatives on an annual basis leading toward achievement of their 5-year outcomes. In the past, these annual outcomes were identi�ed at three levels: outcomes they would “Expect to See” (which would be easy to achieve), “Like to see” (through active learning or engagement), and “Love to see” (profound change). How-ever, after trying this system out for awhile, AWID realized that the process was incred-ibly time consuming and not producing the type of streamlined information they wanted to see. As such, they adapted the model for more meaningful tracking of outcomes and annual progress. An example of more recent outcomes with more streamlined progress markers for tracking and assessing progress appears below for Building Feminist Movements and Organizations’ (BFEMO) initiative:

WITM Outcomes 2006-2012OUTCOME 1: Women’s rights activists have an in depth-understanding and cutting edge knowledge on key issues related to funding sectors and resource mobilization for women’s rights work.

OUTCOME 2: A signi�cant increase in the quantity and quality of funding available to support the work of women’s organizations in all regions.

OUTCOME 3: An increase in the capacity of women’s organizations and movements to build more equitable and political relationships with donors, network, use informa-tion and improve their strategies to mobilize resources for their women’s rights work.

OUTCOME 4: A transformation in the relationship that women’s rights activists, organi-zations and movements have with money, moving towards a more political perspec-tive on resource mobilization as a crucial part of women’s rights agendas

Where is the Money for Women’s Rights? 2010 M&E Plan

52

Annual Outcome 1

Annual Outcome 1 Annual Outcome 1 Annual Outcome 1

Continue to update the knowledge on funding trends for women’s rights work and organizing, mainly tracking the information available within the donor agencies and monitoring the impact of the crisis on di�erent funding sectors

1. A paper containing updated analysis of the current global context and trends that are in�uencing the funding landscape for women’s organizations and movements. 2. 5 briefs (initiated in 2009) with an update on funding trends for various sectors as well as an analysis of the impact of the �nancial crisis and economic recession. Briefs will be on funds, bi/multilateral agencies, INGOs and private foundations as well as the crisis’ impact on women’s organizations. 4. Strengthening our strategy for information dissemination. This includes continuing the quarterly WITM newsletter, reaching out to partners to identify useful means of disseminating resources through their networks etc…

• An increased demand for and use of the Fundher reports, fact sheets and other information and tools produced by WITM. • Women’s organizations report a greater understanding of dynam-ics shaping resource mobilization for women’s rights work. • A larger and more engaged WITM constituency • WITM briefs used for donor advocacy purposes • More in-depth region-specific information and analysis is available on resource mobilization in CEE/CIS

• Track number of requests and demands that come in for WITM action research • Track mention of WITM in research, reports of other organizations • Designing questions for the 2011 WITM survey on: the impact of the crisis on women’s rights work and organizing and the utility of the WITM research • Place a question in the AWID membership newsletter inviting women’s organizations and donor agencies to share their experiences and stories of how they have been a�ected by the crisis • Launch the bi/multilateral and the women's organizations brief at the 2010 CSW in events aimed at increasing bilateral support for women's rights work, collect feedback. • Analyze statistics of the WITM landing page on awid.org and using Google analytics to track how often people are accessing the landing page and its contents. • Post-meeting evaluation of the CEE Strategy meeting.

Measuring the “Immeasurable”

Chapter 3

Governance is a complex concept. It includes the state's institutions and structures, its decision-making process, its capacity to implement guidelines and the relationship between government o�cials and the public. The e�ort to develop governance indi-cators has been undertaken in international, national and even subnational level. But as governance indicators have proliferated in recent years, so have their use and the controversy that surrounds them.

In using the Good Governance indicators, it is vital to be aware of the ins and outs of the indicators: what are the methodologies behind the establishment of the indicator, the practical and political needs that motivate the production of the indicators, etc. – in order for us to decide when and where we can use that indicator! Understand precisely what we can and can’t say using the data!

Some of the best-known governance indicators are the World Bank Institute’s Gov-ernance Indicators/ WGI and Transparency International’s Corruption Perception Index/ CPI. Both are aggregates of indicators that are already exist.

Let’s look further at Corruption Perception Index (CPI). CPI is an aggregate index of perceptions of corruption in the public sector. Why is CPI only based on percep-tion?

“Corruption generally comprises illegal activities, which are deliberately hidden and only come to light through scandals, investigations or prosecutions. There is no meaningful way to assess absolute levels of

corruption in countries or territories on the basis of hard empirical data. Possible attempts to do so, such as by comparing bribes reported, the number of prosecutions brought or studying court cases directly linked to corruption, cannot be taken as de�nitive indicators of corruption levels. Rather they show how e�ective

prosecutors, the courts or the media are in investigating and exposing corruption. Capturing perceptions of corruption of those in a position to o�er assessments of public sector corruption is the most reliable

method of comparing relative corruption levels across countries”

Therefore, CPI has its limitation: it cannot measure changes over time. CPI cannot be used as a measure to national performance in combating corruption. How-ever, it is e�ective as a tool for raising global awareness & advocacy, which relates to naming and shaming exercise – it can be used as an incentive for govern-ments to improve. CPI can be used as an entry point for further country level discus-sion. CPI does not tell the full story of corruption in a country.

54

15: Williams, Gareth, “Policy Practice Brief 6: What makes a Good Governance Indicator?”, http://www.thepolicypractice.com/papers/17.pdf 16: http://www.transparency.org/cpi2012/in_detail#myAnchor2

A. Utilizing Good Governance Indicators 15

16

Example #1:

IGI (Indonesia Governance Index) is Indonesia Governance Index (IGI) is composite index assess good local governance based on objective and measurable criteria. IGI has 89 actionable indicators of (actionable indicators) good governance, which guide policy maker and other stakeholders to re�ne what need to be �xed. IGI was �rstly introduced in 2009 and initiated by Kemitraan (Partnership for Governance Reform).

IGI is produced through a complex process, which used robust methodology and variety of data to produce score of assessment that re�ects performance and quality of good provincial governance. By using some statistical and mathematical tech-niques and procedures, all of the three types of data which spread to 89 indicators, were transformed into 1 -10 range value. Score of all indicators are aggregated by weighting indicator to generate the index. The weight of each indicator is obtained by separate method, called Analytical Hierarchy Procedure.

IGI’s strengths: 1. IGI uses a lot of actionable indicators or indicator that can be followed-up immediately by actors to improve good governance quality of provincial gov ernment, local parliament, bureaucracy, civil society and economic society. 2. IGI highly correlates with some measurements such as HDI, poverty rate and other economic indicators, among others Provincial Income Per Capita and Unemployment rate. IGI’s cross validation with other measurements showed the robustness of IGI’s methodology. 3. IGI is di�erent with other indexes, which are mostly based on perceptions. IGI uses data which obtained from observation of accessibility of o�cial public documents.

With its robust methodology, IGI has multiple uses, starting from informing a project or program’s intervention, measurement of outcome or impact level, advocacy tool, etc. But IGI is developed speci�c for Indonesia’s context and currently limited to provincial performance.

Nevertheless, the result of the IGI has been used by a variety of government and non-government stakeholders. The national Government uses the IGI as a mirror of their performance and reference for their development planning processes. Civil society organizations use it as reference in engaging with the government while the academic community uses it as reference to teach in class. The followings are some examples:

• In December 2010, IGI was used as the good governance criteria to select the Pilot Province for REDD+ (Government of Indonesia-Norway LoI) and men tioned as one of references in “the Guidance for the Provision of Information on REDD+ governance” initiated by UN-REDD, UNDP, FAO, & UNEP.

55

17: IGI’s project document, Partnership for Governance Reform 2012.

Example #2 :

17

• Recently, the Corruption Eradication Commission (KPK) was proposed to include the IGI in the draft of the National Strategy for Preventing and Combat ing Corruption (STRANAS PPK) prepared by BAPPENAS (National Planning Agency) and UKP4 (President’s Delivery Unit for Development Monitoring and Oversight) as one of the progress indicators together with other measurements such as the Corruption Perception Index, Compliance to UNCAC, National Integ rity Index.

56

57

The following table illustrates the users and use of governance indicators. Please do remember that there are many initiatives in national, regional and international level in developing such indicators – that might not be represented in this table, thus can be added. The purpose of the table is merely to show example of the governance indi-cators available and what can we use them for.

Table 1 – Users and Uses of Governance Indicators

1.1 Following governance trends in a country (national and sub-national), identifying broad priorities for country strategies

1.2 Using indicators to inform program design

1.3 Monitoring & evaluating program performance

Repeat measurement over time, single and aggregate measures

Actionable indicators linked to the performance of public policies and institutions, and pointing to required improve-ments

Public Expenditure and Financial Accountability (PEFA) Assess-ment

National & sub-national indica-tors, such as IGI (Indonesia Governance Index)

Issue-speci�c indicators: RGI

Speci�city to the program’s intended purpose and results

Program logframeNational & sub-national indica-tors such as IGI for evaluating outcome & impact level

1.4 Allocating aid between countries according to govern-ance performance (selectivity)

Inter-country comparability. Robust, impartial and defensi-ble indicators

Millennium Challenge Account country scorecards World Bank CPIA scores EC Governance Incentive Tranche

World Governance Indicators

Regional indicators: Asian Barom-eter, etc.

National indicators: IGI (Indonesia Governance Index), GGI (Philippine’s Good Govern-ance Index), etc.

Issue-speci�c indicators: CPI (Corruption Perception Index), RGI (Resource Governance Index)

Users and uses

1)Development agencies and practitioners

ExamplesRequired features of indicators

18: Williams, Gareth, “What makes a good governance indicator?”, Policy Brief 6, January 2011, www.thepolicypractice.com

18

58

1.6 Using indicators to inform policy dialogue with govern-ment

Selection of indicators based on issues of joint concern to donors and government. Robust methodology required. Indicators must be contestable and veri�able.

Performance Assessment Frameworks for Budget SupportHuman Development IndexNational & sub-national indica-tors such as IGI.

2.1 Monitoring performance in relation to constitutional rights, service delivery and public sector reform

Speci�city to the policy and institutional context of the country.

PRSP monitoring frameworks

National & sub-national indica-tors such as IGI (annually).

3.1 Understanding causal processes explaining govern-ance and development trends

Cross country comparability and long time series to identify explanatory variables using econometric techniques. Suitability for the use in randomized Controlled Trials to test the e�ect of speci�c governance interventions.

POLITY IV dataset Quality of Governance Dataset, World Freedom Atlas.

41 examples of governance related RCTs catalogued in Moehler, 2010.

2) Governments and CSOs in developing countries

3) Researchers

4.1 Country rankings, naming and shaming exercises.

Ease of communication Cross-country comparability. Speci�city to particular human rights abuses and constraints on freedom.

Transparency International Corruption Perceptions Index Press Freedom Index Human Development Index

5.1 Supporting evidence for journalism

Credibility of sources, ease of interpretation and communi-cation.

Source: “What Makes a Good Governance Indicators?” by Gareth Williams, Policy Practice Brief 6, January 2011

Issue-speci�c indicators:CPI: Transparency International CorruptionPerceptions Index, etc.RGI: Resource Governance Index

National and sub-national indica-tors:IGI: Indonesia Governance Index, etc.

4) International advocacy

5) Media

6.1 Assessing political risk6.2 Understanding the costs of doing business in di�erent countries

Cross-country comparabilityCross-country comparability. Disaggregation by di�erent investment climate constraints.

International Country Risk Guide (ICRG) ratings

IFC Doing Business Survey

6) International investors

59

Table 2. List of Governance Indicators and Online Resources

AfrobarometerCingranelli-Richards Human Rights DatasetCountry Policy and Institutional Assessment Doing BusinessDatabase of Political InstitutionsGlobal Insight DRIEconomist Intelligence UnitFreedom HouseGlobal Competitiveness SurveyGlobal Integrity IndexGallup World PollHeritage FoundationInvestment Climate AssessmentLatinobarometroIbrahim Index of African GovernanceOpen Budget IndexPolity IVPolitical Risk ServicesPublic Expenditure and Financial AccountabilityReporters without BordersWorld Competitiveness YearbookResource Governance Index (RGI)Etc.

Indonesia Governance Index (IGI)Good Governance Indicator (GGI) - PhilippineEtc.

AnnualTriennial

TrennialAnnualAnnualAnnualAnnualQuarterlyQuarterlyAnnualAnnualTriennialAnnualAnnualIrregularAnnualTriennialAnnualAnnualMonthlyIrregularAnnualAnnualAnnual

www.afrobarometer.orgwww.humanrightsdata.comwww.worldbank.orgwww.doingbusiness.orghttp://econ.worldbank.orgwww.globalinsight.comwww.eiu.comwww.freedomhouse.orgwww.weforum.orgwww.globalintegrity.orgwww.gallupworldpoll.comwww.heritage.orgwww.investmentclimate.orgwww.latinobarometro.orgwww.moibrahimfoundation.orgwww.openbudgetindex.orgwww.cidcm.umd.edu/polity/www.prsgroup.comwww.pefa.orgwww.rsf.orgwww.imd.chwww.revenuewatch.org/rgi

www.kemitraan.or.id/igiwww.nsbc.gov.ph/ggi

181921361751781171201921174113116194174859161140421654758

Additional online source:“Governance Indicators: A User’s Guide”, 2nd edition, UNDPhttp://www.undp.org/content/dam/aplaws/publication/en/publications/democratic-governance/oslo-governance-center/governance-assessments/governance-indicators-2nd-edition/governace_indicator_undp_users_guide_online_version.pdf

Source: Governance Indicators: Adapted from “Where Are We, Where Should We Be Going?” By Daniel Kaufmann and Aart Kraay, January 2008

Name

National and sub-national indicators

N u m b e r o f c o u n t r i e s covered

Frequency of Surveys Website

19: Kaufmann, Daniel and Kraay, Aart, ““Where Are We, Where Should We Be Going?” MPRA Paper No. 8212, posted 10 April 2008, http://mpra.ub.uni-muenchen.de/8212/1/MPRA_paper_8212.pdf

19

The Golden Rules of Evaluation:

Knowledge on evaluation methods. Most of the evaluation activities are done by external evaluators, thus, in practice; we unintentionally “underestimate” the importance of having internal knowledge in robust evaluation method. Evaluation is not designed well since the beginning project planning phase, thus budgets are not prepared in accordance with the evaluation we need. Evaluation’s Terms of Reference often too general and evaluation process as well as methods are handed over to the hands of external evaluators.. Donor-funded evaluation provides an opportunity for us, organizations, to utilize it for learning, however, to be able to do it, we have to have su�cient knowledge on robust evaluation methods.

The willingness to learn and unlearn. The problem is not what we don’t know and think we need to know – it’s also what we feel that we already know or what we are so used to doing that we need to unlearn. Sometimes we are stuck in our organization’s habits and practices, doing things in ways that used to work, things that we have got the skills for and that we are experienced in, even if they are no longer needed. The willingness to learn and unlearn - or recognize we did something wrong and need to change it – should be the foundation of evaluation.

Ensure that external accountability purposes do not outweigh the poten-tial to use this evidence for learning and improvement. It is, of course, important for us to �nd ways of demonstrating our results to the public, donors and the people we work with, and as we seek to learn more about what works, where and how. Even a large NGO has serious limits to the capacity & resources available for impact evalua-tion. Thus, we need to decide on the level of investment and e�ort internally in our organization to assess the impact of our work.

Share the learning result and feed it back to the planning. Not only the evaluation is not designed by engaging our own sta� (particularly the �eld sta�) and/or our partners, the result of evaluation is rarely shared to them. This practice will only strengthen the notion that evaluation is merely an obligation to satisfy the donor and none of our sta� or partners will feel the added value of such activity, moreover, feed it back to future project or program planning.

60

B. Tackling the Evaluation Challenge

There are di�erent evaluation tools, methods and approaches developed and used by the di�erent organizations. The methods re�ected in the picture above do not cover all available methods. More information on di�erent evaluation tools, methods and approaches can be accessed online, such as to betterevaluation.org.

61

Get to know Various Evaluation Tools, Methods & Approaches!

How Can Governance Programs be Designed that Rigorous ImpactEvaluation Becomes Possible?

62

Include evaluator at the outset. The presence of an evaluator in the early phases of program design can signi�cantly increase the potential of conducting rigorous impact assessment. Evaluations of new program call for the evaluator to be involved in all aspects of the program - including observations on changes in interventions and potential outcomes. This makes it easier to identify components of the program that can be evaluated quantitatively. Programs are more di�cult to evaluate when they are being run or have been completed, since the evaluator’s assessment is limited to project reports and interviews with donor sta�. In addition, the initial indicators and data collected may no longer be appropriate at the time of evaluation. More impor-tantly, the evaluator’s knowledge and hands-on expertise regarding the dynamics of the intervention and the characteristics of the target population may be limited if the evaluation is conducted under severe time constraints. For practitioners, this means obtaining advice from technically skilled evaluators right from the start of the program.

However, including evaluators at the outset is easier said than done. In most cases, evaluators are consultants hired externally by donor agencies, and keeping them on for the entire duration of the program is not economical. An alternative approach is to include the evaluator as a member of an advisory committee before project imple-mentation so that appropriate outcome measurements and data collected have better chance of remaining valid for the evaluation design. For instance, the World Bank’s Development Impact Evaluation (DMIE) Initiative provides support teams of experts and research groups to help government counterparts to design and carry out impact evaluations.

Collect good-quality baseline data. There is a di�erence between collecting base-line data and quality baseline data. Quality baseline data comes from careful planning of the program. Data are collected from the relevant bene�ciaries of speci�c interven-tions as well as from stakeholders. Data should include both initial characteristics of the target groups and expected outcomes. This is typically di�cult to plan ex ante as some indirect outcomes are revealed only during or after implementation.

Identify components of the program that permit rigorous impact evaluation. At the outset, it is important to identify potential components of the program to which quantitative analysis can be applied. Early identi�cation means that the best possible method can be used for the evaluation and that the required data can already be gathered.

Tackling the di�culties in evaluating governance intervention :

Small sample size. A common problem encountered in the evaluation of governance interventions (especially at higher levels of government) in that the units of observa-tion are often too few for the assumptions associated with parametric statistics to hold. In other words, the accuracy of inferences based on the methods described hinges on statistical “laws” that require a large sample size.

First, in the cases where the sample has moderate size (say 20 observations), the importance of obtaining baseline data must be emphasized in this context. Remem-ber that when randomizing, as sample size increases, the di�erent characteristics between the treatment and control groups would cancel out and the remaining di�erence would only be the intervention. If sample size is of moderate size, this means that the comparison group may not be very similar to the treatment. Thus, the more pre- and post-treatment data available, the more credible the impact estimate could be. Fewer observations also imply that collecting more than just one pre-treatment set of data is logistically easier than where samples are larger.

Second, if randomization is possible, a technique known as ‘repeated measures’ can sometimes be used. ‘Repeated measures’ means that baseline data and post-treatment outcomes will be collected repeatedly for each unit over a period of time. Imagine a country with ten provinces in which a program is to be implemented. After the baseline data have been obtained, the program can be gradually “phased in” in at least one of the provinces for a certain period (let’s say six months), with the others used as controls (NRC 2008). Repeating this process until only one control observation is left will produce a total of 10 pre- and at least 10 post-treatment controls. Conside-ring the uniformity of the treatment e�ects across provinces already nables the reliability of the impact estimate to be gauged. Although the estimated e�ect may still be biased because some confounding factor has not been taken into account, it may nevertheless allow some inference as to the e�ectiveness of the program.

Third, in the case of very small sample size, the evaluation needs to conform to a more qualitative approach. In case of non-random assignment of treatment, instead of using propensity score in matching of the control or treatment groups, one can match qualitatively as accurately as possible the key factors that are important in a�ecting outcome. Again, collection of baseline data is highly advisable.

63

20: “Micro-Methods in Evaluating Governance Intervention”

C. Methodological Challenges in Evaluating Governance Intervention

16

No baseline data. Collecting baseline data or pre-intervention information is a chal-lenging task for development practitioners, since (i) interventions may tend to change over time; (ii) all potential outcomes – expected and unexpected - must be taken into account at the outset; (iii) collecting baseline data requires additional �nancial resources; (iv) the implementation of the project could be delayed. Those who are to collect baseline data must have an overall idea about how the project will be evalu-ated later on. Otherwise, the baseline data collected may be insu�cient or no longer relevant at the time of evaluation.

If baseline data are not available, using a triple-di�erence (DDD) method is an option under certain conditions. Ravallion et al. (2005) adopted this approach for the Trabajar program in Argentina, which provides work for the unemployed poor for six months. They examined the di�erence in the incomes of participants who had left and partici-pants who had remained in the program. A simple DD between “stayers” and “leavers” would lead to biased results, since work opportunities are not the same for both groups. Ravallion et al. therefore formed an entirely separate control group who had never participated in the program. DDD is the DD of stayers with matched non-participants minus the DD of leavers with matched non-participants. The result is the income gained by stayers for participating in the program. DDD techniquehas allowed changes in the economy or labor market to be controlled for.

Another possible approach if no baseline data are available is to reconstruct a baseline in retrospect, a practice common to many clinical studies. However, the accuracy of recall, i.e. collecting data from individuals’ recollections of what happened a year or more earlier, is a serious problem in this type of study.

As discussed above, some quasi-experimental techniques, namely single di�erence (control versus treatment group) approaches such as PSM or instrumental variable techniques, can be applied even in the absence of baseline data.

Complex governance interventions. Sometimes policymakers are interested in not one but a range of policy interventions. In conducting RCTs, one or a combination of treatments can be accommodated with factorial design. This means that various treat-ments can be tested in a single experiment, thus reducing evaluation costs. Factorial design can also investigate the interaction e�ects of treatment. However, sample size must be adjusted signi�cantly, depending on the number of interventions to be examined. An example of this factorial design is taken from the study of corruption in the issue of driving licenses discussed earlier, which was conducted by Bertrand et al. (2007). They recruited and randomly assigned applicants to three groups: (i) a com-parison group (ii) applicants o�ered cash bonuses if they could obtain their licenses within 32 days and (iii) a group o�ered free driving lessons. This allowed the authors to make comparative assessments of the various types of intervention.

64

16

Measuring outcome. An important obstacle encountered in the evaluation of gov-ernance interventions is quantifying outcome that is largely characterized by human behavior and di�cult to pin down. Unlike some development programs, there is no easy or straightforward measure of governance outcome. For impact studies in the traditional sectors, the measures of outcomes can be de�ned and is not as controver-sial. For instance, the e�ects of education on labor are captured by wages and employment; nutritional improvements are measured by such anthropometric indi-cators as weight, height and body mass index; and the e�ectiveness of fertilizers is measured by farm yield. But what is the best way to capture a decline in corruption? Or an increase in civic participation? Or an improvement in law enforcement?

It is important to acknowledge that governance outcomes cannot be measured with a single, all-encompassing indicator. The practical approach would be to unpack the intervention and assess which of its dimensions can be rigorous analyzed. Although measuring outcome is a challenging task, some authors have been very creative in solving this problem. A very good example is that randomized experiment conducted by Olken (2007), which was used in the context of a road project. The project interven-tions include (i) increasing government audits and (ii) grassroots participation. To measure the decrease in corruption, instead of using a perception-based corruption measure as the outcome variable, he devised a direct measure of corruption: the di�erence between reported expenditure and independent engineer’s estimate.

65

16

Using Experimental Methods . Experimental methods are often called the “gold standard” because they establish attribution, or a direct causal link between the inter-vention (project, program or policy X) and the outcome (change in Y) by eliminating the possibility that other factors contributed, since changes in outcome can be due to a multiplicity of factors. In addition, experimental methods allow evaluators to estab-lish a testable counterfactual and compare it to what did happen, allowing impact to be calculated from the di�erence between the two outcomes.

Randomized control trials (RCTs) represent a key method for combining these two approaches. Essentially, participants are randomly assigned from the entire (statistically signi�cant) population; for example, of all the prosecutors working in Bulgaria, a certain amount is chosen. Then within this group, those who receive the treatment are randomly assigned – some prosecutors receive the training and some do not. Baseline data, such as a survey, is drawn from each group before they receive the treatment and again after the treatment. With this information, impact can be calculated by a means to test or regression framework.

66

21: Gauck, Jennifer, “Measuring Impact on the Immeasurable?Methodological Challenges in Evaluating Democracy and Governance Aid”, Paper prepared for the 2011 Annual Meeting of the American Political Science AssociationSeattle, September 4, 2011 22: Garcia, M. (2011): Micro-Methods in Evaluating Governance Intervention. Evaluation Working Papers. Bonn: Bundersministerium fur wirtschaftliche Zusammernabeit und Entwicklung. Available online at: http://www.bmz.de/en/what_we_do/approaches/evaluation/Evaluation/methods/index.html

D. Rigorous Evaluation of Governance Intervention: Is it Possible?

1621

22Box 12. Olken’s Study on Corruption in Road Infrastructure Projects in Indonesia

The most oft-cited example of applying the RCT method to a D&G outcome is Olken’s (2007) study of 600-plus villages in Indonesia that received funds from the World Bank for road infrastructure projects. Olken tested the e�ect of di�erent treatments on corruption in the projects which he measured as the di�erence between o�cially reported expendi-ture and an independent engineer’s estimate.

Six hundred and eight villages were randomly selected and divided into four groups. Olken informed the selected villages in the �rst group that the road would be audited by central government. In the second group, he organized village-level accountability meetings. In the third group, he again organized accountability meetings and, in addition, anonymous survey forms were distributed to villagers. The fourth group served as the control group.

67

23 : Ibid

23Box 13. The Evaluation of Impact of international

Election Observers on Election Quality in Indonesia

Since it is di�cult to capture corrupt practices, he constructed a simple measure of corrup-tion. Instead of using perception-based corruption measure, he devised a direct measure of corruption as the di�erence between reported expenditure and the independent engi-neers’ estimate. The independent estimate was derived from the samples taken by the engineers after project completion to measure the quantity of materials used from inter-views with villagers to determine the wages paid and from a survey of suppliers carried out to estimate prices.

He found that increasing government audits had a positive e�ect by reducing corruption. “Missing expenditures” feel by eight percentage points when the villages were audited by central government. He also found that grassroots participation in monitoring had no impact. Rapid assessments and M&E approaches to evaluation may lead to a di�erent conclusion. Olken’s result is convincing because, through randomization, he ensured that the only di�erence between the treatment and control groups was the intervention.

Source: Garcia, M. (2011): Micro-Methods in Evaluating Governance Intervention. Evaluation Working

This example shows how RCTs can be used to evaluate interventions in democracy promotion. Democratic processes in developing countries are weak because some voters are less well informed or because of elite capture. Elections often feature ballot fraud, violence and vote-buying. To minimize this, donors have deployed interna-tional observers to new democracies to monitor the election process. While some argue that international

To capture the impact of international election observers on election quality, Hyde (2010) randomly assigned international observers during the 2004 presidential elec-tion in Indonesia to villages identi�able on a local map, in each of which they visited one to four polling stations. The random assignment generated two groups of villages, those that were observed and those that were not. Her results show that, as international observers tend to increase the total number of votes cast for the incum-bent, they may change voters’ election-day behavior disproportionately. She also found that election o�cials tend to take greater care to comply with election regula-tions if international observers are present.

68

However, RCTs method largely focuses on interventions at village or community level, mainly because RCT requires large samples. Thus, RCTs or randomization method might not be suitable for complex program with more than a single intervention.

So to say that this is the biggest drawback of conducting experimental technique, even though as we can see that IT IS POSSIBLE to use this technique to produce a rigorous evaluation for democracy and governance intervention, albeit limited to single intervention at village or community level.

Other considerations in using the experimental method include: 1) the time taken to prepare for evaluation before program’s roll-out – which will be very time consuming particularly for big and complex partnership program; 2) the unwillingness of indi-viduals to participate; 3) the complexity of intervention; and 4) institutional factors and costs. Needless to say, the rule of thumb is to randomize whenever possible.

In recent years, there has been increasing use of RCTs to test the impact of di�erent programs on D&G-related outcomes. The examples described here show that RCTs method is very useful in providing new evidence that refutes what was initially hypothesized at the programming stageor in other words, testing our intervention. In the case of corruption, the implicit assumption was that grassroots monitoring was central in eliminating local corruption. The evidence challenges this hypothesis and encourages further evaluation to see if the �ndings are similar in di�erent settings.

A broader message conveyed by the examples concerns the feasibility of applying RCTs in evaluating such di�cult concepts as corruption and democracy promotion, rather than the success of the intervention itself. These examples have proved that the method is not only theoretically appealing, but can also be managed in the �eld.

As it is also very simple to understand, the RCTs method is becoming increas-ingly popular in social policy. Moehler (2010) has identi�ed 41 RCT studies on democracy and governance that used randomized �eld experiments. She found that substantial work has been done on elections, community-driven development and improved public service delivery.

69

The four quasi-experimental methods commonly cited in impact evaluation studies are di�erence-in-di�erence, propensity score matching, instrumental variable and regression discontinuity design.

An example of PSM application has been done to determine whether transparency of government performance motivates people to participate or engage in civic activi-ties. In many countries, donors actively participated in developing performance benchmarks for local government at the height of decentralization. One of the most popular is the Citizens Report Card in India, Bangladesh and Mexico. In the Philip-pines, the Report Card Survey is just one of 30 di�erent performance assessment systems funded by donor agencies. Most of these systems are aimed at improving public service delivery and welfare. However, public service delivery can be improved only if people become more active members of the community. Anecdotal evidence has provided some clues to the e�ectiveness of these measures. But a causal link between information on local government performance and civic participation is yet to be established.

Box 14. The e�ect of information on the performance of local government o�cials on people’s participation during decentralization in the Philippines

Table 3. Four Quasi-experimental Methods

This double-di�erence approach calculates impact by utilizing information before –and after the intervention and calculating the change in outcome over the two periods between the control and treatment group.

To minimize confounding factors due to the non-random assignment of the intervention, matching requires �nding a comparison group that matches the characteristics of the treatment group.

The observable characteristics are used to generate a propensity score, which is the probability of participation. Each treated individual is matched to a non-participant on the basis of this propensity score.

RDD requires a speci�c eligibility rule in the targeting of participants for the program. The degree to which the intervention changes the outcome of the treatment group compared to non-participants near the eligibility’s cut-o� is the impact.

In simple terms, this approach involves identifying a special variable that a�ects outcome and intervention, but without the two having any causal-ity. IV approach is needed if the causal relationship between intervention and outcome run the opposite direction. Such a scenario occurs when (i) the intervention has been deliberately targeted or (ii) participants have joined the program for speci�c reasons. This implies that potential unobserved characteristics or omitted variables have not been taken into account.

Di�erence-in-di�erence (DID)

Propensity score matching (PSM):

Regression discontinuity design (RDD):

Instrumental variable (IV)

Quasi-Experimental Methods

70

Capuno and Garcia (2010) investigate the e�ect of information concerning the perfor-mance of local government o�cials on people’s participation during decentralization in the Philippines. A local governance performance index was introduced in 12 cities and municipalities. At the eight treatment sites, the index scores were announced by means of public presentations, posters, magazines and stickers, while the index scores were not announced at the four control sites. Since not all individuals in the treatment municipalities receive treatment, there is reason to suspect selection bias. Propensity score matching is therefore used to match individuals who actually received treat-ment (in the treatment municipalities) with those in the control municipalities.

The data were derived from three rounds of random household surveys conducted at all the pilot sites. At the eight treatment sites, 178 individuals reported exposure to the treatment – 95 in the 2002 mid-pilot survey and 83 in the 2003 post-pilot survey. The treated individuals were those who had read the magazines, seen the posters or attended the public presentation. There were three comparison groups: (i) 400 indi-viduals from the four control municipalities in 2002, (ii) 400 individuals from the four control municipalities in 2003 and (iii) 800 individuals in the two years combined. A number of individual and village-level characteristics, including gender, age, school-ing, income, family size, employment and civil status, were used to determine partici-pation. For each comparison group, the result shows that information on local govern-ment performance leads to increased civic participation.

Source: Garcia, M. (2011): Micro-Methods in Evaluating Governance Intervention. Evaluation Working

71

The quasi-experimental methods try to �ll this di�culty in experimental methods, where the treatment and control groups need to be similar in all other ways, by constructing a counterfactual via methods that make assumptions about the data that are not directly testable. In some ways, it manages to address this particular chal-lenge, however, it is still very di�cult to apply this method in such a big partner-ship program with complex intervention, especially with regional or interna-tional scope. The previous section considered the ideal situation where sample sizes are large, no spillovers occur, the unit of analysis in most cases is the individual, the interventions are clear-cut and can be manipulated, outcome measure is straightfor-ward, and there is room to collect control groups. But what happens if this is not the condition?

It is still very possible to develop evaluation for governance intervention in a partner-ship program with multiple issues. Nevertheless, there are pre-requisites that must be satis�ed:

#1 Design and consolidate the M&E since the beginning of the intervention or program development phase. Donor or aid agencies need to lead and facilitate this process since the beginning.

#2 Engage the partners in designing the M&E approach and tools. The partnership might decide to adopt an approach or develop a distinct hybrid approach that matches their needs. Either way, the approach and tool should be resulted from intensive discussion among the partners and agreed upon.

#3 Involve the expert. As we can see, there are numerous tools for M&E with its own complexities. Designing and developing the M&E for partnership program require assistance from experts in M&E �eld to help the partnership learn and decide which approach and tool that they want to adopt.

Other Evaluation Approach?

72

A practical Guide to Advocacy Evaluation, Innovation Network, Inc

AWID’s Evaluation Strategy: Approach, Framework, and Mechanisms. (2006 –2010)”, accessed at: http://awidme.pbworks.com/w/page/36200176/AWID%20ME%20Framework

Batlilawa, Srilatha, “Strengthening Monitoring & Evaluation for Women’s Rights: Twelve Insights for Donors” for AWID (Association for Women’s Rights in Develop-ment)

Davies, Rick and Dart, Jess, “The Most Signi�cant Technique (MSC): A Guide to Its Use, by Rick Davies and Jess Dart, funded by CARE International, United Kingdom;Oxfam Community Aid Abroad, Australia | Learning to Learn, Government of South Australia; Oxfam New Zealand | Christian Aid, United Kingdom | Exchange, United Kingdom; Ibis, Denmark | Mellemfolkeligt Samvirke (MS), Denmark; Lutheran World Relief, United States of America, April 2005

Garcia, M. (2011): Micro-Methods in Evaluating Governance Intervention. Evaluation Working Papers. Bonn: Bundersministerium fur wirtschaftliche Zusammernabeit und Entwicklung. Available online at: http://www.bmz.de/en/what_we_do/approaches/evaluation/Evaluation/methods/index.html

Gauck, Jennifer, “Measuring Impact on the Immeasurable? Methodological Chal-lenges in Evaluating Democracy and Governance Aid”, Paper prepared for the 2011 Annual Meeting of the American Political Science Association Seattle, September 4, 2011

Handbook of Democracy & Governance Program Indicator, Center for Democracy and Governance Bureau for Global Programs, Field Support, and ResearchU.S. Agency for International Development - Washington, D.C, August 1998

HIVOS. 2009. “Working with a Theory of Change in Complex Change Processes. An Introduction

Hunter, Justine,“Monitoring and Evaluation: Are we making a di�erence?”, Namibia Institute for Democracy, 2009

References

73

In�uential Evaluations: Evaluations that improve performance and Impacts of Devel-opment Program, The World Bank 2004

Institutionalizing Impact Evaluation Within The Framework of A Monitoring & Evalua-tion System; Poverty Analysis, Monitoring, and Impact Evaluation Thematic Group - PREM Network, The World Bank, 2009

James, Cathy, “Theory of Change Review: A report commissioned by Comic Relief”, September 2011, http://dmeforpeace.org/sites/default/�les/James_ToC.pdf

JISC: Six Steps to E�ective Evaluation: A Handbook for Programme and Project Managers,Glena�ric Ltd (2007) SixSteps to E�ective Evaluation: A handbook for programme and project managers

Kaufmann, Daniel and Kraay, Aart, ““Where Are We, Where Should We Be Going?” MPRA Paper No. 8212, posted 10 April 2008, http://mpra.ub.uni-muenchen.de/8212/1/MPRA_paper_8212.pdf

Lessons Learned IV Report - IKAT US Partnership , Lenny Hidayat

Monitoring & Evaluation: Some Tools, Methods & Approaches, the World Bank, 2004

Nyanggaga, Julius, Development in Practice, “Research that matters: outcome map-ping for linking knowledge to poverty reduction actions” by Julius Nyangaga, Terry Smutylo, Dannie Romney & Patti Kristjanson, 15 October 2010

Outcome Mapping: A Basic Introduction” by Research to Action, http://www.researchtoaction.org/2012/01/outcome-mapping-a-basic-introduction/

Outcome Mapping and the Logical Framework Approach: Can they share a space?

Review of OXFAM GB’s quantitative methodology for its e�ectiveness review, Ipsos MORI, 2012

Savedo�, William, “Impact Evaluations Everywhere: What a Small NGO to do?”, for Center for Global Development, 17 Sept 2012

“Storytelling” http://evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=61&Itemid=157

74

Successful Partnership: A Guide” by OECD LEED Forum on Partnership and Local Gov-ernance

Theory of Change: A thinking and action approach to navigate in the complexity ofsocial change processes, HIVOS, 2012

Vogel, Isabel, “Review of the Use of “Theory of Change” in International Development”, UK Department of International Development, April 2012

Online resources:

http://policy-practice.oxfam.org.uk/blog/2013/02/lessons-learned-from-a-year-of-advocacy-impact-evaluation

https://civicus.org/view/media/Monitoring%20and%20Evaluation.pdf

http://policy-practice.oxfam.org.uk/blog/2012/10/e�ectiveness-reviews

http://policy-practice.oxfam.org.uk/blog/2013/02/lessons-learned-from-a-year-of-advocacy-impact-evaluation

http://evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=51&Itemid=131

http://evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=51&Itemid=131

www.betterevaluation.org

http://www.transparency.org/cpi2012/in_detail#myAnchor2

http://www.knowledgebrokersforum.org/wiki/item/theory-of-change-vs-logframeshttp://mande.co.uk/2008/lists/the-logical-framework-a-list-of-useful-documents/http://repub.eur.nl/res/pub/19048/wp303.pdfhttp://mande.co.uk/2011/uncategorized/theory-of-change-a-thiking-and-action-approach-to-navigate-in-the-complexity-of-social-change-processes/http://monitoring.cpwf.info/background/theory-of-changehttp://monitoring.cpwf.info/m-e-tools-and-workbook/project-workbook/outcome-pathways-and-outcome-logic-modelhttp://monitoring.cpwf.info/m-e-tools-and-workbook/project-workbook/outcome-pathways-and-outcome-logic-model/example-of-a-�lled-in-olm

75

https://docs.google.com/viewer?a=v&pid=sites&srcid=Y3B3Zi5pbmZvfG0tZS1ndWlkZXxneDo3ZDQzMjQzOTdlZmM1ODkz

http://www.slideshare.net/ikmediaries/theory-of-change-and-outcome-mapping-for-intermediary-workhttp://www.aecf.org/upload/publication�les/cc2977k440.pdf

http://www.organizationalresearch.com/publicationsandresources/advocacy_and_policy_change_evluation_a_primer.pdf

http://www.researchtoaction.org/2011/05/theory-of-change-useful-resources/

http://www.organizationalresearch.com/publicationsandresources/pathways_for_change_6_theories_about_how_policy_change_happens.pdf

http://www.accountability.or.tz/wp-content/uploads/2013/03/Update-on-Think-Piece-Final.pdf

http://www.theoryofchange.org/wp--content/uploads/toco_library/pdf/2001-Earl-Book-Outcome-mapping-building-learning.pd

http://www.outcomemapping.ca/forum/�les/OM-LFA_DRAFT_165.pdf

http://feedthefuture.gov/sites/default/�les/resource/�les/ftf_volume1_monitoringevaloverview_feb2012.pdf

http://ec.europa.eu/dgs/information_society/evaluation/data/pdf/lib_master/oecd_01e91637_improving_evaluation_practices.pdf

http://betterevaluation.org/approaches

http://www.jrmckenzie.org.nz/sites/default/�les/attachments/Review%20of%20Useful%20Resources%20for%20Advocacy%20Evaluation.pdf

http://www.innonet.org/resources/�les/path�nder_advocate_web.pdf

http://www.aspeninstitute.org/policy-work/apep/tools

http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/advocacy-and-policy-change/a-guide-to-measuring-advocacy-and-policy

http://www.managingforimpact.org/sites/default/�les/resource/research_and_advocacy_for_policy_change__measuring_progress.pdf

http://www.gsdrc.org/go/topic-guides/measuring-results/applying-monitoring-and-evaluation-tools#pol

http://www.undp.org/content/dam/aplaws/publication/en/publications/democratic-governance/oslo-governance-center/governance-assessments/governance-indicators-2nd-edition/governace_indicator_undp_users_guide_online_version.pdf

This module is prepared and compiled by:IKAT US Component 2 Team

For further inquiries, please contact:

Siti Ismah Afwan / siti.ismah@kemitraan.or.idLenny Hidayat / ananya_lenny@yahoo.comRuth Alicia / alicia.ruth@gmail.com

Kemitraan O�ceJl. Wolter Monginsidi No. 3, Kebayoran Baru,Jakarta Selatan 12110 - INDONESIAPhone. +62 21 727 99566Fax. +62 21 720 5260, 720 4916Email. info@kemitraan.co.id www.kemitraan.or.id

76