Measuring Effectiveness, MelbourMeasuring Effectiveness, Melbourne, Sept 2007ne, Sept 2007

11

Measuring effectivenessMeasuring effectiveness

a network perspective…a network perspective…

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 22

The questions asked…The questions asked…

►How we assess relationships How we assess relationships ►How we need to change as individuals and How we need to change as individuals and

organisations to be open to feedback organisations to be open to feedback ►How we empower communities to hold us How we empower communities to hold us

and others to account and others to account

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 33

Why bother?Why bother? People, and their relationships with each other, are the means and end of People, and their relationships with each other, are the means and end of

developmentdevelopment

“…“…if donors are to achieve their aims of contributing to the reduction of global poverty if donors are to achieve their aims of contributing to the reduction of global poverty they need to invest as much or more time in managing relationships as they currently they need to invest as much or more time in managing relationships as they currently spend in managing their money”spend in managing their money”[1][1]

[1][1] “Relationships Matter For Supporting Change In Favour Of Poor People” R. Eyben 2004. IDS“Relationships Matter For Supporting Change In Favour Of Poor People” R. Eyben 2004. IDS

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 44

Three approachesThree approaches

►Analysis of Analysis of individual individual relationships relationships ► Most commonMost common

►Analysis of Analysis of portfoliosportfolios of relationships of relationships► Not so commonNot so common

►Analysis of Analysis of networks networks of relationshipsof relationships► Less commonLess common

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 55

1. Analysis of 1. Analysis of individual individual relationships relationships

► ““Autonomy or Dependence: Case Studies of Autonomy or Dependence: Case Studies of North-South NGO PartnershipsNorth-South NGO Partnerships” ” Brehm et al 2004Brehm et al 2004

► ““This book explores the concept and practice of This book explores the concept and practice of ‘partnership’ between non-government ‘partnership’ between non-government organisations (NGOs) in the North and South. organisations (NGOs) in the North and South. Based on a rigorous four-year study, the book Based on a rigorous four-year study, the book draws together the perspectives of a group of draws together the perspectives of a group of European NGOs and compares these with the European NGOs and compares these with the experiences of a selection of their partners in experiences of a selection of their partners in Brazil, Cambodia and Tanzania”Brazil, Cambodia and Tanzania”

► Oxfam GB in Partnership: A Global Strategic Oxfam GB in Partnership: A Global Strategic Evaluation, Evaluation, 20072007

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 66

2. Analysis of 2. Analysis of portfoliosportfolios of relationships of relationships

►Example from Bangladesh 1994Example from Bangladesh 1994► Result of a card sorting exerciseResult of a card sorting exercise

►CAA Country Programme DirectorCAA Country Programme Director► Method at Method at http://www.http://www.mandemande.co..co.ukuk/docs/hierarch./docs/hierarch.htmhtm

► Identified a nested set of “Most Significant Identified a nested set of “Most Significant Differences” between 12 partnersDifferences” between 12 partners

► Strategy can then expressed as choices Strategy can then expressed as choices between types of partnersbetween types of partners► later performance can be compared to strategylater performance can be compared to strategy

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 77

A nested classificationA nested classificationOther rankings

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 88

3. Analysing 3. Analysing networks networks of relationshipsof relationships

► As well as looking at the portfolio of actors we As well as looking at the portfolio of actors we are working with, we also need to consider the are working with, we also need to consider the relationships between relationships between these actors, these actors,

► and their relationships with othersand their relationships with others

► As we increase the number of actors, the As we increase the number of actors, the number of possible relationships between them number of possible relationships between them increases exponentially i.e. A LOTincreases exponentially i.e. A LOT

► In order to describe, or plan, what is happening In order to describe, or plan, what is happening with these relationships, we need to move from with these relationships, we need to move from a hierarchical view to a network viewa hierarchical view to a network view

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 99

One view of 16 NGOs in GhanaOne view of 16 NGOs in GhanaA

BA

NT

U

AR

K

AS

DR

CD

D

CE

PA

FID

A

FO

SD

A

IDE

G

IEA

ISO

DE

C

ISS

ER

NG

ND

TU

C

TW

N

WA

NE

P

WIL

DA

F

ABANTU 0 1 0 0 1 0 4 0 4 1 0 0 0 0 0 3

ARK 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 1

ASDR 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

CDD 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 5

CEPA 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0

FIDA 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1

FOSDA 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

IDEG 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1

IEA 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ISODEC 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 2

ISSER 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

NGND 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

TUC 0 0 0 0 0 0 0 0 1 2 0 0 0 1 0 0

TWN 1 0 0 0 0 0 0 0 2 2 0 0 0 0 0 0

WANEP 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2

WILDAF 0 1 0 2 1 0 0 0 1 0 0 0 0 0 0 1

In social network analysis, this is a “one-mode” network, showing …

Actors x Actors

Cell values = frequency of references to each other, in their Progress Reports between 2005-6

Cells can represent many other aspects of their relationships

Matrices can hold data on thousands of relationships

Up to 120 here

But it is hard to read.

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1010

The same data, presented in network diagram form..

makes many features of the relationships more visibleWeak links

Strong links

Reciprocated links

Well connected actors

Unconnected and marginalised actors

Groups of actors

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1111

Of course the picture can get much more complex.

Here we have all the other working relationships they reported for the 2005/6 period

NGOs

Shared links

Unique links

270 actors in all

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1212

What should we be assessing?What should we be assessing?

► Actors have their own strategies. When Actors have their own strategies. When implemented via relationships with others, these implemented via relationships with others, these strategies connect up, to form strategies connect up, to form a larger structurea larger structure, , which no single actor designed or controlswhich no single actor designed or controls

► Emergent structure = collective strategyEmergent structure = collective strategy

► But the big question is…But the big question is…► How functional is that emergent structure & strategy?How functional is that emergent structure & strategy?

►And how do we ask about this?And how do we ask about this?► Are there standard questions and different “right answers”Are there standard questions and different “right answers”

►According to specific contextsAccording to specific contexts

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1313

Emergence? One view…Emergence? One view…

Complex structuresComplex structures that arise from the interactions of many that arise from the interactions of many

actors, actors, who are following local rules who are following local rules

Their properties not predictable from Their properties not predictable from knowledge of the parts knowledge of the parts Weak form: Weak form: If you are one of the actorsIf you are one of the actors Strong formStrong form: Even if you know what all the : Even if you know what all the

actors are doing (because they react to each actors are doing (because they react to each others reactions)others reactions)

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1414

Some network perspectives and Some network perspectives and questionsquestions

Matrix viewMatrix view One mode: G-rap NGOs, IMHI indicatorsOne mode: G-rap NGOs, IMHI indicators Two mode: Molisa, Vietnam; Oxfams, LaosTwo mode: Molisa, Vietnam; Oxfams, Laos

Diagram viewDiagram view One mode: Ghana workshop participantsOne mode: Ghana workshop participants Two mode: M4P Workshop, OI internal groupsTwo mode: M4P Workshop, OI internal groups

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1515

Matrix view:Matrix view: Ministry of Labour…, Vietnam Ministry of Labour…, Vietnam

Ministry ObjectivesWeighting a b c d e f g h i j # Sum

123 20 5 60 15 4 100

Ministry 4Programs 5

6 5 50 25 5 15 5 100789

10

# 2 1 1 1 2 2

Sum 25 50 5 25 65 30

Vs. Expected

e.g.

bud

get a

lloca

tions

1. List existing programs

2. List future goals

3. Rank expected impact of a program on each objective

4. View summary rows

5. Discuss acceptability of overall result

6. Then what to change

(a two mode network)

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1616

22ndnd matrix view: Oxfam Int. in Laos matrix view: Oxfam Int. in Laos

Oxfam affiliate priorities in LaosOXFAM INTERNATIONAL STRATEGIC PLAN 2007–2012 OAUS OHK OQ OA OSB ON All

Economic Justice 0

1.1 make agriculture work 40 40 80

1.2 achieve fairer trade rules 60 20 60 10 150

1.3 reduce the impact of climate change and energy shocks 10 10

Essential services

2.1 demand that national governments fulfill their responsibilities 30 10 10 30 80

2.2 support civil society organizations and alliances 10 50 60

2.3ensure better policies and more funding from rich countries and international institutions, 60 60

Rights in crisis

3.1 improve our ability to deliver better protection and greater assistance, 10 10

3.2 change policies and practices of the international humanitarian system 40 40

3.3 work within the framework of human security, 0

Gender justice

4.1 support women’s leadership 10 10 20

4.2 work to end gender-based violence 90 904.3 strengthen Oxfam’s own learning and capacities on gender 0

100 100 100 100 100 100

100 priority points available, per affiliate. More points = more important objectiveUse all 100 points

Question : What will change after review and re-iteration?

The objectives, the programs and their contribution, or the desirability of the collective outcome

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1717

11stst diagram view: Oxfam in Laos diagram view: Oxfam in Laos Questions to ask

1. Shared links: actual communication, specialisation, most valued

2. Unique links Most valued, balance of shared vs unique

3. Missing links between these actors

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1818

Oxfam in Laos: A different viewOxfam in Laos: A different view

1. Shared links. Shared preferences = communication about same objectives? And specialisation?

2. Unique links. Which are most valued?

3. Missing links. Is isolation of some objectives a problem, when there should be a causal connection?Strategic

Plan objectives Affiliates

Affiliates involvement

with SP objective

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 1919

Internal diagram view: Internal diagram view: M4P, HanoiM4P, Hanoi

M4P project, Hanoi

Nodes = workshops

Links = movement of participants from one workshop to another

Thicker link = more participants moving

Node colour = type of workshop (by topic )

Unique links

Shared links

Missing links

People’s participation in multiple workshops

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 2020

RolesRoles

MappingMapping Gather dataGather data

• Opportunistic sourcesOpportunistic sources• Planned sourcesPlanned sources

Collate, feedback, raise questionsCollate, feedback, raise questions NavigatingNavigating

Actors choose how they want to Actors choose how they want to respondrespond

The mapped territory then changesThe mapped territory then changes

Measuring Effectiveness, Melbourne, Sept 2007Measuring Effectiveness, Melbourne, Sept 2007 2121

What's new and not new?What's new and not new? Not new:Not new:

Emergent strategy: that which arises from the collision of Emergent strategy: that which arises from the collision of intentions, and circumstancesintentions, and circumstances

Structure as strategy: structure as means to an endStructure as strategy: structure as means to an end NewNew

The capacity to represent and analyse the collective The capacity to represent and analyse the collective structures that are emerging, via new methodsstructures that are emerging, via new methodsSocial Network Analysis, and softwareSocial Network Analysis, and software

NeededNeeded Finding useful generic questions to ask network participants about Finding useful generic questions to ask network participants about

the emergent structure, based on network analysis findings so farthe emergent structure, based on network analysis findings so far More participatory tools, to engage the actors within the networkMore participatory tools, to engage the actors within the network