Thinking systemically: Thinking systemically: Seeing from simple to complex Seeing from simple to complex

in in impact evaluationimpact evaluation

Professor Patricia RogersRoyal Melbourne Institute of Technology, Australia

Dr. Irene GuijtLearning by Design, the Netherlands

Bob WilliamsIndependent consultant, New Zealand

(with thanks to Dr. Jim Woodhill, Wageningen International, the Netherlands)

Expert lecture for AfREA Conference Expert lecture for AfREA Conference March 30 – April 2, 2009March 30 – April 2, 2009 Cairo, EgyptCairo, Egypt

TODAY’S SESSIONTODAY’S SESSION

• Explore what thinking systemically is and how it relates to evaluation

• Introduce a systems approach that we think has the potential to move IDE forward, plus give you something you can use in your own practice

• Give you an opportunity to reflect and play with systems ideas and this method.

SYSTEMS CONCEPTS IN EVALUATIONAN EXPERT ANTHOLOGY

Eds. Iraj Imam & Bob Williams

http://www.iigss.net/gPICT.pdf

INTER-RELATIONSHIPS

PERSPECTIVES

BOUNDARIES

THREE ELEMENTS OF THINKING SYSTEMICALLY

INTER-RELATIONSHIPS

Being deeply aware of their significance

Some inter-relationships matter more than most

Some only matter over time

Some are slower in their impact than others

Some are linear (A affects B), some are non-linear and recursive (A affects B which affects A)

Most critically, systems thinking focuses on the inter-relationship of ideas, assumptions, beliefs as well as actions in the traditional cause and effect.

PERSPECTIVES

Thinking systemically about perspectives is not the same as stakeholder analysis

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

PERSPECTIVES

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

PERSPECTIVES We all bring different perspectives to

bear on anything we do. In this workshop I am handling four different perspectives

1. A session where people learn something

2. Something that allows me to communicate my knowledge

3. A means of expressing friendship and support to colleagues,

4. A way of enjoying myself.

You cannot understand how I behave at this session unless you understand how I juggle these perspectives.

QuickTime™ and a decompressor

are needed to see this picture.

BOUNDARIES

BOUNDARIES

Who or what is “in” and who or what is “out”

1. Purpose of evaluation; how will you judge “success”?

2. Resources for evaluation; what is not in your control?

3. What evidence is considered credible, whose expertise is acknowledged, or ignored?

4. Whose or what interests are not being served by an evaluation ?

BOUNDARIES

Thinking systemically requires you to do two things around those boundary decisions;

1.to identify the consequences of boundary decisions

2.consider how to mitigate any negative consequences of boundary decisions.

Tool for thought - the Cynefin FrameworkTool for thought - the Cynefin Framework

Facilitates seeing situational diversity Based on recognizing different types of cause

and effect relations – a given situation will contain aspects of all

Draws on theories of: Complexity Cognitive Systems Narrative Networks

Developed by Dave Snowden – ex-IBM knowledge management researcher

The Cynefin Framework – knowing The Cynefin Framework – knowing what you are dealing withwhat you are dealing with

ComplicatedComplicatedComplexComplex

ChaoticChaotic SimpleSimple

DisorderDisorder

UnorderedUnordered OrderedOrdered

Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)

Ordered Domain – Simple (known)Ordered Domain – Simple (known)

Cause and Effect: repeatable, perceivable, and predictable

Approach: Sense – Categorise - Respond

Methods: Standard operating procedures Best practices Process reengineering

Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)

Evaluating the ‘simple’Evaluating the ‘simple’

Simple aspects of a situation Causal links are tight, clearly observed and

understood Key variables to assess can be determined

For evaluation Need to know activities and some context If activity takes place, outcomes are known (e.g. polio

vaccination once in the person is guaranteed)

Monitoring is important  “sense, categorise, respond”

But need to guard for slipping into chaos

Ordered domain – Complicated Ordered domain – Complicated (knowable)(knowable)

Cause and Effect Cause-effect knowable with ‘expert’ input

Approach Sense – Analyse - Respond

Methods Analytical/reductionist Results-based thinking Scenario planning Good practices

Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)

Evaluating the ‘complicated’Evaluating the ‘complicated’ Complicated aspects

less predictable, less self-evident, subject to some debate and discussion

usually an evidence base: ‘if A in relation to X under Y conditions, then Z likely’

For evaluation outcome hierarchy/results chain to identify information

needed to understand impact activities, chain(s) of results and assumptions that link

them contextual information to explain results consulting ‘experts’ is key - “sense, analyse, respond”

Unordered Domain - ComplexUnordered Domain - Complex Cause and Effect:Cause and Effect:

Coherent in retrospect and do not repeatCoherent in retrospect and do not repeat

Approach:Approach: Probe – Probe – Sense Sense - Respond- Respond

Methods:Methods: Pattern managementPattern management Perspective filtersPerspective filters Circular dialogueCircular dialogue Emergent practiceEmergent practice Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)

Evaluating the ‘complex’Evaluating the ‘complex’ Complex aspects of a situation

unpredictable in advance; no clear understanding of chain of results; highly dependent on context and starting conditions

View as (a set of) experiments and figure out what ‘sticks’: “probe, sense, respond”

For evaluation: observe activities and possible results of those activities,

context fundamental, pathway of decisions to change construct sense-making by drawing people into a dialectic

Unordered Domain - ChaoticUnordered Domain - Chaotic

Cause and Effect: not perceivable

Approach: Act – Sense - Respond

Methods: Stability-focused intervention Crisis management Novel practice

Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)

Evaluating the ‘chaotic’Evaluating the ‘chaotic’

Chaos totally unpredictable; no clear understanding of chain

of results

For evaluation: Observe context, prioritise needs, act, observe again Afterwards (if/when situation stabilises), evaluate if

best possible action under the circumstances was taken (real time evaluation)

“act, sense, respond” - assess worth of “act” and subsequent effects and follow-up actions

Implications for IE PracticeImplications for IE Practice

How do we understand what is happening?

How do we change what is happening?

Intervention is both necessary and sufficient to produce Intervention is both necessary and sufficient to produce the impactthe impact

Impact

Intervention

‘‘Silver bullet’ simple impactsSilver bullet’ simple impacts

No impact

No intervention

Intervention is necessary but not sufficient to produce Intervention is necessary but not sufficient to produce the impactthe impact

Impact

Intervention

‘‘Jigsaw’ complicated impactsJigsaw’ complicated impacts

Favourable context

InterventionUnfavourable

context

No impact

Intervention is sufficient but not necessary to produce Intervention is sufficient but not necessary to produce the impactthe impact

Impact

Intervention

‘‘Parallel’ complicated impactsParallel’ complicated impacts

No intervention

Impact

Alternative activity

Plan B3Plan B2

Impact / Vision

Plan B

‘‘Life is a path you beat by walking’ Life is a path you beat by walking’ complex impactscomplex impacts

Intermediate Results (at 1, t+1, t+2)

Plan A

Plan C Plan D Plan D2 Plan E Plan E2

Plan D2

Plan F

ConceptualisingConceptualising the Intervention the Intervention

‘Silver bullet’ impactsintervention is necessary and sufficient to produce impact

Counterfactual approaches comparing similar groups with/without the intervention

‘Ducks lined up’ impactsintervention only produces impact with other things in place

Need to compare with and without full causal package

‘Multiple path’ impactsimpacts achievable by different means

Need to compare alternative ways of achieving impacts

‘Complex’ impacts –achieved by right combination for that particular situation

Need to assess quality and impact of non-standardised interventions

Differentiated study of causality

So what is ‘impact evaluation’?

SIMPLE COMPLICATED COMPLEX

Question answered

What works? What works for whom in what contexts?

What’s working?

Process needed Knowledge transfer

Knowledge translation Knowledge generation

Nature of direction

Single way to do it

Contingent Dynamic and emergent

Metaphor for direction

Written directions

Map and timetable Compass

Take home messagesTake home messages

1. Seeing ‘ontological diversity’ in situations (and interventions) enables a more conscious, appropriate methodologically mixed approach – it’s about being a good professional.

2. Cynefin is just a heuristic - a tool for thinking systemically

3. Thinking systemically is about a deep understanding inter-relationships, perspectives and boundaries. Boundary critique is the area where evaluation can learn most by drawing on the experience of the systems field.

ReferencesReferencesEoyang, Glenda. (2008) So, what about accountability?

http://www.cognitive-edge.com/blogs/guest/2008/12/so_what_about_accountability_1.php Glouberman, S. and Zimmerman, B. (2002) Complicated and Complex Systems:What Would Successful Reform of

Medicare Look Like? Commission on the Future of Health Care in Canada. Discussion Paper 8. Available at http://www.healthandeverything.org/pubs/Glouberman_E.pdf

Guijt, I. (2008). Navigating Complexity. Report of an Innovation Dialogue, May 2008. http://portals.wi.wur.nl/files/docs/Innovation%20Dialogue%20on%20Navigating%20Complexity%20-%20Full

%20Report.pdf

Guijt, I. and P. Engel. (2009). ‘Nine Hot Potatoes. Current Debates and Issues in Results-Oriented Practice.’ Presentation for Hivos In-house Training on Reflection-oriented Practice.

Mackie, J. (1974). The Cement of the Universe. Oxford University Press, Oxford.Mark MR. 2001. What works and how can we tell? Evaluation Seminar 2. Victoria Department of Natural Resources

and Environment.Rogers, P.J. (2008) ‘Using programme theory for complicated and complex programmes’ Evaluation: the international

jourmal of theory, research and practice. 14 (1): 29-48.Rogers, P.J. (2008) ‘Impact Evaluation Guidance. Subgroup 2’. Meeting of NONIE (Network of Networks on Impact

Evaluation), Washington, DC.Rogers, P.J. (2001) Impact Evaluation Research Report Department of Natural Resources and Environment, Victoria. Ross, H. L., Campbell, D. T., & Glass, G. V (1970). Determining the social effects of a legal reform. In S. S. Nagel

(Ed.), Law and social change (pp. 15-32). Beverly Hills, CA: SAGE.Williams, B. and I. Imam. (2007). Systems Concepts in Evaluation: An Expert Anthology. American Evaluation

Association.Woodhill, J. (2008). The Cynefin Framework: What to do about complexity? Implications for Learning, Participation,

Strategy and Leadership. Presentation for the ‘Navigating Complexity Workshop’, Wageningen International. http://portals.wi.wur.nl/files/docs/File/navigatingcomplexity/CynefinFramework%20final%20%5BRead-Only%5D.pdf