Context-dependency, risk analysis and policy modelling

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, CRW@ECCS ’2012, Brussels, September 2012, slide 1

Context-dependency, risk analysis and policy modelling

Bruce EdmondsCentre for Policy Modelling,

Manchester Metropolitan University


the ProblemPart 1:


“A large amount of research is pertinent to public policy. But a seemingly tiny amount of that knowledge makes it through to the policy

world. This is a frustration not only for the research community but also for the policy

community…” (Chris Tyler 2010)


The “gap” between the worlds of research and that of policy• There is often a mismatch between the wishes of

policy advisors/makers and academic researchers in terms of the kind of conclusion desired.

• Those in the policy world seem to wish for a relatively simple “best guess” (or tools to produce these) possibly accompanied by a supporting narrative

• Those in the research community might be all-too aware of the difficulties of the issues under consideration and so wish to be much more cautious in their conclusions, and wish to include special cases, caveats and complexities.


Some Bad Solutions

• The policy world can try to ‘force’ researchers to produce simple answers which academics are unhappy with or insist they solve impossible problems with inadequate means

• Researchers can essentially try to ‘con’ those who commission research, claiming that they will answer all the questions that are specified, but re-defining the goals/terms as they go along or ‘discovering’ obstacles to them satisfying the policy makers/advisors as they envisaged

• The researcher and/or the policy advisor mutually convince themselves and each other that the research gives definitive, simple and possibly general answers and rush to implement the policy indicated


The “gap” between the worlds of research and that of policy

• There are many reasons behind these very different views…

• …including different: kinds of goal, motivations, training, strategies, language, habits, constraints, working environments etc.

• This talk will focus on a particular difficulty, that caused by the context-dependency of human socio-political behaviour and make some suggestions to improve things


An Analysis of a Part of the Problem:Context-Dependency

Part 2:


A (simplistic) illustration of context from the point of view of an actor


Situational Context

• The situation in which an event takes place• This is indefinitely extensive, it could include

anything relevant or coincident• The time and place specify it, but relevant

details might not be retrievable from this• It is almost universal to abstract to what is

relevant about these to a recognised type when communicating about this

• Thus the question “What was the context?” often effectively means “What about the situation do I need to know to understand?


Context-dependency of Cognition

• “The” context is the situation of an event, but this is indefinitely extensive

• The brain somehow categorises and recognises different kinds of situation and preferentially gives access to knowledge on this basis, it is context-dependent

• Many aspects of human cognition seem to be context-dependent, including: memory, visual perception, choice making, reasoning, emotion, and language


The Context Heuristic

• The kind of situation is recognised in a rich, fuzzy, complex and unconscious manner

• Knowledge, habits, norms etc. are learnt for that kind of situation and are retrieved for it

• Reasoning, learning, interaction happens with respect to the recognised kind of situation

• Context allows for the world to be dealt with by type of situation, and hence makes reasoning/learning etc. feasible

• It is a fallible heuristic…• …so why do we have this kind of cognition?


Social Embedding

• Granovetter (1985) • Contrasts with the under- and over-socialised

models of behaviour• That the particular patterns of social

interactions between individuals matter• In other words, only looking at individual

behaviour or aggregate behaviour misses crucial aspects

• That the causes of behaviour might be spread throughout a society – “causal spread”

• Shown clearly in some simulation models


Illustration of Causal Complexity

Lines indicate causal link in behaviour, each box an agent (Edmonds 1999)


Implications of Social Embedding

• In many circumstances agents can learn to exploit the computation and knowledge in their society, rather than use or reason with their own knowledge

• That is perceiving and acting through their network of social relationships

• Social embedding means that human behaviour can not be understood well separate from the surrounding social context

• This is not a problem for normal human action as human cognition automatically (and unconsciously) adapts to context

• However such normal action and cognition is hard to systematically justify or check


Some Researchers’ Responses to Context-Dependency

A number of responses:• Only do research within a single context,

resisting any generalisation • Only use discursive, natural language

approaches where context is implicitly dealt with (and thus masked)

• Try to see what (inevitably weaker knowledge) is general across the various contexts in what is being studied


Some Policy-World Responses to Context-Dependency

• Wishing to know what is the “educated guess” as to which policy is best for most

• Using their detailed and informed intuition as to what the best policy is and looking for a more “scientific” justification for it

• Wanting a general narrative that “explains” (in comprehensible terms) why a certain policy direction is desirable or which can be used to build a consensus around


The “Simple is more General” Fallacy

• If one has a general model one can make it more specific (less general) by adding more processes/aspects…

• …in which case it can become more complex• However, the reverse is no true…• If one simplifies/abstracts then you don’t get a more

general model (well almost never)!– there may be no simpler model that is good enough for

your purpose– But, even if there is, you don’t know which aspects can be

safely omitted – if you remove an essential aspect if will be wrong everywhere (no generality)


Context-Dependency and Randomness

Lots of information lost if randomness used to “model” contextual variation


Context-Dependencyand “Being Scientific”

• If the relevant context can be reliably indentified then…

• …context-dependency is not the same as subjectivity

• Generality is nice if you can get it, but its no good pretending to have it if you can’t

• Science should adapt to what it wishes to understand, not the other way around

• It does mean (often) an acceptance that general/generic approaches are not appropriate


What does not work…

• Inventing a model which only relates macro-level variables, not taking into account the social structures, using randomness to “cover” the variation across contexts (might as well use a straight line projection or random number generator!)

• Presenting a policy maker/advisor with a “warts and all” model/account that is so complicated it is hard to work out a coherent policy response


Some SuggestionsPart 3:


Fund Ethnographic Observations of Researchers Interacting with Policy Makers/Advisors

• To “debug” what goes wrong (or right) with this relationship (randomly selected) projects need to be observed by a third party who…– Watches and makes notes about the interactions– Interviews the participants at various stages– Produces an impartial report reporting on the

observations and conclusions (but anonymising the participants/project involved)

– Participants get to add their thoughts on reading it as appended commentaries

• Periodically collections of such reports be publically released


Encouraging the development of multiple models• Getting competing models from contrasting

researchers for the same target• Allowing/encouraging different models for different

aspects of complex issues• Encouraging the staging of abstraction from evidence

to conclusions• Making models publically accessible at an early stage

(well before the project ends) to allow comment and/or criticism

• Models must be accompanied by documentation on: intended use, what is validated, whether replicated, etc.


Tools that could (used appropriately) help with context-dependency• Multi-dimensional data in its original state (or near to), that is

different kinds of data about the same set of actors/events• Descriptive agent-based simulation where the different

contexts agents might face are matched with different behavioural models

• Local Data-Mining which searches out patterns that hold but also what parts of the data they hold for (indicating context-dependency)

• Visualisation techniques which allow for researchers and stakeholders to explore, envisiage and check ideas about data

• Tools to aid the tracking of relations between multiple models and multiple data sets


Layering Reports on Research

Structuring Policy Modelling Research Conclusions:1. A general summary of the results and recommendations

without caveats2. A short summary of the reasons for the above in narrative

form (non-technical language)3. A risk analysis specifying how the above might fail, i.e. the

contexts under which the recommended policies will work less well or not at all

4. A description of the kinds of situation where the policy will be most effective (to aid targeting of a policy)

5. A (perhaps lengthy) appendix with details, maybe broken down into the various cases, documenting assumptions, evaluation criteria etc.


Towards some Rules for Engagement

Establish a set of rules/norms that should hold in policyresearch interaction1. Researchers accepting layered reporting in exchange for

ensuring the risk-analysis does not get seperated from the summary advice 1&2 (which is what they fear)

2. An acceptance of context-specific solutions/models/specifications (or that general versions might not be applicable or effective)

3. Not asking for, nor offering, predictions of complex social systems, but rather accepting evaluations of possibilities

4. Not asking for impossible things from researchers (e.g. “general solutions for complexity”)

5. Researchers being honest about their limitations and admitting when they fail (post hoc evaluation)


Writing “User Manuals” for Policy Advisors and Researchers

Two Manuals:1. A manual written for

academics/researchers for how to deal with and communicate with policy adivsors/makers

2. Another written for policy makers/advisors of what to expect from academics/researchers, how to effectively use them and their shortcomings (as might be seen from the policy world)


The End

Bruce Edmondshttp://bruce.edmonds.nameCentre for Policy Modelling

http://cfpm.orgManchester Metropolitan University

http://www.business.mmu.ac.uk

Documents

Context-dependency, risk analysis and policy modelling