Open policy practice: open science-based assessments for decision-makers

Jouni Tuomisto

National Institute for Health and Welfare, Kuopio, Finland

Outline

• Information flow in policy support

• Examples of open assessments

• Shared understanding

• Six principles of open policy practice

• Structure of information objects

• Lessons learned from open policy practice

Information flow in current decision support

Original data

Scientific publications

CBA, impact assessment etc.

Scientific review

Report

Practical knowledge and lobbying

Civil servant

Stakeholders

Researchers

Expert

Decision maker

Open policy practice

Open original data

Scientific analysis

Other scientific literature

Open assessment

Report

Practical knowledge and lobbying

Civil servant

Stakeholders

Researchers

Expert

Decision maker

What are open assessment and Opasnet?

• Open assessment– How can scientific information and value judgements

be organised for informing societal decision making in a situation where open participation is allowed?

– [Previous names: open risk assessment, pyrkilo]

• Opasnet– What is a web workspace that contains all

functionalities needed when performing open assessments, based on open source software only?

Why Opasnet?

• Need for systematic flow of and place for relevant information

– Scientific data and interpretations– Valuations and discussions– Decision options and objectives– Models and scenarios

7

Tendering process for pneumococcal vaccine• Need to buy a new vaccine for the Finnish vaccination

program (ca. 150000 doses per year).• What should be the decision criteria?• This question was answered by using an open hearing in

Opasnet.– Epidemiological model about health impacts of vaccines.– Cost effectiveness model including price and health costs.– Online discussion forum about valuations and

assumptions.• Best outcome: no outrage!

– Reasons: Specific question, moderation?• Drug companies were active, anti-vaccine groups were

not.• Little outside researcher involvement.

8

Other projects and assessments in Opasnet (1)

• Climate change policies and health in Kuopio, Finland

• Future overview reports of Finnish ministries (Transport and Logistics; Health; Environment)

• Evaluation and summary of several climate policy reports, strategies, and programs of the city of Helsinki.

• Health and ecological risks of mining (guidance and models)

• Water guide for assessing health risks of raw water contamination.

9

Other projects and assessments in Opasnet (2)

• http://fi.opasnet.org/fi/Ymp%C3%A4rist%C3%B6nsuojelulaki Environmental protection law

• http://fi.opasnet.org/fi/Rauman_sataman_laajennuksen_vaikutus_terveyteen Urban planning in Rauma

– http://fi.opasnet.org/fi/Pienhiukkasp%C3%A4%C3%A4st%C3%B6t_Raumalla Assessment of fine particles from the Port of Rauma

• http://fi.opasnet.org/fi/Kilometrikorvaus_AM_2012 Compensation for work-based driving

• http://fi.opasnet.org/fi/Puijon_metsien_k%C3%A4ytt%C3%B6suunnitelman_p%C3%A4%C3%A4t%C3%B6ksenteko Use of Puijo forest area

Main findings from Pohjola et al 2012:

• In environmental health assessments there are tendencies towards:

• a) increased engagement between assessors, decision makers, and stakeholders

• b) more pragmatic problem-oriented framing of assessments

• c) integration of multiple benefits and risks from multiple domains

• d) inclusion of values, alongside scientific facts, in explicit consideration in assessment

Shared understanding: graph

• Pohjola MV et al: Food and Chemical Toxicology. 2011.

Shared understanding: definition

• There is shared understanding about a topic within a group, if everyone is able to explain what thoughts and reasonings there are about the topic.

– There is no need to know all thoughts on individual level.

– There is no need to agree on things (just to agree on what the disagreements are about).

Six principles of open policy practice• Intentionality: All that is done aims to offer better

understanding to the decision maker about outcomes of the decision.

• Shared information objects: all information is shared using a systematic structure and a common workspace where all participants can work.

• Causality: The focus is on understanding the causal relations between the decision options and the intended outcomes.

• Critique: All information presented can be criticised based on relevance and accordance to observations.

• Reuse: All information is produced in a format that can easily be used for other purposes by other people.

• Openness: All work and all information is openly available to anyone interested. Participation is free. If there are exceptions, these must be publicly justified.

• http://en.opasnet.org/w/Open_policy_practice

An example of an open assessment

• Health impact of radon in Europe

An example of a variable in a model

An example of a statement and resolution of a discussion

• Is Pandemrix a safe vaccine?

Application of soRvi in Opasnet

Results from soRvi

Problems perceived about open participation1. It is unclear who decides about the content.2. Expertise is not given proper weight.3. Strong lobbying groups will hijack the process.4. Random people are too uneducated to contribute

meaningfully.5. The discussion disperses and does not focus.6. Those who are now in a favourable position in the

assessment or decision-making business don’t want to change things.

7. The existing practices, tools, and software are perceived good enough.

8. There is not enough staff to keep this running.9. People don’t participate: not seen useful, no time, no

skills.10. People want to hide what they know (and publish it in a

scientific journal).

Problems observed about open participation1. People want to hide what they know (and publish it in a

scientific journal).2. People don’t participate: not seen useful, no time, no

skills.3. The existing practices, tools, and software are perceived

good enough.4. There is not enough staff to keep this running.5. Those who are now in a favourable position in the

assessment or decision-making business don’t want to change things.

6. The discussion disperses and does not focus.7. It is unclear who decides about the content.8. Expertise is not given proper weight.9. Strong lobbying groups will hijack the process.10. Random people are too uneducated to contribute

meaningfully.

Main rules in open assessment (1)

• Each main topic should have its own page.– Sub-topics are moved to own pages as necessary.

• Each topic has the same structure:– Question (a research question passing the

clairvoyant test)– Answer (a collection of hypotheses as answers to the

question)– Rationale (evidence and arguments to support,

attack, and falsify hypotheses and arguments)• ALL topics are open to discussion at all times by

anyone.– Including things like ”what is open assessment”

Main rules in open assessment (2)

• Discussions are organised around a statement.• A statement is either about facts (what is?) or moral

values (what should be?)• All statements are valid unless they are invalidated,

i.e. attacked with a valid argument [sword].• The main types of attacks are to show that the

statement is– irrelevant in its context,– illogical, or– inconsistent with observations or expressed values.

• Statements can have defending arguments [shield].

Main rules in open assessment (3)

• Uncertainties are expressed as subjective probabilities.

• A priori, opinions of each person are given equal weight.

• A priori, all conflicting statements are considered equally likely.

Future promises and challenges• Technically, Opasnet works surprisingly well.

• Personally, I am able to do almost all my work in Opasnet.

• Many people see open participation and expert + decision-maker collaboration as a promising approach.

• However, many reasons for resistance:

– Open practices are a threat to expert authority.– People don’t want to show intermediate work.– Old tools are considered better for each specific task.

Now it is time for new ambitious collaboration and community for open online modelers/assessors.

Course DARM starts at UEF Jan13, 2015!

26

Conclusions

• We could do most of our scientific work online using shared information systems and web workspaces (such as Opasnet).

• These tools exist and are functional.

• The work would be quicker and better.

• There are major obstacles of new practices:– Lack of awareness.– Lack of practical knowledge to use tools.– Current practices and incentives are against sharing.– Reluctance to change things.

• Join Decision Analysis and Risk Management in 8 Jan – 14 Feb 2013!

History briefly: borrowing and combining ideas

– 1996: EU Parliament visit: ”Information does not flow!”– 1997: Idea of Internet-based assessments– 2000: Decision analysis (Harvard University)– 2005: Wikipedia, wiki approach– 2006: Opasnet wiki launched– 2006: Wikinomics, mass collaboration, wisdom of crowds– 2006: Argumentation rules (Amsterdam University)– 2007: Open assessment– 2009: Wiki government– 2011: online wiki modeling using R– 2012: MongoDB database– 2013: Open policy practice (guidance for making

decisions)– 2014: Several policy projects without research funding

29

Framework for TEKAISU method

Open risk management: overview

QRA

• Mikko V Pohjola and Jouni T Tuomisto.. Environmental Health 2011, 10: 58 doi

Public health data

How Opasnet helps in assessments

• https://docs.google.com/drawings/d/1f1s1drjo8qMJ-vWR3BQgsfRbH2DO0E43Xb01eRddWcg/edit?hl=en_GB&authkey=CN_oqbYK&pli=1