Expert judgment and Bayesian updating

L9

2

Expert Judgment

• Throughout the process of identifying and characterising uncertainties and undertaking assessment calculations, there is a need for expert judgment

• There are specific issues associated with using expert judgment to characterise uncertainties and apply numerical values

• There are guidelines and protocols that should be applied for all such judgments

• Expert elicitation is a structured approach that helps experts reach judgments

Use of Judgments

• Expressing uncertainties in terms of “degrees-of-belief” requires use of judgments as well as site characterisation data

• Judgments may be individual judgments, group judgments, or elicited judgments

• In all cases, documentation is the key process leading to traceability and transparency

• Risk cannot be regarded as an absolute quantity, but is the result of many assumptions and judgments

Role of Judgment

• “If a risk assessment is based on assumptions and judgments, and includes arbitrary parameter values, it can yield almost any answer”

• Expert judgment is pervasive in scientific studies. The question is not whether to use expert judgment but whether to use it in an overt manner, documenting its use, or to hide its use.

Examples of Judgment

• Giving values to variables, in the absence of measurements

• Deciding on conceptual model, deciding on scenarios

6

Guidance on use of Elicitation

• Recent UK regulatory guidance provides useful summary:

– Where this type of expert judgment is used, the developer/operator should, to an extent proportionate to the significance of the judgments to the environmental safety case: • explain the choice of experts and method of elicitation;

• document explicitly expert judgments that have been made and the reasons given by experts to support their judgments;

• take and document reasonable steps to identify and eliminate or minimise any biases resulting from the use of expert judgment and/or the elicitation methods adopted.

7

Expert judgment protocols

• Train experts to provide formal opinions

• Identify and minimise the effect of biases

• Define the issue to be assessed with no ambiguity

• Make available to the experts all the relevant information about the issue under study

• Check the rationality and consistency of the opinions given

• Make a final verification, repeating the whole process if needed

Aspects of Expertise

• When estimating uncertainties, there are two parts two a person’s expertise

• Informativeness – this is a measure of an expert’s knowledge about his subject

• Calibration – this is a measure of an expert’s ability to quantify his uncertainty i.e. how well he knows what he does not know!

• Without the latter ability, the former is not much use!!!

• With appropriate training and feedback, experts can become almost perfectly calibrated in estimating their uncertainty in terms of probabilities, and avoiding bias

9

Expert elicitation

• Structured approach using an elicitor or facilitator – Supported by modeller or technical expert

– Minimum of three experts

• Used to derive PDFs by asking experts to define bounds and express relative likelihoods – What is the largest value that you are virtually certain X will not fall

below? Why?

– Smallest value? Why?

– How confident are you that the value is greater than X?

– What value do you think 50% of the distribution lies above?

• Not guesswork – there must be a basis for judgments

Difficulties with Experts

• Overconfident – range too narrow

• Biased, prejudiced

• Poorly informed

• Experts agree – all wrong!

• Experts disagree – how to document result?

• Questions poorly constrained

• Questions too difficult, e.g. ‘what is standard deviation of …?’

Combining Elicitation and Data

• Useful when there are some measurements, but too few to fit a PDF

• Bayes theorem

– Revised probability of Y given X is the probability of X given Y times probability of Y divided by probability of X

Key steps

• Individual experts, or a panel? • Agree the variable to be elicited – define conditions, units…

– Definition may need to be extensive – encapsulate all questions asked by experts

• Train experts • Define extreme values, with justifications

– NEVER start with central values

• Define quartiles (25%, 75%, 50%) • Feedback tertiles (33%, 67%) • Combine results, if not using a panel • Feedback distribution • Document results and reasons

PAMINA protocol

Phases: 1. Selection of the project team. 2. Preparation of supporting material and definition of the questions to be

studied. 3. Selection of experts. 4. Training sessions. 5. Refinement of the questions to be studied. 6. First individual work period. 7. Presentation of individual approaches adopted by the experts. 8. Second individual work period. 9. Elicitation sessions. 10. Analysis and aggregation of results. 11. Review. 12. Documentation.

Individuals or a Panel?

• A panel is easier!

– Only one (set of) training sessions

– No need to combine

• A panel will usually produce a narrower result

– Panel should have more members

Ways of Combining results

• Reaching a consensus

• Mathematical • Linear pool

• Log-linear pool

• Bayesian combination

• Not combining • Leave different distributions

for sensitivity analysis

Experiment 1/1 - training

• How many bones in a human body?

• How many bones in an adult human body, according to Wikipedia?

Copyright Wikipedia

Experiment 1/2 Degree of belief?

• 76

• 104

• 105

• 106

• 107

• 203

• 204

• 205

• 206

• 304

• 305

• 306

• 307

• 413

Experiment 2

• What is the flow in the Thurso River?

Copyright NERC

Annual average rainfall

1000 mm/a

Copyright NERC