Upload
dinah-campbell
View
218
Download
0
Embed Size (px)
DESCRIPTION
3 Illuminating the Path Visual Analytics Agenda - Recommendations –Rec. 4.10: Develop new methods and technologies for capturing and representing information quality and uncertainty –Rec. 4.11: Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment. – Summary Rec: Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process
Citation preview
Click to edit Master title style
Click to edit Master subtitle style
A PRACTICAL LOOK AT UNCERTAINTY MODELING
Steve UnwinRisk & Decision Sciences Group
March 7, 2006
2
"The fundamental cause of trouble in the world today is that the stupid are cock-sure while the intelligent are full
of doubt.“
Bertrand Russell
3
Illuminating the Path• Visual Analytics Agenda - Recommendations
– Rec. 4.10: Develop new methods and technologies for capturing and representing information quality and uncertainty
– Rec. 4.11: Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment.
– Summary Rec: Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process
4
Uncertainty Analysis as Resource to Visual Analytics
• VA Agenda
– Develop new methods and technologies for capturing and representing information quality and uncertainty
– Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment.
– Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process
• UA Insight
– Probabilistic techniques• Elicitation methods• Aggregation methods• Information-theoretic approaches
– Nonprobabilistic alternatives• Dempster-Shafer• Possibility theory
– Uncertainty propagation techniques• Analytic• Numerical
– Risk communication• Risk representation• Decision-analysis methods
5
MEASURING UNCERTAINTY
CLASSICALMETHODS BAYESIAN
METHODS
NON-PROBABILISTIC
METHODS
6
Classical Statistics
• Focus on Aleatory Uncertainty– random variation inherent in the system
• Sampling produces confidence intervals• Need a sampling model
– Generally unavailable for many real-world complex situations
• Confidence intervals are not probability intervals– Propagation difficulties in even the simplest models
7
Bayesianism
• de Finetti, Ramsey, Savage (1920s-50s)• Subjectivism – Epistemic Probabilities
– Probability as a degree of belief• Classicists are coin tossers• Bayesians are believers
– What is the basis for forming probability?• “ Probabilities do not exist”
– Bruno de Finetti
8
Problems with Bayesianism• Because probabilities don’t exist, they have to be
created– but how?
• Bayes’ Theorem• Subjectivity is explicit
– judgment of evidence• Do probabilities really reflect the way we conceive
belief?– is probability theory a good theory of evidence?– what are the options?
9
One Option:Dempster-Shafer Theory
• Withholding belief distinct from disbelief• Seahawks or Steelers will win?• Set of possibilities: {sea, steel}• Probability theory:
– Weight of evidence attached to each exclusive possibility– p(sea), p(steel)
• D-S theory:– Weight of evidence attached to each subset– m(Ø), m(sea), m(steel), m(sea U steel)
• Allows: m(sea U steel) = 1, all other m=0– A compelling ignorance
10
Support and Plausibility
• Probability replaced by two belief measures:– Each calculated from weights of evidence– bel(sea) is the support for proposition ‘sea’– pl(sea) is the plausibility of ‘sea’– bel(sea) ≤ pl(sea)– Upper and lower “probabilities”
• Complete ignorance• SDU: bel(sea) = 0, pl(sea) = 1, i.e., complete
ignorance on the matter of proposition ‘sea’
11
Complementary Cumulative Belief Functions
ESD Sensor System On-Demand Failure Rate
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.0E-04 1.0E-03 1.0E-02 1.0E-01 1.0E+00
Failure Rate per Demand
Belie
f Met
ric
Complementary Cumulative Support/ Belief
Complementary Cumulative Plausibility
Complementary Cumulative Probability
12
Possibility Theory
• Genesis in fuzzy sets• Possibility is an uncertainty measure that
mirrors the fuzzy set notion of imprecision
13
The Set of Tall Men
0
0.2
0.4
0.6
0.8
1
5' 8" 5' 9" 5' 10" 5' 11" 6' 0" 6' 1" 6' 2"
Height
Mem
bers
hip
to S
et TallVery Tall
m(h)
m'(h) = m2(h)
14
Possibility Theory
• 2-tier belief: possibility and necessity• nec(X) ≤ pos(X)• Distinctive combinatorial logic
– nec(X^Y) = min[nec(X), nec(Y)]– pos(XvY) = max[pos(X), pos(Y)]
• No conceptual connection to probability– although probability/possibility can co-exist
15
Possibilistic Interpretationof Intelligence Statements (Heuer)
Probability
Possibility
Chances are slight
Little chanceBetter than even
Highly likely
16
Experience with Nonprobabilistic Methods
• Not all good:– Standardization of belief metrics?– Treatment of dependences?– Treatment of conflicting evidence?– Computational demands?– Interpretation of results?– Incorporation into decision process?
• Plan B: Confront the problems with probabilistic methods
17
Principled Basis for Probability Formulation
• Analysts uncomfortable producing probabilities– justified discomfort
• Alternative:– Produce defensible basis for probability formulation based on
nonprobabilistic judgment• Maximize expression of uncertainty subject to judged
constraints• Borrow uncertainty metrics from:
– statistical mechanics– information theory
• Entropy = -∑i pi.ln pi – discrete probability distribution, pi
18
Application of Information-Theoretic Methods
• Two USNRC programs:– QUEST- SNL
• Quantitative uncertainty evaluation of source terms
– QUASAR – BNL• Quantitative uncertainty analysis of severe accident releases
• Both studies used the same form of input to the same deterministic models– non-probabilistic input
• expert-generated input parameter uncertainty ranges
• QUEST: Bounding analysis
• QUASAR: Information Theory used to generate probability distributions from bounds
• Probabilistic analysis internal to methodology – no elicitation of probability
19
Information Theory and the Preservation of Uncertainty
Uncertainty Bands
1.00E-04 1.00E-03 1.00E-02 1.00E-01 1.00E+00
QUEST
QUASAR
QUEST
QUASARI-131
Cs-137
Release Fraction
20
Uncertainty Analysis as Resource to Visual Analytics
• VA Agenda
– Develop new methods and technologies for capturing and representing information quality and uncertainty
– Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment.
– Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process
• UA Insight
– Probabilistic techniques• Elicitation methods• Aggregation methods• Information-theoretic approaches
– Nonprobabilistic alternatives• Dempster-Shafer• Possibility theory
– Uncertainty propagation techniques• Analytic• Numerical
– Risk communication• Risk representation• Decision-analysis methods
21
Merit Criteriafor Uncertainty Analysis in Intel
• Makes the analyst’s job easier• Represents strength of evidence intuitively• Can reflect dissonant evidence• Appropriately propagates uncertainty from analyst
to decision-maker• Characterizes output uncertainty in a standardized
and interpretable way• Computationally tractable• Promotes insight