Embed Size (px)
DESCRIPTION
2014 SIAM Conference on Uncertainty QuantificationMS42: Reliability of Computational Research Findings: Reproducible Research, Uncertainty Quantification, andVerification & Validation
Citation preview
2014 SIAM Conference on Uncertainty Quantification
MS42:!e Reliability of Computational Research Findings: Reproducible Research, Uncertainty Quantification, and Verification & Validation
Prof. Lorena A. BarbaMechanical and Aerospace Engineering Department, George Washington University
Twitter: @LorenaABarba
1
About me
2
http://lorenabarba.com
Not an expert in UQ
3
Uncertainty Quantification
Reproducible Science
Validation & Verification
4
5 6
7
Can UQ make “irreproducible” large-scale simulations reliable?
8
Why does it matter?Computer simulations create scientific knowledge.
9
SocrativeGo to:
http://m.socrative.com
‣ Room number : 75443
10
True/False?Computer simulations create scientific knowledge.
11 12
True/False?Simulation is a method.
13 14
When do we have evidence that claims to knowledge from simulation are justified?
E. Winsberg, “Science in the Age of Computer Simulation” (2010)
15
!e Method of computer simulation
‣ Same as von Neumann et al. in 1945
Mathematical Model
Start:
Transform into Algorithm ComputeProblem we can’t solve
16
!e Method of computer simulation
‣ Same as von Neumann et al. in 1945
Mathematical Model
Start:
Transform into Algorithm ComputeProblem we can’t solve
Already trusted
16
!e Method of computer simulation
‣ Same as von Neumann et al. in 1945
Transform into Algorithm ComputeMathematical
Model
Start:
Problem we can’t solve
17
!e Method of computer simulation
‣ Same as von Neumann et al. in 1945
Transform into Algorithm Compute
Discretize the equations:...what if accuracy demands a grid that is too fine to compute?
17
!e Method of computer simulation
Deviates from the mathematical model to make it computable:
‣ Simplifying assumptions— symmetry, ignoring variables
‣Add elements that are not in the original equations, e.g., eddy viscosity, cloud models
‣What if the initial algorithm is unstable?
Transform into Algorithm
Mathematical Model
Problem we can’t compute
18
True/False?A reproducible simulation does not need to be accurate.
19 20
True/FalseIs there value to a Reproducible Wrong Answer?
(If yes, V&V and Reproducibility are distinct.)
21 22
Validation & Verification
‣Validation — confirm that the simulation parallels the physics we want to study
‣Verification — confirm that the simulation results match the solutions to the mathematical model
23
Validation & Verification
‣Validation — confirm that the simulation parallels the physics we want to study
‣Verification — confirm that the simulation results match the solutions to the mathematical model
Fitness of the model
Fitness of the solver
23
Validation & Verification
‣Validation — confirm that the simulation parallels the physics we want to study
‣Verification — confirm that the simulation results match the solutions to the mathematical model
(Can these really be divided neatly?)
Fitness of the model
Fitness of the solver
23
How do we show evidence that a simulation gives reliable data about the real world?
24
How do we come to rationally believe in an experimental result?
A. Franklin, “!e Neglect of Experiment” (1986)
25
True/False?Simulation is an experiment.
26
27
Not so neat as V&V
‣What about comparisons with other simulations?
‣What is the role of visualization? data interpretation? analysis?
‣And all the judgements in between ...
Important for Reproducibility!
28
Not so neat as V&V
‣What about comparisons with other simulations?
‣What is the role of visualization? data interpretation? analysis?
‣And all the judgements in between ...
Important for Reproducibility!
V&V --> trust the simulation resultsR --> trust the science results? trust the scientist?
28
True/FalseIs V&V a pre-requisite to Reproducibility?
29
30
Confidence in results ...
‣mathematical rigor needed, but not enough!
‣ involves physical intuition
‣ often relies in visualizations
‣ requires skilled judgement
‣ value questions
31
True/FalseUQ aims to give objective confidence levels for the results of simulations.
32 33
True/FalseIs V&V a pre-requisite to UQ?
34 35
True/FalseUQ presupposes verification and informs validation.
36 37
True/FalseVerification should be done before validation.
38 39
True/FalseIn verification, the actual value of the error is generally unknown.
40 41
Uncertainty Quantification
Reproducible Science
Validation & Verification
?
42
True/FalseCan reproducible science be uncertain?
43
44
Repeatability & Reproducibility
‣ Repeatability — ability to re-run the same experiment with the same method on the same syste, and obtain the same/similar results
‣ Reproducibility — ability to independently confirm a scientific hypothesis via reproduction of published work
Vitek & Kalibera, “R3—Repeatability, Reproducibility, Rigor” (2011)
45
True/FalseUQ is a requirement of reproducibility.
46 47
My conclusion
My Reproducibility PI Manifesto [*] is incomplete. Making code/data open and publicly available is not all, we need to publish our uncertainties, our judgements.
Lorena A. Barba
[*] http://lorenabarba.com/gallery/reproducibility-pi-manifesto/
Twitter: @LorenaABarba
48 49
