MODELING AND SIMULATION Towards an epistemic (and epistemological) distinction between verification and validation Vitaly Pronskikh Models and Simulations

MODELING AND SIMULATION

Towards an epistemic (and epistemological) distinction between verification and validation

Vitaly Pronskikh

Models and Simulations 6, Notre Dame, 9 May 2014

2

Outline

• Modeling and Simulation• Modeling as theorizing• Simulation as numerical experimentation

• Modelers and simulationists as ideal types• Verification. Connections to Modeling• Validation. Connections to Simulation• Role of Calibration• Entanglement arguments and rebuttal• Conclusions

V. Pronskikh, Modeling and simulation…, Models and Simulations 6, Notre Dame, 9 May 2014

3

Modeling and simulation• Two approaches exist:

• Used interchangeably• Simulation as part of modeling

• Proposed definitions:• Modeling:

• Creation of computational mathematical models of elementary objects and processes (particles, atoms, their interactions)

• Simulation:• Creation of composite object models (embedding elementary ones)

and studying them experimenting with parameters• (accelerator, thunderstorm, Universe)• Example: Maxwell equations vs magnet alignment etc.

M=S M S

M S

M. WeisbergE. Winsberg


Verification Validation

4

Simulation and experimentation• Common sense: simulation is part of experimentation

(practiced together)

• Confutation: yes and no.• Yes: Franklin’s strategies apply – experimentation.• No: Conventional experimentation is preparation and

measurement of “real-world” phenomena (states of systems under scrutiny). Simulation has an ideal, model object.

• Interlaced, close, but distinct• Simulation makes possible both production and measurement

of phenomena (design of apparatus), and informs data interpretation

• It is Experimentation II – experimentation with ideal objectsV. Pronskikh, Modeling and simulation…, Models and Simulations 6, Notre Dame, 9 May 2014

5

Experimental strategies• 1) Checks and calibration, reproducing known results (Franklin).• “Show that simulation outputs match analytical solutions or data

from experiments” (E. Winsberg)• Modeling – analytical solutions (verification),

underdetermination by data.• Simulation – experimental data (validation). Due to complexity,

opacity, analytical solution is not generally available. Validation experiments designed to quantitatively assess the model.

• 2) Vary a parameter to see the response (adding ink to a sample and observing the color change in microscope).

• 3) Measuring the same observable in different kind of apparatus.

• 4) “Life of their own” I.HackingV. Pronskikh, Modeling and simulation…, Models and Simulations 6, Notre Dame, 9 May 2014

6

Modeler and simulationist as ideal types• Roles in simulation: An actual

practitioner can be represented by a simulationist A instructed by a modeler B and an IT expert C.

• She is either a simulationist (A) or a modeler (B) or an IT user (C) (usually a mix of those).

• B provides A elementary process models and instructions.

• C provides A computer code

(scripts) and instructions.

IT advanced user

Simulationist Process modeler


Rather than practitioners, practicesare not mixed types. One even expertin both domains plays either the roleof a modeler or a simulationist.

Confusing roles can lead to entanglementand mistakes.

7

Verification• “The process of determining that a model implementation

accurately represents the developer’s conceptual description of the model and the solution to the model”, AIAA, 1998.

• “Verification deals with mathematics”, Oberkampf, 2004• Code verification (fixing mistakes in the code)• Solution verification (estimating solution error, accuracy of

input and output)• Modeling – reference analytical solution can be used.• Verifies that the pre-built model is correctly implemented

in the code (accurately solved).


8

Validation• “The process of determining the degree to which a model

is an accurate representation of the real world from the perspective of the intended uses of the model.”

• “Validation deals with physics” (Oberkampf, 2004).• Reference – a real-world object.• Relation to experimental data. Benchmark.• Analytical solution is not generally available (complexity).• Simulation:

• Composite (real-world) objects. • Experimentation with parameters.

• Involves experimentation with parameters.• Possible issues: inputs fixed at the verification stage

should not be experimented with nor altered.V. Pronskikh, Modeling and simulation…, Models and Simulations 6, Notre Dame, 9 May 2014

9

Calibration• “The process of adjusting numerical or physical modeling

parameters in the computational model for the purpose of improving agreement with real-world data”.

• Model tuning.• “when validation is not feasible or practical, calibration is

appropriate” Oberkampf, 2004.• Calibration … is often a legitimate and important factor, and

may even be decisive, in determining the validity of an experimental result.” (A. Franklin, 1994).

• Calibration of a model – an experimental strategy. Should be applied to simulation – experimentation with composite model parameters.

• But: should not be applied to the elementary model parameters (which are in the scope of modeling)


10

Entanglement arguments and rebuttal• Computer simulations are not open to direct inspection. (J.

Jebeile). Epistemic opacity (P. Humphreys). • Distinction between modeling and simulation is key

(eliminates Duhem problem for V&V).• Modeling, model implementation, and code verification is open

to direct inspection by modelers (theorists). Methods exist to estimate numerical solution error at this stage.

• Simulation is opaque, but is a numerical-experimental practice (different scope), and proceeds relying on Franklin’s strategies.

• E. Winsberg: “model success due to piecemeal adjustment”.• - Verification stage inputs should not be adjusted at validation.• - Calibration stage parameters should not be altered.


11

Conclusions• I have argued that:

• Modeling (defined here as building mathematical computer models of elementary processes) and simulation (building models of composite objects and processes and numerical experimenting with them) are different in epistemic scope, first being theorizing involving idealization, and second being numerical experimenting based on Franklin’s strategies. These are roles and should not be confused.


12

Conclusions (cont’d)• Because of that distinction, verification (part of modeling) and validation (simulation) are distinct epistemologically (and not only in practice).

• This distinction eliminates Duhem problem for V&V.• For this distinction to hold and to prevent the entanglement and piecemeal adjustment, elementary model parameters chosen at verification in practice should not be modified during simulations.

• Verification and Validation parameters can be occasionally confused, but that does not entail their entanglement.


13

•Spare Slides

14

Positions on modeling and simulation• Simulation is “the kind of “theorizing” […] – the construction of

local, representative models.” Also as experimenting with computer. Model is simulation. E.Winsberg

• “a simulation imitates one process by another process” S.Hartmann

• “a computer simulation is any computer-implemented method for exploring the properties of mathematical models”; also a computational device producing solutions to a model P.Humphreys

• Modeling [is] “the indirect study of real-world systems via the construction and analysis of models”, simulation [is] “computing the behaviour of the model using a particular set of initial condition” M.Weisberg

• Simulation: Is it construction or exploring (computing behaviour), or both (system study, like modeling) ?


15

Experimental strategies II• 3) Measuring the same observable in different kind of

apparatus.• Process modeling – has no such sense for developing and

verification of models, different models can use different idealizations, do not have to lead to the same outcomes.

• Simulation – the same properties of a real-world system can be reproduced in simulation using sets of different elementary (embedded) models.

• “Life of their own” I.Hacking• Simulation (produces results not contained in embedded

models, opacity)• Process modeling – strictly determined by idealization.


16

Positions on modeling and simulation• Simulation is “the kind of “theorizing” […] – the construction of

local, representative models.” Also as experimenting with computer. Model is simulation. E.Winsberg

• “a simulation imitates one process by another process” S.Hartmann

• “a computer simulation is any computer-implemented method for exploring the properties of mathematical models”; also a computational device producing solutions to a model P.Humphreys

• Modeling [is] “the indirect study of real-world systems via the construction and analysis of models”, simulation [is] “computing the behaviour of the model using a particular set of initial condition” M.Weisberg

• Simulation: Is it construction or exploring (computing behaviour), or both (system study, like modeling) ?


17

Kinds of simulation• Computer simulations• Analog simulations (damped oscillator with an electric

circuit)• Scale simulations (Drosophila, genetic changes in other

species)• Agent-based

18

Simulations and progress of science• Advantages: Exploratory studies. Discoveries can be made that are later derived analytically. (Counterrotating vortex structures were first discovered by simulations, Hawley, analytically by Goodman (in Humphreys).

• Caveats:Tuning of simulation parameters to fit data• Kepler’s elliptical orbits vs Ptolimaic epicycles and deferents.

• Epicycles apparatus could account for Tycho Brage data if Kepler fitted parameters.

• Imagine, Kepler “fits the best circular orbit for Mars by least squares, puts in a dummy variable for the exceptional observation – and publishes” (Freedman quoted in Humphreys).

19

Simulations and Neo-Pythagoreanism

• Pythagoras: world is made up of numbers• Anti-realists: “any consistent theory can be reinterpreted

to be ‘about’ natural numbers” (Humphreys)• Computer operates on strings if binary states (these

strings are those of syntax, calling them numbers is a convention.

• Computer interprets theory in numerical terms.• Feynman: “the same equations have the same solutions”

– the methodological doctrine of computer simulations.• Simulation and realism – the model always contains non-

detachable interpretation.

20

Abstraction and idealization in simulation

• What is often known as ‘simplification’ in modeling is either abstraction or idealization.

• Abstraction: elimination of property (relation) from a set of properties. (other properties remain invariant).

• Eliminated properties 1) continue to exist but are cognitively ignored (a piece of radioactive uranium without alpha-particles emitted), 2) are set to zero in simulation (representational), 3) effects are physically eliminated.

• Idealization: transforming a property into the one related to the original but possesses desirable features lacking in the original. (treating planets as spherical)

21

Duhem-Quine problem in simulation• Theory is embedded in a network of auxiliary

assumptions, any of which can fail. Epistemic holism.• Discretization. Physics systems behavior is often

described by differential equations. Discretization transforms them into algebraic equations to be calculated by computer in the course of simulation. (Winsberg)

• Differential equations are transformed to difference ones. “Finite differencing”. Constitutes approximation.

• Arakawa operator (J.Lenhard). First climate models blew up because of instabilities in discretization of Jacobi operator. Arakawa substituted it with his own operator. His assumptions ran counter to experience and laws underlying the theoretical model.

22

Duhem-Quine problem cont’d

• D-Q problem has to do with falsification.• If model does not fails – what is wrong, theory or assumptions ?

• If model fails – balance of approximations is possible. Piecemeal adjustment (E.Winsberg).

• Verification and validation model of epistemology of simulation. Is there a divide ?

• Separable in practice.• Entangled, discretization (J.Jebeile)

Documents

MODELING AND SIMULATION Towards an epistemic (and epistemological) distinction between verification and validation Vitaly Pronskikh Models and Simulations