Simulating an Imperfect World - Engineering Integrity …e-i-s.org.uk/wp-content/uploads/2017/01/Peter-Lavelle.pdf · conditions for the calculations (eg. RBE2 or RBE3?) ... Usefulness

© 2016 HBM

HBM Prenscia: Public

© 2016 HBM

Simulating an Imperfect World

Peter Lavelle, Application Engineer

© 2016 HBM


2

1. Introduction

2. Simulation and the Design Process

3. Simulation Accuracy & Precision

4. Simulation Fidelity

5. Validation & Verification

Agenda

www.altairhyperworks.com/industry/Automotive

© 2016 HBM


3

“simulation”Simulation is the imitation of the operation of a real-world process or system over

time. The act of simulating something first requires that a model be developed.This model represents the key characteristics or behaviours/functions

of the selected physical or abstract system or process.[https://en.wikipedia.org/wiki/Simulation]

Introduction

“imperfect”Damaged, containing problems, or not having something.

[http://dictionary.cambridge.org]

The Matrix, Warner Bros, 1999

© 2016 HBM


4

Why describe the world as “imperfect”? Imperfect compared to what…

Engineering is about Maths and Physics!

An Imperfect World

E = mc2

http://worldartsme.com/images/funny-einstein-clipart-1.jpg

Periodic

Table

Electricity

© 2016 HBM


5

Electricity

But do we really know everything…...or is engineering about “knowing enough”?

An Imperfect World

E = mc2

http://worldartsme.com/images/funny-einstein-clipart-1.jpg

Periodic

Table

Higgs-Boson?

Unsolvable?

Quantum?

Empirical!

Statistical!

© 2016 HBM


6

…and is our world really “perfect”?

An Imperfect World

www.mirror.co.uk/news/uk-news/uk-weather-watch-terrifying-moment-4950461

www.thisismoney.co.uk/money/cars/article-2192993/How-compensation-pothole-damage-car.html

www.skynet.be/diensten/reizen/dossier/853527/overladen/slideshow

© 2016 HBM


7

…and do we know how imperfect it actually is?

An Imperfect World

HBM nCode GlyphWorks, Worked Example

https://en.wikipedia.org/wiki/Electronic_color_code

https://en.wikipedia.org/wiki/Mars_Exploration_Rover

www.belvalves.com/engineering-excellence/design/fea/

www.slideshare.net/DemianBorba1/ux-super-powers-with-projectcomet

Difficult to

measure

New & Unknown

Environments

Noise & Interference

Tolerance (Ω) ± 5%Statistical

Variation

© 2016 HBM


Simulation and the Design Process

http://www.altairhyperworks.com/industry/Automotive

© 2016 HBM


9

Discrete Event Systems

• Monte Carlo, Timing level simulation for circuits…

Particles

• Snooker balls, Galaxies & Space Travel, Proteins…• 00

Agent-based / Behavioural

• Market modelling, Financial modelling, Human Interactions…

Differential Systems (single variable, ODEs)

• Circuit simulation, Structural mechanics, Chemical kinetics…

Differential Systems (multiple variable, PDEs)

• Heat, Elasticity, Electrostatics, Multi-Body Physics…

Computer Games, Virtual Reality

• Entertainment, Education, Training (eg. flight simulator)…

Types of Simulation

http://medicalxpress.com/news/2014-07-proteins-scientists-drug-discovery-tool.html

www.simscale.com/blog/2016/05/update-052016-conjugate-heat-transfer/

© 2016 HBM


10Simulation and the Design Process

Design Stage Simulation Tools

SpecificationsMarket modelling

Human interactions and behaviour (Agent-based)

Conceptual

Design

Computer Aided Design (CAD)

Topology Optimisation

Engineering

Design

Finite Element Analysis (FEA)

Computational Fluid Dynamics (CFD)

Fatigue and Durability

Vibration (NVH)

Thermo-Dynamics

Multi-Body Dynamics

Multi-Physics Simulations

ManufacturingProcesses : Stamping, Casting, Forging, Welding…

Production : Planning, Scheduling, Factory design & efficiency…

Products Virtual / Augmented Reality

Evaluation Reliability, Weibull, FMEA, Predictive Maintenance

© 2016 HBM


Simulation Accuracy & Precision

http://knowyourmeme.com/photos/918810-funny-error-messages

© 2016 HBM


12Accuracy & Precision

“accuracy”The degree to which the result of a measurement, calculation,or specification conforms to the correct value or a standard.

[https://en.oxforddictionaries.com/definition/accuracy]

“precision”Refinement in a measurement, calculation, or specification,

especially as represented by the number of digits given.[https://en.oxforddictionaries.com/definition/accuracy]

https://manoa.hawaii.edu/exploringourfluidearth/physical/world-ocean/map-distortion/practices-science-precision-vs-accuracy

© 2016 HBM


13Sources of Inaccuracy

Accuracy

Software

Bug

Geometry

& Meshing

Statistical

Variability

User

Error

Assumptions

& Boundary

Conditions

Computational

Precision

Measurement

Precision

Manufacturing

& Materials

© 2016 HBM


14

A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.

[https://en.wikipedia.org/wiki/Software_bug]

Software Bug

Stay aware of announcements & communications from software companies

Update your software regularly

Contact Support and report bugs to the software companies

Validate simple calculations by hand to check you get the answers you expect

Try turning it off and on again…

https://www.linkedin.com/pulse/top-5-windows-errors-lisa-j-whelan

© 2016 HBM


15

Ways to avoid/reduce the likelihood of User Errors...

User Error

• Software training courses

• Worked Examples

• User manuals

• Online help & forums

• Demonstration videos

• Contact support hotlines

https://uk.pinterest.com/thatkelseygirl/call-center-humor/

© 2016 HBM


16

The “Dunning-Kruger effect” is a cognitive bias in which low-ability individuals suffer fromillusory superiority, mistakenly assessing their ability as much higher than it really is.

Their research also suggests corollaries: high-ability individuals may underestimate their relative competence and may erroneously assume that tasks which are easy for them are also easy for others.

User Error

[https://en.wikipedia.org/wiki/Dunning-Kruger_effect]

WISDOM(Knowledge + Experience)

High

Know-Nothing

CO

NF

IDE

NC

E

Guru

Low

Peak of “Mt. Stupid”

Slope of Enlightenment

Valley of Despair

Plateau of Sustainability

© 2016 HBM


17

By using the correct resources and attending training courses, the impact of the “Dunning-Kruger effect” can be significantly reduced and the risks associatedwith “Mt. Stupid” are mitigated, but not eliminated!

“Despair” will be reduced, the path to “Enlightenment” becomes both fasterand simpler, allowing the “Plateau” to be reached much sooner…

User Error

WISDOM(Knowledge + Experience)

High

Know-Nothing

CO

NF

IDE

NC

E

Guru

Low

“Mt. Stupid” reduced

& reached sooner

“Enlightenment” happens sooner

“Despair” reduced

& reached sooner

“Plateau” reached sooner

© 2016 HBM


18

When setting up a simulation, engineers must make assumptions and choose boundary conditions for the calculations (eg. RBE2 or RBE3?)

Engineers need to ensure the appropriate modelling techniques are used, and should therefore understand the physics and maths being used to drive the simulation.

Assumptions & Boundary Conditions

• Best-practice guides and theoretical training are crucial; available fromboth software vendors and independent providers (eg. NAFEMS)

• Advice can be found in industry standards and approved methods

http://www.quickmeme.com/meme/36jk4r

© 2016 HBM


19

A truly “accurate” geometry would require every individual atom to be defined and for the mesh to represent the crystal lattice!

Therefore, geometries are a compromise between different simulation methods, the limitations of mathematical analysis and final manufacturing techniques…

Geometry

Engineers Simulation Geometry

Design CAD “idealised”

Analyst FEA++ “solvable”

Manufacture Processes “feasible”

• Mathematical singularities need to be avoided by analysts. This can result ingeometry being altered purely for simulation purposes leaving a model that isno longer “accurate” when compared to the original design

• Simulation geometries are often simplified using things like shells(instead of solids) or symmetry to reduce calculations

• Manufacturing has it’s own limitations, tolerances and inaccuracies

http://machinedesign.com/fea-and-simulation/good-solid-modeling-bad-fea

© 2016 HBM


20

Manufacturing processes are not perfect, and each carries its own tolerances, geometric precision, and potential for inaccuracy when compared to simulation…

Manufacturing Tolerances

Machining tools and equipment have tolerances

Casting can leave flaws, porosity and inclusions

Welding is often considered an “art” performed by skilled workers

Residual stresses are left over from the manufacturing processes

Surface finishes can vary (eg. from worn out cutting tools or casting methods)

http://www.flexbar.com/shop/pc/SURFACE-ROUGHNESS-STANDARDS-SET-p4484.htm

© 2016 HBM


21

Often overlooked and expensive to obtain; good materials data is vitalfor performing an accurate simulation…

Materials Data

Data Book / Standards

Supplier (general)

Supplier (batch)

Material Testing (coupon)

Material Testing (component)

Online Did you read off a pixelated image?

When was the data generated?

How consistent is their product?

Includes manufacturing stresses…

Pay for peace of mind!

Do they have material test certification?

http://www.clipartkid.com/money-clip-art-BZpa5B-clipart/

© 2016 HBM


22

Materials themselves are inherently variable, and for most engineering purposes theyare modelled empirically to account for inconsistencies on an atomic level

Materials Data

Solids

Grain shapes and sizes, crystal defects and dislocations…

Liquids & Gasses

Brownian motion affecting enthalpy, entropy…

http://dtrinkle.matse.illinois.edu/MatSE584/metamod/preface.html

http://jimmyakin.com/2012/05/brownian-motion-explained.html

HBM nCode, Practical Fatigue Theory (Training)

© 2016 HBM


23

Assumptions and boundary conditions are often derived from measured data which itself is a source of inaccuracy and imprecision, carried through to simulation results…

Measurement Accuracy & Precision

When collecting data to be used as part of a simulation, it is important to choose the appropriate sensor type/sensitivity and acquisition hardware settings in order to understand and be aware of tolerances, limits and potential errors…

Sample

Points

Nyquist sampling rate

= 2 x Frequency

https://www.hbm.com

https://en.wikipedia.org/wiki/Electronic_color_code

© 2016 HBM


24

Even the best data acquisitions kits rely on being handled correctly…

Measurement Accuracy & Precision

Attachment (eg. strain gauge bonding)

Positioning (eg. hotspot)

Calibration (eg. thermal)

Interference & Noise (eg. electrical spikes)

Old / Broken sensors (eg. drift)

Raw data should be treated with caution, understood and ‘cleaned’ before use…

HBM nCode GlyphWorks

http://bradburybrothers.com/does-your-air-conditioner-have-faulty-wiring/

© 2016 HBM


25Significant Figures & Decimal Places

0.978V x 0.187A = 0.182886W ???

0.183W ???

0.18W ???

0.9775 ≤ 0.978 < 0.9785 0.1865 ≤ 0.187 < 0.1875

0.9775 x 0.1865 = 0.18230375 0.9785 x 0.1875 = 0.18346875

0.18230375 ≤ 0.978 x 0.187 < 0.18346875

© 2016 HBM


26

Question: How many significant figures should you use after multiplying “float x double” ???

Computational Accuracy

“float” 8 s.f.π = 3.14159265358979323846…

float (π) = 3.1415926XXXXXXXXXXX…

“double” 16 s.f.π = 3.14159265358979323846…

double (π) = 3.141592653589793XXXX…

IEEE 754 : Standard for Binary Floating-Point Arithmetic

© 2016 HBM


27

Many of the previous sources of inaccuracy are themselves subject tostatistical variability…

Statistical Variability

Source Statistical Variability

Assumptions &

Boundary Conditions

Physical differences in the user (eg. size & weight)

Different habits and usage profiles (eg. extreme users)

Environments will vary (eg. hot summer)

MeasurementsHardware tolerances (+/-) mean that measurements are never completely

exact and the ‘true’ value will always be obscured by background noise

ManufacturingProduction tolerances (+/-) mean that some components will be slightly

larger or smaller than the design, and could contain flaws

Materials Inherent statistical variation, therefore modelled empirically

ComputingDifferent computers will produce different ‘floating point digits’

therefore giving slightly different answers

http://sittingsafely.com/adjustment/

© 2016 HBM


28Sources of Inaccuracy

Accuracy

Software

Bug

Geometry

& Meshing

Statistical

Variability

User

Error

Assumptions

& Boundary

Conditions

Computational

Precision

Measurement

Precision

Manufacturing

& Materials

© 2016 HBM


29Usefulness -vs- Accuracy

Having explored the variety of inaccuracies that can affect a simulation, it is

important to consider how much time and effort (aka. money!) must be spent

to achieve 100% accuracy, and how useful this might actually be…

Ac

cu

rac

y

Time Accuracy

Us

efu

lne

ss

Harty, D. The Myth of Accuracy, The Journal of the Engineering Integrity Society, January 1999

Usefulness = Accuracy / [-ln(Accuracy - 1]

Accuracy = 1 - e(-duration)

© 2016 HBM


Simulation Fidelity

https://minecraft.net

© 2016 HBM


31

“fidelity”The degree of exactness with which something is copied or reproduced

[https://en.oxforddictionaries.com/definition/fidelity]

Simulation Fidelity

“real answer”

Mesh Density

Re

su

lt

• Compromises between fidelity of the model’s geometry/mesh and the available

computing power (time/cost) must be made for any simulation.

• It is impossible to get a ‘perfect’ answer because individual atoms would need to be

defined in the geometry and the mesh would have to be infinitely small!

• There is no point creating a fantastic mesh if others inputs are inaccurate!

www.comsol.fr/multiphysics/mesh-refinement

© 2016 HBM


32Simulation Fidelity

Simulation

FidelityVariables

Geometry

Measurement

Accuracy & Precision

Calculation Precision &

Correct Software Use

Statistical

Variability

Manufacturing


Physics

& Maths

Assumptions



© 2016 HBM


33Simulation Fidelity



Simulation

FidelityVariables

Geometry

Assumptions

UNDERSTAND

INFORMED

DECISIONS

Measurement


Calculation Precision &

Correct Software Use

Statistical

Variability

Manufacturing


Physics

& Maths

© 2016 HBM


Verification & Validation (V&V)

© 2016 HBM


35Verification & Validation (V&V)

“verification”The process of determining that a model implementation and its associated data accurately represent the developer’s conceptual description and specifications.

“validation”The process of determining that a model implementation and its associated data accurately represent the developer’s conceptual description and specifications.

DoD Modelling and Simulation Verification, Validation and Accreditation

(DoD instruction 5000.61; Dec 9, 2009)

http://www.dtic.mil/whs/directives/corres/pdf/500061p.pdf

How to Perform Credible Verification,Validation, and Accreditation for Modeling and Simulation, STC April 2005 (Dr. DA Cook and Dr. JM Skinner)

© 2016 HBM


36Validation & Verification (V&V)

SIMULATIONINPUTS OUTPUTS

PHYSICAL

TESTINGINPUTS OUTPUTS

© 2016 HBM


37Verification

"Engineers are not superhuman.

They make mistakes in their assumptions, in their calculations, in their conclusions.

That they make mistakes is forgivable; that they catch them is imperative.

Thus it is the essence of modern engineering not only to be able to check one's own

work but also to have one's work checked and to be able to check the work of others.”

Henry Petroski

© 2016 HBM


38Validation

There are a wide variety of different validation methods that usually involve physical testing in order to collect data and validate the results of a simulation…

Test tracks Wind tunnels Shaker tables Static loading Durability testing Crash tests Pressure tests Environmental tests

etc…

© 2016 HBM


39Validation

Does the FE-model accurately represent the real component ???

VIRTUAL Strain Gauge REAL Strain Gauge

Cross-Plot

Validation

http://www.caranddriver.com/bmw/2-series

Vehicle image for illustrative purposes only

© 2016 HBM


40

Use statistical methods such as “Design Of Experiments” and “Weibull” analysis to help with optimised testing schedules and for reliability modelling from physical tests…

Validation

© 2016 HBM


41Verification & Validation

How to Perform Credible Verification, Validation, and Accreditation for Modelling and SimulationDr. D A Cook & Dr. J M.Skinner; STC, 18 April 2005http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.640.7396&rep=rep1&type=pdf

© 2016 HBM


www.hbmprenscia.com

Documents

Simulating an Imperfect World - Engineering Integrity …e-i-s.org.uk/wp-content/uploads/2017/01/Peter-Lavelle.pdf · conditions for the calculations (eg. RBE2 or RBE3?) ... Usefulness