ЄDINBVRGH

Dimensions of Credibility in Models and Simulations

Martin J. Steele, Ph.D.National Aeronautics and Space Administration (NASA)

IT-C1 / Kennedy Space Center, FL 32899Martin.J.Steele@nasa.gov

June 2008ЄDINBVRGH

08E-SIW-0762

ЄDINBVRGHOverview

• Recent Historical Perspective• The Idea of Credibility in M&S• Dimensions of Credibility• Concerns• Findings• Summary• Final Thoughts

08E-SIW-0763

ЄDINBVRGHM&S Standard Development

2 Credibility Scales 1 New Credibility Scale

InterimNov 06

Final SubmittedNov 07

CAIBReport

NASAOCE

Direction M&SLiterature

DiazReport

MgtDecision Maker

Interviews

ExternalEfforts

NASA-wideFormal Review

PilotStudies

08E-SIW-0764

ЄDINBVRGHRecent Historical Perspective

Inside NASA• The NASA Standards

for Models & Simulations – 2007 Summer Computer Simulation Conference– Interim Version

• Development of NASA’s Models and Simulations Standard – 2008 Spring Simulation Interoperability Workshop– Final Version

Outside NASA• DoD

– M&S Validation (VPMM)– M&S Quality

• DoE – Predictive Capability Maturity Model (PCMM)

• ASME – M&S V&V in Computational Solid Dynamics

• AIAA – M&S V&V in Computational Fluid Dynamics

08E-SIW-0765

ЄDINBVRGHGoals for Excellence in M&S

Goals of the M&S Standard• Ensure that the credibility of M&S results

is properly conveyed to those making critical decisions

• Assure that the credibility of M&S meetsthe project requirements

Additionally,• Form a strong foundation for disciplined

development, validation and use of M&S

• Include a standard method to assess the credibility of the M&S presented to the decision maker

ЄDINBVRGH

The Idea of Credibility in M&S

08E-SIW-0767

ЄDINBVRGH

Accreditation

Verification Validation

Development© Sargent, R. G. (c. 1980).© Sargent, R. G. (c. 1980).

Development Foundation

08E-SIW-0768

ЄDINBVRGHOperational Foundation

Obtaining Credible Results Requires:

• ‘Good’ Input• Understanding of

Uncertainty• Sensitivity Analysis

Ver Val

Development

InputPedigree

UncertaintyQuant. Robustness

Operations

Accreditation

Results

08E-SIW-0769

ЄDINBVRGHSupporting Evidence

Adding to Credibility:• Past Use• M&S Mgt• Developer / User /

Analyst Qualifications

Ver Val

Development

InputPedigree

UncertaintyQuant. Robustness

Operations

ResultsAccreditation

UseHistory

ModelMgt

PeopleQual.

Supporting Evidence

08E-SIW-07610

ЄDINBVRGHBreadth to Consider

• Problem Statement• System Understanding• Modeling Choice• Conceptual Model

Accreditation

Ver Val InputPedigree

UncertaintyQuant. Robustness Use

HistoryModelMgt

PeopleQual.

Development Operations Supporting Evidence

Results

System Understanding

Real World System

Computational Model

Conceptual Model

SystemAnalysis

Verification

ValidationValidation

ImplementableConceptual Model

Coding theModel Verification

Validation

DetailedUnderstanding

ЄDINBVRGH

08E-SIW-07612

ЄDINBVRGHVerification & Validation

Verification• Structure• Flow• Fidelity

• How:– Comparing to

Conceptual Model– Code Tracing– Primitive Tests– Min/Max Value Tests

Validation: “… determining the degree to which a model or a simulation is an accurate representation of the real world …”

08E-SIW-07613

ЄDINBVRGHInput Pedigree

Input:• Source

– Notional– Subject Matter Expert– Applicability to current

problem• Referent Quality

relative to current problem

– Referent System– Referent Environment

• Quantity

• Authoritative Data

Input Form:• What’s the character of

your analysis?

– Average

– Uniform

– Triangular

– Curvilinear

08E-SIW-07614

ЄDINBVRGHAccuracy & Uncertainty

Accuracy:

Uncertainty:• Sources• ‘Size’• How Confident

• Epistemic– Reducible– Subjective– Model Form– Incomplete Information

• Aleatory– Irreducible– Variability– Inherent – Stochastic

True Value ‘Modeled’ Value

Uncertainty in‘Modeled’ Value

Uncertainty inTrue Value

08E-SIW-07615

ЄDINBVRGHRobustness

Robustness of Results, i.e.,Sensitivity of:

• The Real World System (RWS)

• The M&S

Not a Good Situation

M&S is not robust, butRWS is

- Validation Issue- Results will be overly

conservative

Best Situation

RWS is robust (Insensitiveto Changes)

&the M&S matches

the RWS

Worst Situation

M&S shows aRobustness not present

in the RWS- Validation Issue

- M&S not so useful

OK Situation

RWS is sensitive to change& the M&S matches

the RWS

RWSM

&S

InsensitiveTo Changes

SensitiveTo Changes

SensitiveTo Changes

InsensitiveTo Changes

Better

08E-SIW-07616

ЄDINBVRGHUse History & Management

Use History:• Similarity of Uses

– Analogous Systems– Exact Systems

• Length of Time in Use– Just Developed

• Just Updated– Long-Term Successful

Use

M&S Management:• Models & Data under

Configuration Control• Models are

– Maintained– Sustained

08E-SIW-07617

ЄDINBVRGHPeople Qualifications & Tech Review

People Qualifications:• Education• Training• Experience

• In M&S• With the Modeled (Real

World) System

Technical Review:• When accomplished

– During M&S Development– During M&S Operations

• Qualifications & Independence of the ‘Peer’Review Group:– Self– Internal Organization– External– Non-Expert to Expert

• Level of Formalism– Planning– Documentation

Ver Val

Development

InputPedigree

UncertaintyQuant. Robustness

Operations

InputPedigree

UncertaintyQuant. Robustness

Operations

ЄDINBVRGH

When to Apply the Standard?

08E-SIW-07619

ЄDINBVRGHScope of the M&S Standard

• Standard covers the use of M&S affecting:• Human Safety• Mission Success

• The focus is on M&S for flight and ground support projects– Operations– Test & Evaluation– Manufacturing and Assembly– Design and Analysis

Critical Decisions

As defined by each Program

4: Controlling 3: Significant 2: Moderate 1: Minor

M&S Results Influence

0: None IV: Negligible III: Marginal II: Critical I: Catastrophic

Project Consequence

Sample Risk Matrix

08E-SIW-07620

ЄDINBVRGHBasic Ideas

• Documentation of M&S Activities (Sections 4.1 – 4.6)

• Credibility Assessment (Section 4.7 & Appendix A)

• Reporting to Decision Makers (Section 4.8)– M&S Analysis Results– A statement on the uncertainty in the results– Credibility of M&S Results– Identify

• Unfavorable outcomes• Violation of assumptions

– Unfavorable Use Assessment• Difference Between V&V & Use Assessment

08E-SIW-07621

ЄDINBVRGHUse Assessment

Expected OutputRange

Intended Input Domain

ValidatedInput Domain

ValidatedOutput Range

Note – this is a 2-dimensional example of a potentially multi-dimensional input domain & multi-dimensional output range

ЄDINBVRGH

Sommaire& Future Directions

08E-SIW-07623

ЄDINBVRGHFuture Directions

• Track 1 – Internal – Deploy– Use of Standard /

Collection of Data on Use– Assess– Revise

• Track 2 – M&S Discipline Specific

– Develop Guides (RPGs, Handbooks)

– For each M&S type in relation to the NASA M&S Standard

• Track 3 – External Collaboration

– Other M&S Standards/Guides

– Professional/Academic Organizations

ЄDINBVRGH

Broadening the Perspective of M&S V&V

and Simplifying It !

08E-SIW-07625

ЄDINBVRGHNetwork Layered Protocol Approach

Like the Layered Network Protocol

Model

08E-SIW-07626

ЄDINBVRGHLayered M&S View(Influences in M&S Results)

Computer Hardware

Operating System Software

Application Software

Model / Simulation

M/S Input M/S OutputM&S V&Vand

Credibility Assessment

Analyzing Outputincluding Post-Processing of Output Data

Industry Standardsand

Broad Use

User Inputincluding

Run Setup

Need for a Clearinghouse for Commercial & Open Source M&S Languages & Application Software

ЄDINBVRGH

Final Thoughts

08E-SIW-07628

ЄDINBVRGHConcerns & Findings

• General Concerns– Broad Applicability– Subjectivity– Software Similarity– The Analysis vs. The

People

• If we don’t have this Standard– Many varieties of

Credibility Assessment will be are being presented

• Findings– Yet Another Standard– Terminology

(Denotations & Connotations

• Types• Purposes• Education, Experience,

& Training

08E-SIW-07629

ЄDINBVRGHWhy doesn’t/can’t the Software Engineering Req’ts do the job?

• Models and simulations may be implemented in software, butthey encompass much more than software engineering

• The SW Engineering NPR (7150.2) does not address many issues critical for M&S, e.g.,– development of models– validation against experimental or flight data– uncertainty quantification– operations and maintenance of M&S

• Models are sometimes developed in hardware– Physical Models (Scaled)– Analog Models

SoftwareEngineering

Models &Simulations

08E-SIW-07630

ЄDINBVRGHComparison

Software• Performs a task within a

system

• Purpose – performance of tasks/functions for a system

• Requirements are functionsthe s/w shall perform– More discrete functions in the

code (function-centric)• Software shall do this• S/W shall do that• Etc.

• Traditional s/w functions either work or don’t work– Typically, there is no

uncertainty

• Totally focused on Software Development

M&S• Provides a representation of a

system

• Purpose – Analysis orRepresentation of a system or of system functions for insight

• Requirements are behaviorsthe simulation model shall exhibit– System behavior-centric

• Includes more ‘assumptions’how different parts of the simulation model work– Typically includes

uncertainty in the behavior of the simulation

• Must include requirements (& reporting) on M&S Use

08E-SIW-07631

ЄDINBVRGHSoftware & M&S

• You can follow software standards correctly in model development– BUT STILL produce

inaccurate or miss-leading results

• Software Standards do not deal with Software (or Model) use !

08E-SIW-07632

ЄDINBVRGHFinal Statements

• Communicating Results• Framework for Discussion of Results Credibility• Credibility Factors Beyond VV&A

– With dimensions composing each factor of M&S Credibility

• It is:– High Level– Broadly Applicable (To All M&S Types!)

Accreditation

Ver Val InputPedigree

UncertaintyQuant. Robustness Use

HistoryModelMgt

PeopleQual.

Development Operations Supporting Evidence

Results

Technical Review

ЄDINBVRGH

Thank You …

ЄDINBVRGH