Life Cycle Risk Management in a Product

Tailoring Environment

David R. Pinkley

Page 2

Life Cycle Risk Management Agenda

• Life Cycle Risk Management

• Product Tailoring Impact

• Architecture Optimization

• Heritage Reuse

• Risk Tracking

• Integration Readiness

• Anomaly Risk Rating

• Cumulative Risk Management

Normalcy Bias: Lack of exposure to failure and small

sample size of operating hours:

• Rejection of proposed failure modes

• Seizing on any ambiguities to infer less credibility

• Interpretation of warnings in the most optimistic way

Bounded Rationality: Decision-making, rationality of

individuals is limited:

• By information

• Cognitive state

• Finite decision times (Herbert A. Simon)

Epistemic Failures due to erroneous technological

assumptions, even thought there were good reasons to

hold that assumption. (John Downer)

Page 3

Lifecycle Risk Management Components

Establish Baseline and Assess Compliance

Residual Risk and Uncertainty Management

Integration Readiness and Anomaly Rating

Cumulative Risk Analysis

Pre-Award!RFP !Selection

Reqmts - Design!SRR!PDR!CDR

Build and Test!MRR! TRR!PER

Sell-Off !PSR !MRR

Operations !FRR !ICR

" Development" Implementation" Risk Profile/Strategy

Architectural Optimization

" Resource" Schedule" Technical

Heritage Reuse Compliance

" Baseline" Realized" "Unknowns Bounding

Risk ID, Tracking, Control

" Next Step Integration" Product Liens" Constraints

Integration Readiness

" Severity" Cause" Corrective Action

Anomaly Risk Rating" Residual Risk" In-Phase Burn Down" Cumulative MS Impact" Process Effectiveness

Cumulative Risk Tracking

Establishing A Compliant Baseline

Page 5

Process and Product Architectures Risk Exposure

• Type 1: Large risk exposure — Factors: Longest Mission Length, Custom Development

— Overlapping Mitigation Actions

• Type 2: Moderate risk exposure — Factors: Long Mission Length, Heritage Development

— Part Quality/Margins/Redundancy

• Type 3: Low risk exposure — Factors: Short Mission, Heritage Development

— Part Quality/Test Actions

• Type 4: Minimum risk exposure — Factors: Non-flight prototype development

— Safety and Interface Focus

Product Type Risk Management has Demonstrated Equivalent Mission Success

ResidualEngineering PrototypeDevelopment

Residual - Single string- Streamlined Process

ResidualInternal Core BPS STDs

Mitigated - Interfaces- Safety

Mitigated - Part Quality- Test Margins- MA Audits- Supplier Risk- Heritage- Internal MRL

Mitigated - Minor SPFs- Margin- MA- V&V- L2/L2- parts- Supplier risk- Heritage- Internal MRL

Mitigated- No SPFs- Large Margin- MA & MS- Full V&V- L1/L2 parts- Supplier OS- TRL assured- MRL assured

Type 4Type 3Type 2Type 1

Residual Minimal

Mission Success Assurance Activities

Pro

du

ct

typ

e t

ota

l ri

sk e

xp

osu

re

Pro

du

ct

Typ

e

Investm

en

t

Mission Success

T1

T2

T3

T4

Cost

IR&D

FlightProductTypes

Probability of Success

Page 6

Mission Class Risk Surface Legend

Radical Spokes Vector Characteristic

Risk

Management

· People: Risk Process Informal, Sparse documentation

· Programmatic: Risk Process cost/schedule/technical focus

· Mission Success: Programmatic Plus Residual Risk Management

Supplier

Interaction

· COTS: Buying product as catalog item with little data

· Insight: Formal Data available throughout development

· Oversight: Customer/Supplier interactive development

EEE Parts

· COTS: Consumer commercial and Industrial Grade

· L3: Hi-Rel parts with screening but little qualification data

· L2: Military grade parts will full screening and Qual program

· L1: Space grade parts with delta screening and qualification

Complexity · Single String: Simplex assemblies performing mission

· Block Redundant: Parallel active and standby assemblies

· No SPFs: No simplex assemblies beyond DFMR items

Product Type Instantiation Optimizes Development

Architecture • Critical evaluation

— Programmatic Constraints: Acquisition, Requirements, Funding

— Mission Needs: Reliability, Life, Performance

— Acquisition Risk Strategy: Mission Class, Mission Type

Optimal Development Architecture Translates Into Managed Risk and Controlled Cost

SupplierInteraction

EEEParts

Complexity

L2 L3 COTS

BlockRedundant

SingleString

Insight

COTSProgrammatic

People

Risk Management

MissionSuccess

Oversight

NoSPFs

Minimum Practical

Risk L1

Page 7

Category Process

Program

Execution

1 Design Assurance

2 Requirement Analysis and Validation 3 Parts, Materials and Processes 4 Environmental Compatibility 5 Reliability Engineering 6 System Safety 7 Configuration/Change Management 8 Integration, Test and Evaluation

Risk, Oversight

and Assurance

9 Risk Assessment and Management 10 Independent Reviews

11 Hardware Quality Assurance 12 Software Assurance 13 Supplier Quality Assurance

Triage, Information &

Lessons

Learned

14 Failure Review Board 15 Corrective/Preventative Action Board

16 Alerts, Information Bulletins

Product Type Instantiation Optimizes Mission Success

Processes

• Process Application Level: Inductive/Deductive Analysis

• Rigor: Patent Defect Inspections, WCCA Circuits

• Oversight: Independent Internal and Customer Assessments

• Relationships: Test level, Build-in vs. Procedural

Optimal Mission Success Processes Avoid, Eliminate, Precipitate, & Mitigates Risks

Page 8

Early and Continual Assessment of Hardware-Software

Reuse Key to Fulfilling Heritage Promise

Assessment of TRUE Heritage Key to Managing Capability Based Risks

Residual Risk and Uncertainty Management

Page 10

Residual Risk Augmented Classical Risk Management

Augmented Risk Management

o ID, Analysis, Plan, Track, Control

o Acquisition, Baseline Constraints, Execution

Systems Engineering

o Risk Interfaces and Relationships

o Product Type Risk Exposure Consequence

Subcontract Management

o Capture Subtle Requirement and Product Changes

Peer, Life-Cycle Gate, & Milestones

o Build Upon the Cumulative Risk Posture

Program Failure Review Board

o Root Cause Complexity and Systemic Risks

Metrics

o Trends and Outliers

Residual Risk Management Ensures Baseline Compliance

Page 11

Managing Uncertainty Across Product Types

Retired Risks No Residual Risk Artifacts

Known-

Knowns

Risk Artifacts

· Test as you Fly Validation

· Demonstrated TPM Performance

· Flight or test-validated analysis,

simulations and models

· Incremental knowledge

Buildup

· Complete verification and

validation

Open Risks

Uncertainty

Uncertainties

Open Residual Risks Risk Handling

Known-

Unknowns

Accepted Risk

· Analysis / test limitations

· Unverified Models/Simulations

· Envelope expansion

· Unverified failure modes

· Evaluate Deltas due to

o Baseline limitations

o Margin gaps

o In-complete V&V

o Analysis thoroughness

Unknown-

Knowns

Execution Risk

· Miscommunicate test/analysis

· Understanding of data/ envir

· Poor documentation combined

with loss of corporate memory

· Program communications

/ data sharing

· Incremental knowledge

build-up w/ trending

Unknown-

Unknowns

Hidden Risk

· Bad assumptions

· Unfinished foundation research

· Untested new environments

· Inadvertent operations outside of

limits

· TRL level 6 by PDR

· Envir analysis/test rigor

· Simulators & test-beds

fidelity, TAYF

· Design Margins

Uncertainty Management Core to an Effective Risk Management Strategy

Page 12

Design/Production Flow-down

- Process Requirements

- Guidelines/ Lessons Learned

- Workmanship

- Part/ Material StandardsFMEA,FTA,PRA

- Redundancy, Cross- strapping

- Graceful Degradation

- SPF CIL Mit igat ion

- Common CauseMargins

- Derating, EOL, LLIs, BudgetsInspection and Screening

- Patent Defect

- Latent Defect

- Infant Mortality Burn- in

- Operational Profi le ConfidenceQualification

- Environmental Tolerance

- Life Characterizat ion

- Radiat ion Characterizat ionReliability Growth

- Categorizat ion, Phase, TrendingProcess Capability

- Control with Randomness

- Coverage, Completeness

- Meets User Expectat ions

Safety Compliance VerificationHazard Analysis, Inhibit Design, FTA, Flight/Operations/Range Safety

System Safety Engineering

Fault Tolerance, SPFs, Lifetime, Deratings, Circuit Margins

Reliability & Maintainability

PMPCB, Qual/Screening, Radiation, GIDEP, Compatibility, Allowables

Parts, Materials, & Processes

Workmanship, Process Control, Non-conformances, Inspection

Hardware Quality Assurance

CMMI L3, Rel iabi l i ty, Safety, Firmware, Peer Review, IV&V

Software Quality Assurance

Fault Tolerance/Avoidance Balance

No Infant Mortality & Wearout

Processes Controlled, No Latents

S/W Maturity,Test Effectiveness

SMA DRIVING ACTIVITIES DISCIPLINE PRODUCT

LEVERAGE

- Equipment & personnel hazards- Pyros, batteries, mechanisms

- Anticipated/unanticipated faults- Random & environmental origins

- Screening/Qual exceptions- Degradation, Contamination

- Process control & capability- Latent and patent defects

- Residual defects: phase, interval- Test efficacy, saturation

Risk Avoidance/Mitigation

Process/Product Risk Exceptions Management

SME Residual Risk Identification, Management, Elimination/Mitigation

Integration Readiness and Anomaly Ratings

Page 14

Next Step Readiness Assurance Manages Risks and

Supports Verification

Integration and Test Readiness Verification (I&TRV): Next level integration readiness

Analyses: complete, Production: current, Non-conformances: addressed, Testing: compliant

Mission Assurance/Systems Engineering: Manage residual risk and requirement verification

Subcontracted Hardware: Integrate I&TRV with the pre-ship review process

I&TRV Manages Integration Risk In-Phase with the Development

- Drawings, Specs, Engineering- Change Proposals- PFS, SOW, ICD's- Waivers, MRB/FRB Results- Previous Unit Risks- Environmental Test Results- Reliability/Parts/Design Analysis- Telemetry/Calibration Data- Mass Data- Operational/Handling Constraints

Review Items

- Adequate Compliance Testing- STE and Documentation Readiness- Waivers/Liens Closure Plan- Receiving Organization Readiness- Pre-integration Critical Items- Operations and Handling Constraints

I&TRV Assessment

Page 15

Failure Effect Rating

(excluding redundancy) Failure Cause/Corrective Action Rating

Severity R R Cause/Corrective Action

Negligible (N) 1 1 Known cause/certainty of corrective action

(No residual risk)

Significant (S) 2 2 Unknown cause/effective corrective action

(No residual risk)

Catastrophic (C) 3 3 Known cause/uncertainty in corrective action

(Some residual risk)

High Priority 4 Unknown cause/uncertainty in corrective action

(Residual risk)

Risk Rating and Tracking of Anomalies

• What: System/Hardware/Software/Test Anomaly and Non-Conformance Reports

• Capture: Anomaly/Failure Effect; Current Active Cause and Corrective Action

• Residual Risk: Post investigating/maturity transferred to program risk

• Ratings programmatically prioritizes anomalies and determine risk to mission success

Risk Rating and Essential Tool in Anomaly Residual Risk Burn-Down

Cumulative Risk Analysis

Page 17

0

1

2

3

4

5

6

7

8

9

10

-3

2

7

12

17

22

27

32

37

42

47

Pro

posa

l

Plann

ing

Rqm

ts D

ef.

Prel D

esign

Det

ail D

esign

Pre

-Fab

Fab &

Ass

y I&

T

Del &

Sup

port

Pos

t Deliver

y

Es

ca

pe

s

Sig

nif

ica

nt

Ris

ks

Life Cycle Program Phase

Risk Closure

High Risks

Moderate Risks

Low Risks

Total Risks

Process Escapes

Cumulative Risk Management

• Risk Mitigation Efficacy: Ensuring a “low” risk profile, eliminating and controlling risks

• Risk Triad: Programmatic (cost and schedule), Technical, and Cumulative residual risk

• Metrics: (1) Risk identification, (2) Burn-down efficacy, (3) Orthogonal in-phase detection,

and (4) Risk escapes

Cumulative Product & Process Assessment of Contribution to Mission Success