SynGenics Corporation 72 E Granville Road Worthington OH 43085 614.846.1804 [email protected] ©2003 SynGenics Corporation. All rights reserved. The Use

SynGenicsCorporation 72 E Granville Road • Worthington OH 43085 • 614.846.1804

[email protected]©2003 SynGenics Corporation. All rights reserved.

The Use of Multicriterial Optimization Analysis and Sensitivity as a Measure of

Risk In Aerospace Systems Development

Carol Ventresca and George A. Richards, Ph.D., SynGenicsMarvin C. Gridley and Gregory A. Addington, Ph.D. AFRL/VAA

Presented at INFORMS Annual Meeting, Atlanta, Georgia, 19–21 October 2003Document Number ASC 03-2322. Cleared for public release.

2SynGenics

Corporation

© 2003 SynGenics Corporation. All rights reserved.

Presentation Outline

Structurally Integrated Compact Inlet Technology (STRICT)

Affordability Process and Metrics

Sensitivity Analysis for Robust Design

Results

Conclusions

Status

3SynGenics

Corporation


Goals of STRICT Program

Duct Configuration Best propulsion

system performance 100% LOS blockage L/D goal = 2.5,

threshold = 3.0 ACIS class total

pressure recovery Minimize steady state

distortion Minimize first 4

harmonics Consider structural

integration

Flow Control System Minimize flow/engine

impact Minimize integration

impact, i.e., ducting, structural

Provide best-value technology advancements, while identifying and managing cost and risk drivers as an integral part of the program

4SynGenics

Corporation


Potential Customers for STRICT

Uninhabited Air Vehicles Sensor Craft: high altitude, long endurance,

moderate LO UCAV: DARPA/AFRL UCAV mission, lower cost,

higher performance, more LO

Future Strike M ~ 2.5 long range, high performance, very LO

Transport NASA BWB SOF/ATT Highly integrated high BPR engines Very LO for USAF

5SynGenics

Corporation


Affordability Process

Capture the Voice of the Customer Identify Customers Capture Each Customer’s Criteria Synthesize Criteria into a Composite Set to Guide

the Program Define Exit Criteria

Select and Optimize Solution Concept Identify and Select Best Solution Principle

Define Alternatives Evaluate Each with Respect to Criteria Perform Value Analysis, Create Scorecard

Optimize Selected Alternative

6SynGenics

Corporation


Focus on Affordability

Leads to Better Systems

Performance: Does What the Customer Wants it to Do

Reliability: Meets or Exceeds NeedsCost: At a Price the Customer Will PaySupportability: Available for Use within Resources

More Efficient Design Process

Directed by Air Force Research Laboratory (AFRL) & Department of Defense (DoD)

7SynGenics

Corporation


Block Diagram of the Process Model

Courtesy Air Force Research Laboratory

8SynGenics

Corporation


IPPD Process Artifacts

Quantitative Predictive MeasuresResponse Values Related to CriteriaQuantified Uncertainty and Risk

Documented Criteria (e.g., Exit Criteria)

Worksheets and Scorecards

Relationships: Factors and Responses

Graphics

9SynGenics

Corporation


Requirements/Criteria

Criteria, a.k.a. “Requirements”Carefully DocumentedFor Each CustomerFor the Project (“Constructed Customer”)

Captured in Requirements Matrix or HOQ

Are Specific to Customer

Defined to Be Measurable

10SynGenics

Corporation


Measures of Merit

Two Predictive Measures for Each AlternativeDesirability: a Measure of Customer

SatisfactionRisk: Probability of Failure

Calculated at Three LevelsWith Respect to Each CriterionWith Respect to Each Type of CriteriaOverall

For Each “Customer”

11SynGenics

Corporation


Worksheet

PresentsOne AlternativeAgainst All Criteria of a Single TypeFor One Customer

Metrics for Each CriterionExpected Value Measure of Variabilityd Expected Desirability Probability of Failure to Meet Threshold

12SynGenics

Corporation


The Value Scorecard

ComprisesOne Scorecard for Each TypeThe Affordability Scorecard

Relates to a Single Defined Customer

The “Constructed Customer” with its Composite Set of Requirements Is Key to Project Management

13SynGenics

Corporation


Value Scorecard (Low Level)

Presents Multiple AlternativesFor Criteria of a Specified TypeFor a Single Customer

Metrics for Each CriterionExpected Valuedi Expected Desirability

i Probability of Failure to Meet Threshold

14SynGenics

Corporation


Value Scorecard (Low Level)Continued

Aggregates Metrics across TypeDj Composite Desirability for Type j

j Probability of Failure to Meet One Threshold for a Criterion of Given Type j

Called “The ‘Type’ Scorecard”e.g., “The Performance Scorecard”

Facilitates Identification of Discriminators, Risk Drivers, and Technical Challenges

15SynGenics

Corporation


Value Scorecard (High Level)

“Affordability Scorecard” Presents Multiple AlternativesFor All TypesFor a Single Customer

Metrics for Each Criterion Type Dj Composite Desirability for Type j

j Probability of Failure to Meet One Threshold for Criteria of Type j

16SynGenics

Corporation


Value Scorecard (High Level)Continued

Top-Level Aggregate Metrics CSI Customer Satisfaction Index =

Composite Desirability Aggregated across Types

Aff Pr(Failure) to Meet One Threshold

Called “The Affordability Scorecard”

Facilitates Identification ofCost and Risk DriversTechnical Challenges and Tradeoffs

17SynGenics

Corporation


Attributes of Criteria

ID#, Name, Description

Type (e.g., Performance, Cost)

Unit of Measure

Customer-Specific Attributes Objective Threshold(s) Priority and Weighting Factor within Type Desirability Function Weighting of Types with Respect to Each Other

Design Cost ($K)

0.0

0.2

0.4

0.6

0.8

1.0

65.0 72.5 80.0 87.5 95.0 102.5 110.0 117.5 125.0

Design Cost ($K)

Desir

ab

ilit

y

18SynGenics

Corporation


Structurally Integrated Compact Inlet Technology

Rqmt# Customer Name Priority How Measured ObjectiveLower

ThresholdUpper

ThresholdType

Category: 1. Performance

11 UCAV Vehicle Range High Ratio to ACIS Baseline 1.1 1 Performance

12 UCAV Distortion Low DPCP-DPCP(ACIS) -0.02 N/A 0.02 Performance

13 UCAV Pressure Recovery Low PR-PR(ACIS) 0 -0.035 N/A Performance

14 UCAV Fuel Consumption Med Ratio to ACIS Baseline 0.95 1.05 Performance

15 UCAV Installed Engine Thrust Med Ratio to ACIS Baseline 1.05 0.975 N/A Performance

16 UCAV RCS High Qualitative Delta 1 0 N/A Performance

17 UCAV Throat Mach Number Med Mach Number 0.7 0.6 N/A Performance

42 UCAV Inlet Structural Weight High Lbs 100 175 Performance

Category: 2. Producibility

STRICT UCAV Requirements

19SynGenics

Corporation


Structurally Integrated Compact Inlet Technology

Category: 1. Performance

42 UCAV Inlet Structural Weight High Lbs 100 175 Performance

Category: 2. Producibility

21 UCAV Producibility Med Complexity Scale: 1 to 5 1 N/A 3 Producibility

22 UCAV Fidelity of Demo Article to Final System Low Scale: 1 to 4 4 2 N/A Producibility

23 UCAV Number of Removable Breakout Pieces High Count 10 N/A 20 Producibility

24 UCAV Number of Effectors Med Count 1 N/A 54 Producibility

25 UCAV Number of Sensors Low Count 4 N/A 72 Producibility

STRICT UCAV Requirements Continued

Category: 3. Cost

31 UCAV Flow Control System Procurement Cost High $K/Aircraft 0 85 Cost

32 UCAV Inlet Structural Procurement Cost High $K/Aircraft 300 N/A 500 Cost

33 UCAV Operations and Support Costs High Percent of ACIS Baseline 98 103 Cost

43 UCAV Volumetric Efficiency (L/D Surrogate) Med Square Inches 1500 N/A 2500 Cost

Category: 5. Other

51 UCAV Meets EMD Exit Criteria Low TRL 6 4.5 N/A Other

52 UCAV Sophistication of Demo Low Scale: 1 to 5 5 3 N/A Other

53 UCAV TAD High Months from 10/1/01 48 N/A 54 Other

Category: 6. Tracking Criteria

20SynGenics

Corporation


STRICT Weighting Summary

Requirement PriorityWeight within Type

Weight for Type

Wt for Reqmt Overall

Type

11 Vehicle Range High 5.0 3 10.1% Performance12 Distortion Low 1.0 3 2.0% Performance13 Pressure Recovery Low 1.0 3 2.0% Performance14 Fuel Consumption Med 2.0 3 4.1% Performance15 Installed Engine Thrust Med 2.3 3 4.6% Performance16 RCS High 3.5 3 7.1% Performance17 Throat Mach Number Low 1.0 3 2.0% Performance42 Inlet Structural Weight High 4.0 3 8.1% Performance21 Producibility Fudge Factor Low 0.5 1.5 3.3% Producibility22 Fidelity of Demo Article to Final System Low 0.5 1.5 3.3% Producibility23 Number of Removable Breakout Pieces Med 1.0 1.5 6.7% Producibility24 Number of Effectors Low 0.5 1.5 3.3% Producibility25 Number of Sensors Low 0.5 1.5 3.3% Producibility31 Flow Control System Procurement Cost Low 1.0 2 3.8% Cost32 Inlet Structural Procurement Cost Med 1.5 2 5.7% Cost33 Operations and Support Costs High 3.0 2 11.4% Cost43 Volumetric Efficiency (L/D Surrogate) Med 1.5 2 5.7% Cost51 Meets EMD Exit Criteria Low 1.0 1 3.3% Other52 Sophistication of Demo Low 1.0 1 3.3% Other53 TAD Med 2.0 1 6.7% Other

Total 7.5 100.0%

21SynGenics

Corporation


STRICT Performance Scorecard

22SynGenics

Corporation


STRICT Producibility Scorecard

## #REF! #REF! #REF! #REF! #REF! #REF! #REF! #REF! #REF! #REF! #REF! #REF! #REF! #REF! #REF!

STRICT Scorecard: ProducibProducibility

4

UCAV

Pro

du

cib

ility

S

atis

fact

ion

Ind

ex

Tota

l Pro

du

cib

ility

Zet

a

0 Weight

Technology Alternative d

1 Config_101 2.25 0.00000 3 0.00000 14 0.00000 18 0.00000 30 0.00009 0.697 0.00009

Desirability for Config_101

2 Config_102 2.5 0.02275 3 0.00000 16 0.02275 18 0.00003 30 0.00009 0.649 0.04459


3 Config_20X 2.5 0.02275 3 0.00000 16 0.02275 36 0.02275 30 0.00009 0.605 0.06605

Desirability for Config_20X 0.592

0.914 0.900 0.968 0.2930.702

0.640 0.293

0.592 0.968 0.293

0.834 0.900

0.834 0.900

(25)

Num

ber

of S

enso

rs

(23)

Num

ber

of R

emov

able

B

reak

out

Pie

ces

(24)

Num

ber

of E

ffect

ors

0.51.0 0.5

NewProject

(21)

Pro

duci

bilit

y

0.5

(22)

Fid

elity

of

Dem

o A

rtic

le

to F

inal

Sys

tem

0.5

23SynGenics

Corporation


STRICT Cost Scorecard


STRICT Scorecard: Cost_2Cost

4

UCAV

Co

st S

atis

fact

ion

Ind

ex

Tota

l Co

st Z

eta

0 Weight


1 Config_101 59.4 0.00000 450 0.02275 101.5 0.02275 2361.24 0.11957 0.378 0.15217


2 Config_102 59.4 0.00000 425 0.00135 101.5 0.02275 2164.47 0.00092 0.548 0.02471


3 Config_20X 59.4 0.06039 385 0.00006 101.5 0.02275 1967 0.00000 0.754 0.07984


0.905 0.047 0.5690.650

0.952

0.650 0.836

0.905 0.711

0.905 0.182

(33)

Ope

ratio

ns a

nd S

uppo

rt C

osts

(43)

Vol

umet

ric E

ffici

ency

(L

/D S

urro

gate

)

1.53.0

NewProject

(31)

Flo

w C

ontro

l Sys

tem

P

rocu

rem

ent

Cos

t

1.0

(32)

Inl

et S

truc

tura

l P

rocu

rem

ent

Cos

t

1.5

24SynGenics

Corporation


STRICT Other Scorecard


STRICT Scorecard: OtherOther

4

UCAV

Oth

er S

atis

fact

ion

Ind

ex

Tota

l Oth

er Z

eta

0 Weight


1 Config_101 5.75 0.00000 4 0.00003 52 0.00571 0.899 0.00572


2 Config_102 5.75 0.00000 4 0.00003 52 0.00571 0.899 0.00572


3 Config_20X 5.75 0.00000 4 0.00003 52 0.00571 0.899 0.00572


0.986 0.817 0.900

0.900

0.986 0.817

0.986 0.817

(53)

TA

D

2.0

NewProject

(51)

Mee

ts E

MD

Exi

t C

riter

ia

1.0

(52)

Sop

hist

icat

ion

of D

emo

1.0

25SynGenics

Corporation


STRICT “Affordability” Scorecard

Affordability_2

Affordability

STRICT Scorecard - Con't: Affordability_2 Requirement Type

4

UCAV

Weight

Technology Alternative

1 Config_101 1 0.10662 1 0.00009 1 0.15217 1 0.00572 1 1 0.685 0.23249


2 Config_102 1 0.10026 1 0.04459 0.9 0.02471 1 0.00572 1.0 1.000 0.751 0.16078


3 Config_20X 1 0.11975 1 0.06605 0.9 0.07984 1 0.00572 1.0 1.000 0.809 0.23766

Desirability for Config_20X

3

0.812 0.697

Per

form

ance

Aff

ord

abili

ty

Pro

du

cib

ility

Co

st

Oth

er

0.378 0.899

1.5 2 1

0.854 0.649 0.548 0.899

0.882 0.605 0.754 0.899

NewProject

CSI T

26SynGenics

Corporation


Sensitivity Analysis

Sensitivity Analysis Was PerformedTo Identify Which Criteria Had Significant

InfluenceTo Evaluate the Influence of Weighting

FactorsTo Highlight Areas Where Special Attention

Is WarrantedIndividual Desirability

(di)

Individual Risk (i)

Type Desirability

(DT)

Type Risk (T)

Composite Desirability

(CSI)

Overall Aggregated

Risk (A)Pessimistic 0.368 0.4297 0.344 0.3782 0.670 0.3878Nominal 0.569 0.1196 0.378 0.1522 0.685 0.2325Optimistic 0.748 0.0147 0.400 0.0585 0.694 0.1571Pessimistic 0.709 0.0173 0.529 0.0405 0.745 0.1740Nominal 0.836 0.0009 0.548 0.0247 0.751 0.1608Optimistic 0.914 0.0000 0.558 0.0238 0.754 0.1600Pessimistic 0.910 0.0000 0.747 0.0798 0.807 0.2377Nominal 0.952 0.0000 0.754 0.0798 0.809 0.2347Optimistic 0.976 0.0000 0.758 0.0798 0.810 0.2377

43. Volumetric Efficiency (L/D Surrogate)

101

102

20X

27SynGenics

Corporation


Sensitivity to Criterion Response Values

RequirementPessi-mistic

NominalOpti-

misticPessi-mistic

NominalOpti-

misticPessi-mistic

NominalOpti-

misticPessi-mistic Nominal

Opti-mistic

Pessi-mistic

NominalOpti-

mistic

11 Vehicle Range 0.766 0.847 0.889 0.0912 0.0093 0.0004 0.792 0.812 0.825 0.1769 0.1066 0.0986 0.679 0.685 0.688

12 Distortion 0.929 0.978 0.995 0.0000 0.0000 0.0000 0.810 0.812 0.813 0.1066 0.1066 0.1066 0.684 0.685 0.685

13 Pressure Recovery 0.952 0.979 0.997 0.0000 0.0000 0.0000 0.811 0.812 0.813 0.1066 0.1066 0.1066 0.684 0.685 0.685

14 Fuel Consumption 0.438 0.800 0.947 0.0098 0.0004 0.0000 0.764 0.812 0.826 0.1150 0.1066 0.1033 0.670 0.685 0.689

15 Installed Engine Thrust 0.699 0.828 0.900 0.0014 0.0000 0.0000 0.797 0.812 0.820 0.1078 0.1066 0.1063 0.680 0.685 0.687

16 RCS 0.870 0.921 0.989 0.3953 0.1030 0.0118 0.804 0.812 0.817 0.3331 0.1066 0.0213 0.682 0.685 0.686

17 Throat Mach Number 0.806 0.900 0.951 0.0000 0.0000 0.0000 0.808 0.812 0.815 0.1067 0.1066 0.1066 0.683 0.685 0.685

42 Inlet Structural Weight 0.557 0.613 0.665 0.0014 0.0000 0.0000 0.797 0.812 0.826 0.1078 0.1066 0.1066 0.680 0.685 0.689

21 Producibility Fudge Factor 0.880 0.914 0.940 0.0000 0.0000 0.0000 0.693 0.697 0.700 0.0001 0.0001 0.0001 0.684 0.685 0.685

22Fidelity of Demo Article to Final System

0.805 0.900 0.952 0.0000 0.0000 0.0000 0.684 0.697 0.704 0.0001 0.0001 0.0001 0.682 0.685 0.686

23Number of Removable Breakout Pieces

0.650 0.702 0.748 0.0000 0.0000 0.0000 0.680 0.697 0.712 0.0001 0.0001 0.0001 0.682 0.685 0.687

24 Number of Effectors 0.963 0.968 0.973 0.0000 0.0000 0.0000 0.696 0.697 0.698 0.0001 0.0001 0.0001 0.685 0.685 0.685

25 Number of Sensors 0.048 0.293 0.842 0.0029 0.0001 0.0000 0.515 0.697 0.831 0.0029 0.0001 0.0000 0.649 0.685 0.706

31Flow Control System Procurement Cost

0.873 0.905 0.928 0.0000 0.0000 0.0000 0.376 0.378 0.379 0.1522 0.1522 0.1522 0.684 0.685 0.685

32Inlet Structural Procurement Cost

0.011 0.047 0.182 0.1587 0.0228 0.0000 0.276 0.378 0.504 0.2599 0.1522 0.1338 0.636 0.685 0.733

33Operations and Support Costs

0.214 0.650 0.890 0.1587 0.0228 0.0014 0.234 0.378 0.432 0.2599 0.1522 0.1338 0.612 0.685 0.707

43Volumetric Efficiency (L/D Surrogate)

0.368 0.569 0.748 0.4297 0.1196 0.0147 0.344 0.378 0.400 0.3782 0.1522 0.0585 0.670 0.685 0.694

Type: Performance

Type: Producibility

Type: Cost

Individual Desirability (di)Alternative: Config_101 Individual Risk (i) Type Risk (T)Type Desirability (DT) Composite Desirability (CSI)

28SynGenics

Corporation


Comparison of Sensitivities for the Three Alternative Configurations

RequirementConfig

101Config

102Config

20XConfig

101Config

102Config

20XConfig

101Config

102Config

20XConfig

101Config

102Config

20XConfig

101Config

102Config

20X

11 Vehicle Range 0.091 0.069 0.049 0.0824 0.0163 0.0015 0.024 0.017 0.012 0.0707 0.0145 0.0013 0.007 0.006 0.004

12 Distortion 0.052 0.281 0.559 0.0000 0.0013 0.1376 0.002 0.017 0.048 0.0000 0.0012 0.1135 0.001 0.005 0.016

13 Pressure Recovery 0.033 0.457 0.457 0.0000 0.0214 0.0214 0.001 0.040 0.042 0.0000 0.0191 0.0187 0.001 0.013 0.014

14 Fuel Consumption 0.391 0.391 0.391 0.0094 0.0094 0.0094 0.050 0.053 0.054 0.0090 0.0084 0.0082 0.016 0.016 0.018

15 Installed Engine Thrust 0.148 0.148 0.148 0.0013 0.0013 0.0013 0.017 0.019 0.019 0.0012 0.0012 0.0012 0.005 0.005 0.006

16 RCS 0.085 0.058 0.058 0.3062 0.3062 0.3062 0.009 0.010 0.010 0.2420 0.2437 0.2384 0.003 0.003 0.004

17 Throat Mach Number 0.107 0.107 0.107 0.0000 0.0000 0.0000 0.005 0.005 0.005 0.0000 0.0000 0.0000 0.002 0.002 0.002

42 Inlet Structural Weight 0.076 0.038 0.009 0.0013 0.0000 0.0000 0.021 0.008 0.001 0.0012 0.0000 0.0000 0.006 0.002 0.000

21 Producibility Fudge Factor 0.043 0.153 0.153 0.0000 0.1376 0.1376 0.005 0.021 0.019 0.0000 0.1232 0.1204 0.001 0.004 0.004

22Fidelity of Demo Article to Final System

0.108 0.108 0.108 0.0000 0.0000 0.0000 0.015 0.013 0.013 0.0000 0.0000 0.0000 0.003 0.003 0.003

23Number of Removable Breakout Pieces

0.069 0.185 0.185 0.0000 0.1376 0.1376 0.023 0.070 0.066 0.0000 0.1232 0.1204 0.004 0.015 0.016

24 Number of Effectors 0.007 0.090 0.427 0.0000 0.0013 0.1376 0.001 0.010 0.082 0.0000 0.0013 0.1204 0.000 0.002 0.021

25 Number of Sensors 0.601 0.601 0.620 0.0028 0.0472 0.0028 0.226 0.211 0.198 0.0028 0.0027 0.0027 0.042 0.046 0.049

31Flow Control System Procurement Cost

0.039 0.039 0.221 0.0000 0.0000 0.2367 0.002 0.004 0.029 0.0000 0.0000 0.1963 0.001 0.001 0.007

32Inlet Structural Procurement Cost

0.140 0.345 0.479 0.1378 0.0214 0.0022 0.162 0.191 0.141 0.1093 0.0207 0.0021 0.069 0.063 0.038

33Operations and Support Costs

0.498 0.498 0.498 0.1376 0.1376 0.1376 0.154 0.222 0.306 0.1093 0.1257 0.1186 0.076 0.084 0.090

43Volumetric Efficiency (L/D Surrogate)

0.269 0.149 0.048 0.3274 0.0164 0.0000 0.040 0.021 0.008 0.2447 0.0159 0.0000 0.017 0.007 0.002

Individual Risk (i) Type Risk (T)Type Desirability (DT) Composite Desirability (CSI)

Type: Performance

Type: Producibility

Type: Cost

Individual Desirability (di)

29SynGenics

Corporation


Explored Partial Derivatives

Calculated Partial Derivatives Of Each Measure of Merit With Respect to Each Performance,

Producibility, and Cost Criterion

Modeled Sensitivity of CSI to Weighting Factors for Types

Individual Desirability

(di)

Individual Risk (i)

Type Desirability

(DT)

Type Risk (T)

Composite Desirability

(CSI)

Overall Aggregated

Risk (A)101 0.269 0.3274 0.040 0.2447 0.017 0.1726102 0.149 0.0164 0.021 0.0159 0.007 0.013220X 0.048 0.0000 0.008 0.0000 0.002 0.0042

Sensitivity to Criterion 43. Volumetric Efficiency (L/D Surrogate)

Constant Term

Performance (WPerf)

Producibility (WProd)

Cost (WCost)

Other (WOther) R2

101 0.685 1.4598 Not Signif. -4.9664 1.3995 94.5%102 0.751 1.1668 -1.1981 -2.8468 1.0300 92.9%20X 0.809 0.8110 -2.7230 -0.5920 0.5889 95.2%

Sensitivity of CSI to Weight-ing Factors for Types

30SynGenics

Corporation


More Sensitivity Analysis Conclusions

The Results Are Most Sensitive To#25, Number of Sensors#32, Inlet Structural Procurement Cost#33, Operations and Support Costs

The Results Are Least Sensitive To #17, Throat Mach Number#42, Inlet Structural Weight#24, Number of Effectors

31SynGenics

Corporation


Status of STRICT

A Good Set of Criteria Has Been Established Further Consideration Given to Whether the Set is

Complete Configuration 20X Was Selected for Further Development O&S, Procurement Cost, # of Sensors Were Discriminators Identification of Criteria with Greatest Influence Supports

Resource Investment Decisions (e.g., Collection of Data) The Criteria with Highest Weightings and Lowest

Desirabilities Are the Ones to Which the Results Are Most Sensitive

STRICT Is Proceeding with IPPD Affordability Objectives in Focus

Documents

SynGenics Corporation 72 E Granville Road Worthington OH 43085 614.846.1804 [email protected] ©2003 SynGenics Corporation. All rights reserved. The Use