Presentation from November 8, 2000 Dinner Meeting

8/9/2019 Presentation from November 8, 2000 Dinner Meeting

1/46

1

Faster, Better, CheaperFaster, Better, Cheaper

An Idea without a PlanAn Idea without a Plan

8 November 2000

W. F. Tosney

Systems Engineering Division

Engineering and Technology Group

2000 The Aerospace Corporation


2/46

2

AgendaAgenda

FBC Origins & Perspectives

Risk Assessment & Lessons Learned Case Studies & Trends

Objectives & Scope

Dealing with Risk Conclusions & Recommendations

the plan unfolds

the recovery begins

the idea



3/46

3

ObjectivesObjectives

Present The Aerospace Corporations Assessment of theFaster, Better, Cheaper Approach from anEngineering Perspective

What are the Mission Success Records and What DrivesEngineering Risks?



4/46

4

Data Analyzed on 458 SVs- 1980 to 1999 timeframe- All weight / mission classes- 1st 3 years of mission reviewed- SVs Normalized for:

Design / Technology Heritage / Redundancy Degree of testing Production maturity

- Mined Lessons Learned

Study ScopeStudy Scope

PARTSand

MATLS

UNITS SYSTEMTEST

LAUNCHPREP

LAUNCH ORBITALOPERATIONS

Manufacture & Test Operations and Support

CONCEPTDESIGN

As - BuiltConfiguration

Release forProduction

Deliver toUser

End-of-life

As - MaintainedConfiguration

As - DesignedConfiguration

Development

FCA



5/46

5

BackgroundBackground

The 90s Saw the Introduction of What Many Refer to asRadical New Directions for Space System Development DoD Acquisition Reformelimination of nearly all specs/standards

deferred largely to commercial best practices

NASA Faster, Better, Cheaperstandardized on fixed schedules and budgets (applied to small SVs)

Commercial Best Practicesproduction processes adopted a least common denominator approach

(optimum meant whatever was least expensive little analytical basis)


Acquisition Improvement is Clearly a Laudable Goal.This Study Questions Knowledge of Inherent Risk Issues.

Acquisition Improvement is Clearly a Laudable Goal.This Study Questions Knowledge of Inherent Risk Issues.


6/46

6

FBC Origins and PerspectivesFBC Origins and Perspectives



7/46

7

Confluence of Factors Behind FBCConfluence of Factors Behind FBC

Costly Failures + Cost/ScheduleGrowth + Public Apathy = NASABudget Cut FY92

STS, Hubble,

Mars Observer,Galileo,NOAA-13

MPL

MCOWIRELEWIS

SDIO Rapid Protoype Examples

Delta 180, Delta 181, Losat-X, RME, LACE (1986+)Clementine (1994): cost $70M (excludes sunk costs) and developed in < 2 years

Leadership

ChangeGigantism of 80s Reduces # Science Missions

STS

0

2468

10121416

NASASVsFlown

1970s 1980s 1990s

Mariner

Magellan

FBC eraGRO

>2500 kg

1000-2500 kg


8/46

8

FBC Exactly Not NewFBC Exactly Not New

Fast, Small, Cheap SVs HaveExisted Since the Dawn

of the Space Age

Basic Development StrategiesRelatively Consistent

Risk is TypicallyAcknowledged Earlyin a Program

FBC Idea AllowedPrograms to DeviseNovel Risk Strategies Cost and schedule fixed

Spurred creativity

Manage risk using reserves

Commercial B

0

100

50

Space Program Risk CategorizationSpace Program Risk Categorization(test and mission assurance)(test and mission assurance)

Commercial A

DoD-Hdbk-343 Class A

DoD-Hdbk-343 Class B

DoD-Hdbk-343 Class C

DoD-Hdbk-343 Class D

(Mil-Std-1540 compliance)

(protoqual concept)

(acceptance only)

(system acceptance only)

0

20

40

60

80

100

120

>2500 kg

1000-2500 kg


9/46

9

Reality of SpaceReality of Space

SVs Normally Associated withConcept of Craftsmanship:

Prototype & low production volume

Technology driven

Complex interfaces Unforgiving environments

No perfect process (e.g, Iridium)

Troubleshooting theproductis vital to mission success

The Techno-economic Reality of Space is Highlighted

in The Machine That Changed the World(summary of Lean manufacturing principles)

The Techno-economic Reality of Space is Highlightedin The Machine That Changed the World

(summary of Lean manufacturing principles)

Orbital Insertion and Space FlightPowered LaunchFactory & Pre- launch

High vacuum;material outgassing

Solar Radiation

-459 heat sink Earth radiation Radiation belt

& solar flares Temp. extremes;

cyclic variation

Magnetic fields RFI Meteoroids

Nuclear sources Atomic oxygen Space plasma Plume contamination Spacecraft charging

Shock & vibration Acceleration Acoustic noise

Aerodynamic heating Thermal transition Pressure decrease Corona/arcing

Thermal cycling& vacuum

Shock & vibration

Acoustic noise Humidity Handling & transport RF radiation Storage conditions Explosive atmosphere



10/46

10

Applying FBC to SpaceApplying FBC to Space

Morphed into Acquisition Reform, Lean Aerospace,and Commercial Best Practices

Became a Mantra that Encourages: Cost and schedule cutting

Minimal or no oversight Wanton risk-taking - Risk is a resource

Removal of established standards Less testing

What is the Track Record of the FBC Idea:Versus conventional approach?

Versus past experiences?

What is the Track Record of the FBC Idea:Versus conventional approach?

Versus past experiences?



11/46

11

Damage AssessmentDamage Assessment



12/46

12

0

10

20

3040

50

60

70

80

90

100

80 8182838485 86 87 88 89 90 91 92 93 9495969798 99

Year

Cumulativ

eSuccessRate,

%

Failure and Degradation TrendsFailure and Degradation Trends

Mission Success Has Proven to be Manageable.

Trends Show a Dramatic Increase in Failure Rates.

Mission Success Has Proven to be Manageable.

Trends Show a Dramatic Increase in Failure Rates.


Calendar Launch Year

SignificantDegrad

ation

(orFailure)in1st3

Years

84 85 86 87 88 89 90 91 92 93 94 95 96 97 98

USSVsLaunch

ed

USSVsLaunch

ed

0

10

20

30

40

50

60

70

80

90

100

0%

10%

20%

30%

40%

50%

I ncreasing Degradation TrendI ncreasing Degradation Trend

1st Year Catastrophic Failures

- 454 SVs -

1st Year Catastrophic Failures

and 3 Year Degradation- 394 SVs -

Convent ional Approach

Commercial

DoD

CivilianFBC-Li ke SVs:

134

Conventional SVs:324

FBC-Like


13/46

13

Cost PerspectiveCost Perspective

Need for Clearer Understanding of Risk Issuesand a More Aggressive Approach to Mission Assurance

Need for Clearer Understanding of Risk Issuesand a More Aggressive Approach to Mission Assurance

Over $11B in Lost Assets Since 1990 Buys a lot of development and testing!!


0

500

1000

1500

2000

2500

3000

90 91 92 93 94 95 96 97 98 99

Year

USSpaceAsset

Loss($M) Military + Civilian + Commercial SVs (+ launch failures)Military + Civilian + Commercial SVs (+ launch failures)

Mars ObserverMars Observer

Landsat 6, NOAA 13, UFOLandsat 6, NOAA 13, UFO--11

Cost PerspectiveCost Perspective

FBC, Acq. Reform,and Best Practices

Begin to Deploy


14/46

14

14 SVs 70s14 SVs 70s 22 SVs 90s22 SVs 90s

Average Mission Cost (99 $) 64M 101MPer Pound Cost (99 $) 90,000 118,000Development Time (months) 34 39

FBC as Executed Today is High Risk > 30% failure rate May be viable for limited applications such as technology

demonstratorsAcknowledge overall program risk at the outset

Emphasize risk management early in development

FBC Reality CheckFBC Reality Check

Cost / Schedule Improvements (are illusionary) Not ClearlyDemonstrated



15/46

15

Commercial SV Failures Increasing in Frequency Deviation from conventional practices

Major issue with insurance brokers

Unconventional Space Ventures are Meeting with Limited

Success Iridium record: > 20 failed (need 66, flew 90)

Commercial launch service providers:

vehicle failures: (AMROC, LMLV, Conestoga, Delta III, Sea Launch, Pegasus)

business failures: (Pioneer Rocket Plane, Astroliner, StarBooster)

Technology Demonstration Plunge is an Uncalibrated

Risk Venture

Commercial Best PracticesCommercial Best PracticesReality CheckReality Check

The Biggest Lesson from X-33 is that Airplane Peopleare Struggling with the Realities of Space Systems

The Biggest Lesson from X-33 is that Airplane Peopleare Struggling with the Realities of Space Systems



16/46

16

Wholesale Attempt to ApplyWholesale Attempt to Apply

Commercial Best PracticesCommercial Best Practices

Iridium Experience is Sobering: Rigorously applied 6-sigma at lower level Still expected medium to high reliability at satellite level

Over 20% failure rate to date

Motorola recently closed down its Satcom division

As Remarkable as Iridium Development Was -

Is it an Applicable Guidepost to the Future?Is There a Chance to Repeat?

As Remarkable as Iridium Development Was -

Is it an Applicable Guidepost to the Future?Is There a Chance to Repeat?



17/46

17

Reductions in Government Oversight Were not Mitigated

by Contractors as Intended Major reductions in DCAA, govt, contractor, and subcontractor

oversight

Internal changes not implemented (Milstar Flt 2-1)

Re-learning of Lessons Learned in Full-Swing Eliminated key independent analysis (Delta III 259)

Contractors Continue to Advocate Cost Cutting Measures

on Government Programs

Reduced development test(Lewis, WIRE, MCO, MPL)

Elimination of key production tests (DSP-19)

Less subcontractor oversight (MCO, Delta III 269)

DoD Acquisition ReformDoD Acquisition ReformReality CheckReality Check



18/46

18

Legacy Systems Still Have Risk

Evidence of Increased Risk,

Not Decreased Risk


19/46

19

Lessons LearnedLessons Learned



20/46

20

Lewis Case StudyLewis Case Study

Payload: 81-kg, 160-W

Orbit: 523-km, sunsynch Design life: 3 years (5 year goal) Mass: 386 kg Launch Vehicle: Athena 1 (1998) ADCS: 0.02/0.002 deg, 3-axis zero

momentum

Propulsion: 8 2.2 N hydrazinethrusters, dual-mode, composite tank

Structures/Thermal: composite,integrated Power: 620 W EOL, Si cells, NiH2

battery

TT&C/C&DH: S-band, 2000 kbps, 4-Gb

Factor Unit Lewis

Spacecraft Cost (FY97$M)

Development Time (mos)

Satellite Dry Mass (kg) 310

Design Life (mos) 48

Heritage (%) 50%

Level of Redundancy (%) 10%

Orbit Altitude (km) 523

BOL Power (W) 682

EOL Power (W) 620Solar Array Area (m^2) 4.5

Solar Array Type Si

Battery Type NiH2

Battery Capacity (A-hr) 33.8

Structures Material Composite

ADCS Type 3-axis

Number of P/L Instruments 4

Pointing Accuracy (deg) 0.02

Pointing Knowledge (deg) 0.002

Number of Thrusters (#) 8Uplink Band S-band

Max Downlink Data Rate (Kbps) 2,000

Recorder Memory (Mbytes) 4,000

Thermal Type active

88%

97%

Complexity Factor

Normalized Complexity Factor

65

36



21/46

21

Lewis ObservationsLewis Observations

Ambitious Program Mandated demonstration of more than 25 advanced technologies 24-month development schedule

Under Tremendous Cost and Schedule Pressure DuringDevelopment

Program experienced 12-month schedule slip

$7M ($65M total) added with unknown contractor investment Model* indicates that insufficient resources were allocated

Contractual Changes Increase in the on-orbit life requirements (3 to 5 years) Launch vehicle change (Pegasus XL to Athena 1)

Lewis Failure On-orbit Flawed design of attitude control system Inadequate monitoring of spacecraft during early operations coupled

with unorthodox orbit-insertion procedure

*: David Bearden Ph.D. Dissertation, May, 1999



22/46

22

NEAR Case StudyNEAR Case Study

Payload: 56-kg, 94-W Orbit: 2.2 AU asteroid Design life: 4 years Mass: 788 kg Launch Vehicle: Delta 7925 (1996) ADCS: 0.1/0.01 deg, 3-axis

Propulsion: 1 450 N biprop thruster, 112.2 N hydrazine thrusters, dual-mode Structures/Thermal: aluminum Power: 330 W EOL, GaAs cells, NiCd

battery

TT&C/C&DH: X-band, 27 kbps, 177 Mb

Factor Unit NEAR





Heritage (%) 50%


Orbit Altitude (km)

BOL Power (W) 349


Solar Array Type GaAs

Battery Type NiCd


Structures Material Aluminum

ADCS Type 3-axis




Number of Thrusters (#) 12Uplink Band X-band

Max Downlink Data Rate (Kbps) 27

Recorder Memory (Mbytes) 221

Thermal Type passive

73%

90%

Complexity Factor


210

36



23/46

23

NEARNEAR ObservationsObservations

Controlled Cost by Utilizing Mature, Well-UnderstoodComponents Adequate good enough prevailed over optimized design Little beyond-state-of-the-art technology

Large Built-in Contingencies and Design Margins Accepted weight penalties in some areas

Tight Schedule Due to Launch Window (36 months) Schedule a driver of technical decisions IMU presented a schedule risk Late deliveries resulted in expensive retesting

NEAR Appears to Have Allocated Adequate Resources

Returned ~$4M to NASA 29-month development



24/46

24

Lewis Case StudyLewis Case Study

Payload: 81-kg, 160-W

Orbit: 523-km, sunsynch Design life: 3 years (5 year goal) Mass: 386 kg Launch Vehicle: Athena 1 (1998) ADCS: 0.02/0.002 deg, 3-axis zero

momentum

Propulsion: 8 2.2 N hydrazinethrusters, dual-mode, composite tank

Structures/Thermal: composite,integrated Power: 620 W EOL, Si cells, NiH2

battery

TT&C/C&DH: S-band, 2000 kbps, 4-Gb

Factor Unit Lewis





Heritage (%) 50%


Orbit Altitude (km) 523

BOL Power (W) 682


Solar Array Type Si

Battery Type NiH2


Structures Material Composite

ADCS Type 3-axis




Number of Thrusters (#) 8Uplink Band S-band

Max Downlink Data Rate (Kbps) 2,000

Recorder Memory (Mbytes) 4,000

Thermal Type active

88%

97%

Complexity Factor


65

36



25/46

25

Lewis ObservationsLewis Observations Ambitious Program

Mandated demonstration of more than 25 advanced technologies 24-month development schedule Under Tremendous Cost and Schedule Pressure During

Development Program experienced 12-month schedule slip

$7M ($65M total) added with unknown contractor investment Model* indicates that insufficient resources were allocated Contractual Changes

Increase in the on-orbit life requirements (3 to 5 years) Launch vehicle change (Pegasus XL to Athena 1)

Lewis Failure On-orbit Flawed design of attitude control system Inadequate monitoring of spacecraft during early operations coupled

with unorthodox orbit-insertion procedure

*: David Bearden Ph.D. Dissertation, May, 1999



26/46

26

Mariner 10 Case StudyMariner 10 Case Study (27 years ago)(27 years ago)

Payload: 77-kg Orbit: interplanetary, Venus-Mercury Design life: 6 months Mass: 562-kg Launch Vehicle: Atlas Centaur (1973) ADCS: 0.29/0.11 deg, 3-axis

Propulsion: 1 monoprop thruster, 6 nitrogenthrusters Structures/Thermal: aluminum/passive Power: 491 W at Mercury (6 days), Si cells,

NiCd battery

TT&C/C&DH: X-band, 117.6 kbps, 180 Mbits

Factor Unit Mariner 10Spacecraft Cost (FY97$M) 270



Instrument Mass (kg) 77

Design Life (mos) 6

Heritage (%) 50%


Orbit Altitude (fly-by) (km) 271 to 48,069

Power at planet Mercury (W) 491

Solar Array Area (m^2) 5.1

Solar Array Type Si

Battery Type NiCd

Battery Capacity (A-hr) 20

Structures Material Aluminum

ADCS Type 3 axis




Number of Thrusters (#) 7

Uplink Band X band

Max Downlink Data Rate (kbps) 117.6

Recorder Memory (Mbits) 180

Thermal Type passive

49%

60%

Complexity Factor


27



27/46

27

Mariner 10 ObservationsMariner 10 Observations

Controlled NASA Cost Mandate by Utilization ofMariner Design Derivation Hardware, software, documentation, and assurance Mandated use of existing flight tested experiments Saved 50% in design and 15% in hardware costs

Science / Engineering Teaming in Development Intensive advanced planning cited as largest cost saver

Tight Schedule Due to Launch Window System contract delayed 3 months due to up-front planning No major schedule slippage

Mariner 10 Appears to Have Allocated AdequateResources Allocated $98M in 69 ($1997 = 270M) 27 month development



28/46

28

Recent, but Repeating LessonsRecent, but Repeating Lessons Ring Laser Gyro Failures

Non-space qualified technology. Life tests and better qualification may haveprevented numerous on-orbit anomalies and failures

Electrical Power and Battery/Charge Problems Enhanced qualification tests may have surfaced problems prior to flight, known

life limiting hardware

SV Processor Failure Adherence to known lessons learned (tin whiskers) may have prevented failureson multiple SVs

Transponder Failures More perceptive thermal modeling and testing may have identified failure mode

(high temperature zones)

Deployment Failures Mechanism verification methods suspected to have been reduced to save time.

Critical mechanisms require full verification

Likely Preventable with Refined Develop. / I&T ProcessesLikely Preventable with Refined Develop. / I&T Processes



29/46

29

Making Sense of LessonsMaking Sense of Lessons

Critical Lessons Learned Need to Become Policy SMC Commanders Policies are a good example

Recurring Engineering Related Lessons Can Lead toMore Accurate Risk Modeling Complexity and technology readiness correlation

Development risk mitigation Failure risk models based on cost, schedule, and test

EXAMPLE: Recent Acoustic Test Data (>1986) Usedto Quantify a Major Risk Issue:

Test or not to test? (ANSWER: Test!!)

Recurring theme with most programs today Cross-program experience data (lessons) are essential to

understanding risk issues



30/46

30

Industry TrendsIndustry Trends



31/46

IndustryIndustry--wide Trendswide Trends

Declining Quality and Safety are not Unique to the SpaceIndustry

Air Force Implemented the Policy on Operational, Safety,Suitability, and Effectiveness (OSS&E) as Result of

Numerous Mishaps

B1B mishap traced to a flawed re-design Loss of technical oversight and

communication between contractors

T-3A Firefly mishap traced to marginal

design and qualification (6 deaths) Procured as COTS product Loss of technical oversight and

intentionally abbreviated testing


32/46

32

AerospaceAerospace --An Industry in CrisisAn Industry in Crisis

Lockheed Martin Stock

01/98 01/99

Industry in Crisis Static business projections Investor confidence on the wane Extreme difficulty attracting top technical talent

Industry in Crisis

Static business projections Investor confidence on the wane Extreme difficulty attracting top technical talent

0

20K

40K

60K

80K

100K

1983 1985 1987 1989 1991 1993 1995 1997 1999Missile/S

paceIndustry(SIC376

)

Emplo

ymentinCalifornia

Source: California Employment Development Dept.



33/46

33

Launch Vehicle BAR: Re-emphasize oversightvs. insightBoeing Review: Quality must be the highest priority

L-M Review Team: Rigorously test like you fly

NASA Mars Panel: Mission Success First - a cultural

shift

NASA FBC Review: Too many Mission Failures failed toadhere to established standards

RAND Study: FBC - an uncontrolled experiment

Space Industry ReSpace Industry Re--calibratescalibrates

FBC As We Knew it No Longer ExistsFBC As We Knew it No Longer Exists



34/46

34

Dealing with RiskDealing with Risk



35/46

Risk Management ProcessRisk Management Process

Strategy/Plan

Acquisition plan

- compliance

- deviations/waivers

System Requirements

Mission&

PolicyDriven

Concept Exploration

PHASE A PHASE B PHASE C

System Development(pre-acquisition)

Production, Operations,and Development

IndependentAssessment Historical Data

Models

Trades

Review &

Feedback

Program / Data

Reviews, Verification Indep Assessment

Decision Support

Acceptance

IPTs

Award Fee

Mission

Technology

Requirement Tradespace

Mission

Technology

Heritage

Test and Evaluation

Integration

Manufacturing

Production

Suitability and

Effectiveness

Change Impacts

Design Trades Cost Assessment

Schedule Assessment

Risk Assessment

Phase Decision Phase Decision Phase Decision

Project Oversight Independent Technical

Analysis

Mission Assurance

Risk Management

Project Oversight Independent Technical

Analysis

Mission Assurance

Risk Management

Preliminary Design

Review

Critical Design Review

HAR / FAR

Mission Assurance Team

Independent Readiness

Review Mission Readiness Review

Architecture Definition

Business Cases

COTS vs Development

System RequirementsReview


36/46

36

Technology Risk AssessmentTechnology Risk Assessment

Level 1 Basic principles observed and reported

Level 2 Conceptual design formulated

Level 3 Conceptual design tested analytically orexperimentally

Level 4 Critical function/characteristic demonstration Level 5 Component/brassboard tested in relevant

environment

Level 6 Prototype/engineering model tested in relevantenvironment

Level 7 Engineering model tested in space

Level 8 Flight-qualified system Level 9 Flight-proven system

Highest risk

Least risk

Technologydevelopment

Advanceddevelopment

Flightsystems


NASA Technology Readiness Levels


37/46

37

Schedule as Function of ComplexitySchedule as Function of Complexity Cost as Function of ComplexityCost as Function of Complexity

R = 0.801

1

10

100

1000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Complexity Index

SpaceVeh

icleCost(FY97$M)

R2 = 0.613

0

10

20

3040

50

60

70

80

90

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Complexity Index

ActualDe

velopment(mos)

100

Evidence of

No Fly Zone

Evidence of

No Fly Zone

Cost and Schedule Risk- Small SV Models -

CoCost andst andSchedule RiskSchedule Risk-- Small SV ModelsSmall SV Models --

Pushing Cost and Schedule Risk Too FarPoses Unacceptably High Risk

Pushing Cost and Schedule Risk Too FarPoses Unacceptably High Risk

SOURCE: D. Bearden, The Aerospace Corporation.

Success

Failure

2



38/46

38

1008060402000

2

4

6

8

10

12

14

ORBITAL SUCCESS COMPARISONS

TEST THOROUGHNESS INDEX (Compliance to Mil-Std-1540)

EarlyFlightFailures/105Parts

Class C & DClass A & B

Test Program RiskTest Program Risk

More Thorough Testing Reduces Orbital Failure RiskMore Thorough Testing Reduces Orbital Failure Risk



39/46

39

Factors Influencing Test

Perceptivity:- Analytical prediction accuracy- Testability- Interfaces- Selected measurements- Measurement uncertainty- Test equipment limitations- Multiple environments

- Integration affects

Scope and Complexity of TestingScope and Complexity of Testing


ElectricalMechanicalSoftware

Pressure/leak


AcousticVibrationShockOrbit tempVacuumPressure/leakEM I/EMC

PARTSand

MATLSUNITS

SYSTEMTEST

LAUNCHPREP

LAUNCH ORBITALOPERATIONS

ElectricalThermalVibration

X-rayAccelLeak


VibrationShockBurn-inTemp cycVacuumPressure/leakEM I/EMC



Complex Integration

MultipleSubcontractors

MeasurementUncertainty

Harsh Environments

Lockheed Martin Corp. 2000 All Rights Reserved.


40/46

40

Modeling and SimulationModeling and Simulation

Architecture, planning, design,development, test and operations areincreasingly reliant

Modeling and simulation have not obviated theneed for a prudent test program

Complex interfaces Synergistic effects

Differences in test fidelity Hardware and material

variability

Workmanship defects

on robust modeling andsimulation tools /techniques



41/46

41

Production Sequence

ObservationsHigh failure rate of first

and second SVs

Most FBC SVs fall intothis group

Failure rate dampensaround third vehicle

Percent Failures vs. Vehicle SequencePercent Failures vs. Vehicle Sequence

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5 > 6

Average Sequence

Perce

ntFailures

0

50

100

150

200

250

NumberofVehicles

% Fail Population

R2 = 0.94

Strong relationship of production sequence to failure rateStrong relationship of production sequence to failure rate


42/46

42

Environmental Test Thoroughness

0

0.1

0.2

0.3

0.4

20 40 60 80 100

0.5

0.6

0.7

0.9

1

0.8

1stYearFailureProbability

Primary Driver: Test Thoroughness

Major Drivers:

Manufacturing maturity Heritage

A multi-variate regression analysis reveals:

System Test TimeSystem Test Time

Driven Largely by Development TimeDriven Largely by Development Time

0

10

20

30

40

50

60

70

80

90

100

110

120

PDR2ST

DryWt

CNTR2ST

Mtype

Mcat

CDR2ST

Pcontr

ETTI

RE1SC

STOWDIM

AVGSCPLX

CLASS

CNTR2PDR

DESLIFE

DRIVDES

HINGES

IAT

IATNRE

IATRE

NRE1SC

NRECMPLX

Orbit

PARTS

PARTS/FT3

PDR2CDR

PWRBOL

RECMPLX

S/C1COST

RelativeImportance

0

20

40

60

#ofSamplesinDataset

Samples Rel Importance

Mission SuccessMission Success

Driven Largely by Test Program and MaturityDriven Largely by Test Program and Maturity

ExperienceExperience--Based Risk ModelsBased Risk Models

Risk Factors, Once identified, Can be Addressedas a Function of Technical Complexity

Risk Factors, Once identified, Can be Addressedas a Function of Technical Complexity


43/46

43

Revisiting StandardsRevisiting Standards Risk Standards and Handbooks Mandated Out of Existence

in the 90s DoD-Hdbk-343, Design, Construction, and Testing Requirements for One of a

Kind Space Equipment, 1986

NMI 8010.1, Classification of NASA STS Payloads, 1986

Cost-benefit

Cost

Total

costs

Minimum

total cost

Cost of testfailures notfound (benefit) Cost of

reliability andtesting efforts

UltimateGoal

High RiskPrograms

Low RiskPrograms

The Goal Has Always Existed.The Goal Has Always Existed. FBC Risk GuidelinesExists but are Orientedtowards Testing and areNon-conservative

Ad-hoc CommercialStandards

An Opportunity Exists toUpdate Selective Specs



44/46

44

Conclusions and RecommendationsConclusions and Recommendations



45/46

45

Invest in EngineeringInvest in Engineering--Based SolutionsBased Solutions

Restructuring Space System Development PhilosophiesRequires a Robust Engineering Model Planning, development, production, and process modeling solutions

need to evolve

Engineering insight is essential (anyone can reduce cost & schedule)

Optimize for mission success

An overnight or one-fix solution is not likely

Evaluate Acquisition/Development Models FromSuccessful and Failed Programs

Develop a modeling strategy based on key influencing variables



46/46

46

ConclusionsConclusions FBC Pushed the Risk Envelope Too Far Without a

Clear Roadmap Cost and schedule models did not address mission risk Risk management a second order concern behind cost and schedule

Unrealistic Schedules and Cost Resulted inAggressive and Risky Compromises

Poor documentation Minimal oversight and IV&V, especially with suppliers Tests often first in line for cost saving measures

Rigorous Post-Mortems Provide Valuable Insight Past mistakes repeated (GAO, LV BAR, NASA.)

Successful programs have a story to tell as do the unsuccessful Small Programs Today are Not Radically Different

in Terms of Development Efficiency But they should be!!


Documents

Presentation from November 8, 2000 Dinner Meeting