A Value-Based Orthogonal Framework for Improving Life ... · Design for Maintainability, Evolvability Automate Operations Elements Anticipate, Prepare for Change Value- and Architecture-Based

University of Southern California

Center for Systems and Software Engineering

Barry Boehm, Jo Ann Lane, Sue Koolmanojwong

http://csse.usc.edu

NDIA Systems Engineering Conference

October 25, 2012

A Value-Based Orthogonal Framework

for Improving Life-Cycle Affordability



Outline

• Affordability definitions, concepts, issues, strategies

– Addressing both costs and benefits

– Using life cycle present value

– Coping with uncertainty: incrementally; pro-actively

– Coping with multi-stakeholder value diversity

– Addressing tradeoffs with other -ilities

• An orthogonal framework for improving affordability costs

– Cost modeling and other insights

• Conclusions

October 16, 2012 2 Copyright © USC-CSSE



Affordability Definitions

• INCOSE: The balance of system performance, cost, and schedule

constraints over the system life cycle, while satisfying mission

needs in concert with strategic and organizational needs.

• MORS: Cost-effective capability (USD/ATL).

• NDIA: The practice of assuring program success through the

balancing of system performance (KPPs), cost, and schedule

constraints, while satisfying mission needs in concert with the

long-range investment and force structure plans of the DoD.

• Webster: Keeping within your financial means.

October 16, 2012 Copyright © USC-CSSE 3



Affordability Concepts Coping with uncertainty: incrementally; pro-actively

Coping with multi-stakeholder value diversity


Life Cycle Cost Improvement

Life

Cycle

Benefits

Improve-

ment

Achievable

Pareto

Boundary

Current

Situation

Primary

Option

Space

Increasing

Cost-Benefit

Improvement

Isoquants



October 16, 2012 Copyright © USC-

CSSE

5

Multi-Stakeholder Value Diversity

Bank of America Master Net



Outline

• Affordability definitions, concepts, issues, strategies

– Addressing both costs and benefits

– Using life cycle present value

– Coping with uncertainty: incrementally; pro-actively

– Coping with multi-stakeholder value diversity

– Addressing tradeoffs with other -ilities

• An orthogonal framework for improving affordability costs

– Cost modeling and other insights

• Conclusions




October 16, 2012 7

Legacy System Repurposing

Eliminate Tasks

Eliminate Scrap, Rework

Staffing, Incentivizing, Teambuilding

Kaizen (continuous improvement)

Work and Oversight Streamlining

Collaboration Technology

Early Risk and Defect Elimination

Modularity Around Sources of Change

Incremental, Evolutionary Development

Risk-Based Prototyping

Satisficing vs. Optimizing Performance

Value-Based Capability Prioritization

Composable Components,Services, COTS

Affordability

Improvements

and Tradeoffs

Get the Best from People

Make Tasks More Efficient

Simplify Products (KISS)

Reuse Components

Facilities, Support Services

Tools and Automation

Lean and Agile Methods

Evidence-Based Decision Gates

Domain Engineering and Architecture

Task Automation

Model-Based Product Generation

Value-Based, Agile Process Maturity

Affordability and Tradespace Framework

Reduce Operations, Support Costs

Streamline Supply Chain Design for Maintainability, Evolvability Automate Operations Elements

Anticipate, Prepare for Change Value- and Architecture-Based

Tradeoffs and Balancing

Copyright © USC-CSSE



8

Costing Insights: COCOMO II Productivity Ranges

Productivity Range

1 1.2 1.4 1.6 1.8 2 2.2 2.4

Product Complexity (CPLX)

Analyst Capability (ACAP)

Programmer Capability (PCAP)

Time Constraint (TIME)

Personnel Continuity (PCON)

Required Software Reliability (RELY)

Documentation Match to Life Cycle Needs (DOCU)

Multi-Site Development (SITE)

Applications Experience (AEXP)

Platform Volatility (PVOL)

Use of Software Tools (TOOL)

Storage Constraint (STOR)

Process Maturity (PMAT)

Language and Tools Experience (LTEX)

Required Development Schedule (SCED)

Data Base Size (DATA)

Platform Experience (PEXP)

Architecture and Risk Resolution (RESL)

Precedentedness (PREC)

Develop for Reuse (RUSE)

Team Cohesion (TEAM)

Development Flexibility (FLEX)

Scale Factor Ranges: 10, 100, 1000 KSLOC

October 16, 2012 Copyright © USC-CSSE

Staffing

Teambuilding

Continuous

Improvement



COSYSMO Sys Engr Cost Drivers

9 October 16, 2012

Teambuilding

Staffing

Continuous

Improvement




October 16, 2012 10


Eliminate Tasks













Affordability

Improvements

and Tradeoffs




Reuse Components






Task Automation



Tradespace and Affordability Framework


Streamline Supply Chain

Design for Maintainability, Evolvability

Automate Operations Elements






11

COCOMO II. 2000 Productivity Ranges

Productivity Range

1 1.2 1.4 1.6 1.8 2 2.2 2.4

























Tools

Collaboration

Technology

Work Streamlining



October 16, 2012 12


Eliminate Tasks













Affordability

Improvements

and Tradeoffs




Reuse Components






Task Automation














Expansion

Factor

The ratio

of machine

lines of

code to a

source line

of code

1

10

100

1000

1960 1965 1970 1975 1980 1985 1990 2000 1995

Order of Magnitude Increase Every Twenty Years

Machine Instructions

Macro Assembler

High Level Language

Database Manager

On-line

Regression Testing

Prototyping

4GL

Subsecond Time

Sharing

Small Scale Reuse

Object Oriented

Programming

Large Scale Reuse

142 113

81 75

47 37.5

30

15

3

475 638

Trends in Software Expansion (Bernstein, 1997)

MBSE:2010



October 16, 2012 14


Eliminate Tasks













Affordability

Improvements

and Tradeoffs




Reuse Components






Task Automation













October 16, 2012 15


Eliminate Tasks













Affordability

Improvements

and Tradeoffs




Reuse Components






Task Automation













October 16, 2012 16

$100M

$50M

Required

Architecture:

Custom; many

cache processors

Original

Architecture:

Commercial

DBMS

1 2 3 4 5

Response Time (sec)

Original Spec After Prototyping

Original Budget

Sequential Requirements-First Risks It’s not a requirement if you can’t afford it




October 16, 2012 17


Eliminate Tasks













Affordability

Improvements

and Tradeoffs




Reuse Components






Task Automation












Center for Systems and Software Engineering The HMMWV

300,000 Produced, 22 Fielded Versions

Initial draft requirements in 1979, Initial delivery in 1984

Bolt on armor

required upgraded

suspension, engine,

and steering

Mattracks or

wheels

Imbalance in cupola

required motorized

drive

Additional armor and

cupola raise the CG

and increase

rollovers

Upper deck space is

always at a premium

Suspension and

steering for CG shift

Upgrades:

•Increased cab space

•Increased payload

capacity

• Strengthened frame

Base cab & flatbed with

mission modules

9



October 16, 2012 19


Eliminate Tasks













Affordability

Improvements

and Tradeoffs




Reuse Components






Task Automation













Post-Acquisition Costs Dominate (%O&M)

• Hardware [Redman 2008]

– 12% -- Missiles (average)

– 60% -- Ships (average)

– 78% -- Aircraft (F-16)

– 84% -- Ground vehicles (Bradley)

• Software [Koskinen 2010]

– 75-90% -- Business, Command-Control

– 50-80% -- Complex platforms as above

– 10-30% -- Simple embedded software

• Apply lack-of-flexibility factor to O&M

component October 16, 2012 20 Copyright © USC-CSSE



October 16, 2012 21


Eliminate Tasks













Affordability

Improvements

and Tradeoffs




Reuse Components






Task Automation













Copyright © USC-CSSE 22

Tradeoffs Among Cost, Schedule, and

Reliability, and Functionality: COCOMO II

0

1

2

3

4

5

6

7

8

9

0 10 20 30 40 50

Development Time (Months)

Co

st

($M

)

(VL, 1)

(L, 10)

(N, 300)

(H, 10K)

(VH, 300K)

-- Cost/Schedule/RELY:

“pick any two” points

(RELY, MTBF (hours))

•For 100-KSLOC set of features

•Can “pick all three” with 77-KSLOC set of

features

October 16, 2012



Copyright © USC-CSSE 23 October 16, 2012

A Value-Priority Tradeoff Equalizer



Conclusions

• Affordability increasingly competition-critical

– Need to balance cost, schedule, performance, functionality

• Orthogonal framework helps tailor improvements

– Getting the best from people

– Making tasks more efficient

– Eliminating tasks

– Eliminating scrap and rework

– Simplifying products

– Reusing assets

– Reducing operations and support costs

– Value- and architecture-based tradeoffs and balancing

• No one-size-fits-all solution




Backup Charts





Agile and Plan-Driven Home Grounds:

Five Critical Decision Factors

• Size, Criticality, Dynamism, Personnel, Culture

Personnel

Dynamism

(% Requirements - change/month)

Culture

(% thriving on chaos vs. order)

Size

(# of personnel)

Criticality

(Loss due to impact of defects)

30 10

3.0 1.0

0.3

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15

Essential

Funds Discretionary

Funds Comfort

Single

Life

Many

Lives

(% Level 1B) (% Level 2&3)

0

10

20

30

40

Personnel

Dynamism

(% Requirements - change/month)

Culture

(% thriving on chaos vs. order)

Size

(# of personnel)

Criticality

(Loss due to impact of defects)

90

70

50

30

10

3

10

30

100

300

35

30

25

20

15

Essential

Funds Discretionary

Funds Comfort

Single

Life

Many

Lives

(% Level 1B) (% Level 2&3)

0

10

20

30

40



Architected Agile Approach

• Uses Scrum of Scrums approach

– Up to 10 Scrum teams of 10 people each

– Has worked for distributed international teams

– Going to three levels generally infeasible

• General approach shown below

– Often tailored to special circumstances

October 16, 2012 27




28

COCOMO II. 2000 Productivity Ranges

Productivity Range

1 1.2 1.4 1.6 1.8 2 2.2 2.4



























October 16, 2012 29

Value-Based Testing: Empirical Data and ROI — LiGuo Huang, ISESE 2005

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 10 20 30 40 50 60 70 80 90 100

% Tests Run

Ret

urn

On

Inve

stm

ent (

RO

I)

Value-Neutral ATG Testing Value-Based Pareto Testing

% of

Value

for

Correct

Customer

Billing

Customer Type

100

80

60

40

20

5 10 15

Automated test

generation (ATG) tool

- all tests have equal value

Bullock data

– Pareto distribution% of

Value

for

Correct

Customer

Billing

Customer Type

100

80

60

40

20

5 10 15

Automated test



% of

Value

for

Correct

Customer

Billing

Customer Type

100

80

60

40

20

5 10 15

Automated test



Bullock data

– Pareto distribution

(a)

(b)




Value-Neutral Defect Fixing Is Even Worse

% of

Value

for

Correct

Customer

Billing

Customer Type

100

80

60

40

20

5 10 15

Automated test

generation tool


Value-neutral defect fixing:

Quickly reduce # of defects

Pareto 80-20 Business Value





Reuse at HP’s Queensferry

Telecommunication Division

0

10

20

30

40

50

60

70

86 87 88 89 90 91 92

Year

Time

to

Market

(months)

Non-reuse Project

Reuse project



Product Line Engineering and Management




Overfocus on Acquisition Cost C4ISR Contracts: Nominal-case requirements; 90 days to PDR

33 Copyright © USC-CSSE October 16, 2012



C4ISR Project C: Architecting for Change USAF/ESC-TRW CCPDS-R Project*


When investments made in architecture, average time for change order

becomes relatively stable over time…

* Walker Royce, Software Project Management: A Unified Framework. Addison-Wesley, 1998.



Relative* Total Ownership Cost (TOC)


* Cumulative architecting and rework effort relative to initial development effort

~5% architecture

investment

~5% architecture

investment

~25% architecture

investment

0.00%

50.00%

100.00%

150.00%

200.00%

250.00%

Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5

Project A Project B Project C



Ilities in Tradespace Exploration: MIT

For this plot, Ĉ=C∞

More changeable(ie including flexible,

adaptable, scalable

and modifiable)

Colored by

outdegree

Enabling Construct: Tradespace Networks Changeability

Survivability

Value Robustness

Enabling Construct: Epochs and Eras

Set of Metrics

seari.mit.edu © 2012 Massachusetts Institute of

Technology 36



Architecture-Based Attribute Trades:

Flexibility Example (RT-18a) Flexibility Arch.

Strategy

Synergies Conflicts

High module cohesion;

Low module coupling

Interoperability

Reliability

High Performance via

Tight coupling

Service-oriented architecture Composability, Usability,

Testability

High Performance via

Tight coupling

Autonomous adaptive systems Affordability via task automation;

Response time

Excess autonomy reduces

human Controllability

Modularization around sources

of change

Interoperability, Usability,

Reliability, Availability

Extra time on critical path of

Rapid Fielding

Multi-layered architecture Reliability, Availability Lower Performance due to layer

traversal overhead

Many built-in options, entry

points

Functionality, Accessibility Reduced Usability via options

proliferation; harder to Secure

User programmability Usability, Mission Effectiveness Full programmability causes

Reliability, Safety, Security risks

Spare/expandable capacity Performance, Reliability Added cost

Product line architecture,

reusable components

Cost, Schedule, Reliability Some loss of performance vs.

optimized stovepipes




07/08/09 38

Value/Risk-Based Tradespace Analysis - Early Startup: Risk due to low dependability

- Commercial: Risk due to low dependability

- High Finance: Risk due to low dependability

- Risk due to market share erosion

COCOMO II: 0 12 22 34 54 Added % test time

COQUALMO: 1.0 .475 .24 .125 0.06 P(L)

Early Startup: .33 .19 .11 .06 .03 S(L)

Commercial: 1.0 .56 .32 .18 .10 S(L)

High Finance: 3.0 1.68 .96 .54 .30 S(L)

Market Risk: .008 .027 .09 .30 1.0 REm

Sweet

Spot

Combined Risk Exposure

0

0.2

0.4

0.6

0.8

1

VL L N H VH RELY

RE =

P(L) * S(L)

Market Share

Erosion

Early Startup

Commercial

High Finance

©USC-CSSE



Magnitude of Overrun Problem: DoD




Magnitude of Overrun Problem: Standish Surveys of Commercial Projects

Year 2000 2002 2004 2006 2008

Within budget and schedule 28 34 29 35 32

Prematurely cancelled

23 15 18 19 24

Budget or schedule overrun

49 51 53 46 44




Some Frequent Overrun Causes

• Conspiracy of Optimism

• Effects of First Budget Shortfall

– System Engineering

• Decoupling of Technical and Cost Analysis

– Overfocus on Performance, Security, Functionality

• Overfocus on Acquisition Cost

• Assumption of Stability

• Total vs. Incremental Commitment




The Conspiracy of Optimism and

The Cone of Uncertainty




Copyright © USC-CSSE 43

Effects of First Budget Shortfall:

Added Cost of Weak System Engineering Calibration of COCOMO II Architecture and Risk Resolution

(RESL) factor to 161 project data points

October 16, 2012



44 October 16, 2012

Assumption of Stability vs. Rapid Change – Need evolutionary/incremental vs. one-shot development

Feasibility

Concept of

Operation

Rqts.

Spec.

Plans

and

Rqts.

Product

Design

Product

Design

Spec.

Detail

Design

Spec.

Detail

Design

Devel. and

Test

Accepted

Software

Phases and Milestones

Relative

Cost Range x

4x

2x

1.25x

1.5x

0.25x

0.5x

0.67x

0.8x

Uncertainties in competition,

technology, organizations, mission

priorities


Documents

A Value-Based Orthogonal Framework for Improving Life ... · Design for Maintainability, Evolvability Automate Operations Elements Anticipate, Prepare for Change Value- and Architecture-Based