Embed Size (px)
Citation preview
National Defense Industrial Association9th Annual Systems Engineering (SE) Conference
October 2006
Al MinkSystems Value / GMU
The Value of Systems Engineering
What do we know about it?How do we discover more?
Al Mink 2Value of SE - NDIA
Value of SEOverview
1. The Problem
2. What We Know Today
3. The Race to Discover More
4. Conclusions
Al Mink 3Value of SE - NDIA
Value of SEThe Problem (Stakeholder Analysis)
Customers Unsure of how to evaluate bidsMay not receive best value for the systems they acquireDoD #1 SE Issue – “Inconsistent SE Practices across life cycle”
Industry (System Developers & Integrators)Unsure of what to bid, and later loath to add SE costs
Associations & AcademiaUnable to fully satisfy their members and students
SE professionalsLack rigorous justification for their recommendations
What – and how much – SE is appropriate for a particular system development program?
Al Mink 4Value of SE - NDIA
Value of SEThe Problem (IDEF 0 View)
SE produces more than products -- It affects the value of operational system produced
Buede pg 413
Perform SE
Operational System
SE Products
Al Mink 5Value of SE - NDIA
Value of SEThe Problem (Pareto View)
Buede, Figure 1.2
SE activity upfront affects entire development
Al Mink 6Value of SE - NDIA
Value of SEWhat we know today – Studies & Models
Gruhl, National Avionics and Space Administration (NASA), 1992Compared upfront expenditures to eventual cost growth
Herbsleb, Software Engineering Institute (SEI), 1994Studied ROI on process improvement in software
Honour, International Council on Systems Engineering (INCOSE), 2002Surveyed industry to compare SE Effort to cost & schedule
Valerdi & Boehm, Constructive System Engineering Cost Model (COSYSMO), 2004Developed parametric estimation model similar to COCOMO
Others…
Al Mink 7Value of SE - NDIA
Value of SEWhat we know today – NASA Study
Total Program Overrun32 NASA Programs
R2 = 0.5206
0
20
40
60
80100
120
140
160
180
200
0 5 10 15 20
Definition Percent of Total Estimate
Prog
ram
Ove
rrun
Definition $Definition Percent = ---------------------------------- Target + Definition$
Actual + Definition$Program Overrun = ---------------------------------- Target + Definition$
GRO76OMV
GALL
IRAS
TDRSS
HST
TETH
LAND76
MARS
MAG
GOES I-M
CENACT
CHA.REC.
SEASAT
DE
UARS
SMM
EDO
ERB77
STS
LAND78
COBE
GRO82
ERB88VOY
EUVE/EP
ULYS
PIONVEN IUE ISEE
HEAO
Source Werner GruhlNASA Comptroller’s Office& Honour 2004
Al Mink 8Value of SE - NDIA
Value of SEWhat we know today – INCOSE Study
0.6
1.0
1.4
1.8
2.2
2.6
3.0
0% 4% 8% 12% 16% 20% 24% 28%
SE Effort = SE Quality * SE Cost/Actual Cost
Act
ual/P
lann
ed C
ost
Average Cost Overrun
90% Assurance (1.6σ)Source:Honour2004
Al Mink 9Value of SE - NDIA
Value of SEWhat we know today – ROI of SE
SE ROI by Software Size of System
Boehm & Valerdi, 2006
-4%99%177%356%840%-10,000
-30%42%91%204%512%-1,000
-54%-10%18%80%248%-100
-77%-58%-45%-20%52%-10
ExtraHigh
VeryHigh
HighNominalLowVeryLow
KSLOC
Al Mink 10Value of SE - NDIA
Value of SEWhat we know today – ROI of SE
SE Activities Affect Software Development
Boehm & Valerdi, 2006
% A
dded
Cos
t
Development Cost Increase: Minimal SE compared to Robust SE
Al Mink 11Value of SE - NDIA
Value of SEWhat we know today – COSYSMO
Limited ability to estimate “effort”
Valerdi, 2005
Pred(30) 50% uncalibratedPred(30) 70% calibrated
Al Mink 12Value of SE - NDIA
Value of SEWhat we know today – Summary
Today we posses a limited understanding of the SE effort required for success of a project
8-15%in
Aggregate*Cost,
Relative to budget
Of General Nature
* COSYSMO tracks 33 SE activities
SE effort success project
Al Mink 13Value of SE - NDIA
Value of SEThe Race to Discover More
Four Separate Efforts Underway
University of South Australia
Honourcode, Inc.
Al Mink 14Value of SE - NDIA
Value of SEThe Race to Discover More
By SE activity
On cost, schedule, &
quality
Tailored to
project
All four should increase our understanding of the SE effort required for success of a project
Al Mink 15Value of SE - NDIA
Value of SEThe Race to Discover More - Methodology
All Four Appear to Follow a General Approach
1. Form Team2. Develop Approach3. Identify Projects4. Collect Data5. Analyze Data6. Publish Results
Al Mink 16Value of SE - NDIA
Value of SEThe Race to Discover More - Methodology
Similar Approach – IDEF Depiction
Al Mink
4
1
2
3
5
6
4. DataCollection
How the pieces fit together
Al Mink 17Value of SE - NDIA
Value of SEThe Race to Discover More
Capturing Data – Three Categories
Al MinkAl Mink
4.3Characterize
Project
4.2Measure Success
4.1Measure SE Effort
4.1Measure SE Effort
(Activities)
Al Mink 18Value of SE - NDIA
Value of SEThe Race to Discover More – Define SE Activities
Defining “SE Activities” – One View
Buede pg 416
SE 101
Al Mink 19Value of SE - NDIA
Value of SEThe Race to Discover More – Define SE Activities
Defining “SE Activities” – Many Views
Fragmented by domain opinions
• Military – DOD/MOD• Space - NASA/ESA• Commercial products• Aircraft• Automobiles• Nuclear waste• Process engineering• Tool vendors• Etc. Etc. Etc.
Fragmented by discipline opinions
• Technical leaders• System architects• System analysts• Requirements
engineers• Operations analysts• Design engineers
Fragmented by standards
• ANSI/EIA-632• IEEE-1220• ISO-15288• CMMI• MIL-STD-499C
Honour 2005
Al Mink 20Value of SE - NDIA
Value of SEThe Race to Discover More – Define SE Activities
How the Different Efforts Define “SE Effort”
ANSI/EIA-632
StandardsStandards
IEEE-1220
ISO/IEC-15288
CMM/CMMI
MIL-STD-499C
Al Mink 21Value of SE - NDIA
Value of SEEmerging Approaches to Move Forward – Define Other Measures
In addition to defining & measuring SE Effort…
Cost, schedule, &
quality
Tailored to
project
Success factors- EVMS- Award Fee- Requirements Trace- Others…
Project characteristics- Size ($)- Size (hours)- Technology- Complexity- Others…
Al Mink 22Value of SE - NDIA
Value of SEThe Race to Discover More - Methodology
Similar Approach – IDEF Depiction
Al Mink
4
1
2
3
5
6
4. DataCollection
One Other Difference – Collection Mechanism
Mechanism
Al Mink 23Value of SE - NDIA
Value of SEThe Race to Discover More – Define SE Activities
How the Different Efforts Collect Data
Data CollectionMechanisms
Data CollectionMechanisms
SurveySurvey
Interview
WorkShops
Al Mink 24Value of SE - NDIA
Value of SEConclusions
Value of SERemains fundamental to furthering SE as a respected discipline
Four approaches underway to determine SE ValueWith a fifth – Bob Bruff – on the horizon…
They share commonalities, but also differ:Differing types of projectsDiffering SE Activities & DeliverablesDiffering success factors (cost, schedule, quality, etc.)
Challenges RemainUseful project data – may not be widely availableFour separate projects – what if they report different results?Success may be elusive – “The Shangri-La of ROI” (Sheard 2000)
Make a difference! Support these approaches
Al Mink 25Value of SE - NDIA
Value of SEConclusion
Sue Rose*[email protected]
Points of Contact
Eric [email protected]
Ricardo [email protected]
* Restricted Participation
