Upload
nathan-hale
View
216
Download
0
Embed Size (px)
Citation preview
Value of Systems Engineering; Summary Report 1/04 1
Value of Systems Engineering
Summary report SECOE and related projects
Eric HonourINCOSE Director, Sponsored Research
Value of Systems Engineering; Summary Report 1/04 2
Agenda
Background Heuristic Claims of SE Value
Gathered results on Value of SE NASA Tracking 1980s “Boundary Management” study “Large Engineering Projects” MIT study “Impact of SE at NASA” (SECOE 02-02)
“Impact of SE on Quality & Schedule” Boeing “SE Effectiveness” IBM study “Value of SE” research (SECOE 01-03)
Value of Systems Engineering; Summary Report 1/04 3
Heuristic Claim of SE
Better systems engineering leads to Better system quality/value Lower cost Shorter schedule
SYSTEMDESIGN
DETAILDESIGN
PRODUCTIONINTEGRATION TEST
Traditional Design
Time
Risk
SavedTime/Cost
“System Thinking” Design Time
Risk
Value of Systems Engineering; Summary Report 1/04 4
NASA Tracking 1980s
Source Werner GruhlNASA Comptroller’s Office
Total Program Overrun32 NASA Programs
R2 = 0.5206
0
20
40
60
80
100
120
140
160
180
200
0 5 10 15 20
Definition Percent of Total Estimate
Pro
gra
m O
ve
rru
n
Definition $Definition Percent = ---------------------------------- Target + Definition$
Actual + Definition$Program Overrun = ---------------------------------- Target + Definition$
GRO76OMV
GALL
IRAS
TDRSS
HST
TETH
LAND76
MARS
MAG
GOES I-M
CENACT
CHA.REC.
SEASAT
DE
UARS
SMM
EDO
ERB77
STS
LAND78
COBE
GRO82
ERB88VOY
EUVE/EP
ULYS
PIONVEN IUE ISEE
HEAO
Value of Systems Engineering; Summary Report 1/04 5
“Boundary Management” Study
Study of 45 high-tech new product development teams
...Ancona and Caldwell, Research Technology Management, 1990
Significant portion of time is spent at team boundaries
Individual Time Spent Individual Time Spent Outside TeamOutside Team 14%*14%*
* Typically limited to few individuals* Typically limited to few individuals
Within TeamWithin Team 38% 38%
AloneAlone 48%48%
Value of Systems Engineering; Summary Report 1/04 6
Boundary Management in Technical Teams
Boundary management occurs in four different roles.
Level of effort in each role changes with time.
• AmbassadorAmbassadorBuffering, building support, Buffering, building support, reporting, strategyreporting, strategy
• Task CoordinatorTask CoordinatorLateral group coordination, info Lateral group coordination, info transfer, planning, negotiatingtransfer, planning, negotiating
• ScoutScoutObtain possibilities from outside - Obtain possibilities from outside - interface with marketinginterface with marketing
• GuardGuardWithhold information, prevent Withhold information, prevent disclosuredisclosure
CREATIONCREATIONDEVELOPMENTDEVELOPMENT
DIFFUSIONDIFFUSION
Value of Systems Engineering; Summary Report 1/04 7
“Boundary Management” Study
Significant Findings:
High-performing teams did more external interaction than low-performing teams
System technical leadershipcreates greater success
Internal team dynamics (goals, processes, individual satisfaction) did not correlate with performance
Process definition is important but not sufficient.
Value of Systems Engineering; Summary Report 1/04 8
Cost Targets
Schedule Targets
Objective Targets
82%
Percent of Projects Meeting:
“Large Engineering Projects”
Study of 60 LEPs (power generation, transportation, oil production, technology)
The Strategic Management of Large Engineering Projects, MIT Press 2000
Evaluation by interviews and by objective and subjective quality measures.
72%
45% 18% 37%
Failed!
Value of Systems Engineering; Summary Report 1/04 9
“Large Engineering Projects”
Significant Findings:
Most important determinant was a coherent, well-developed organizational/team structure
A structure of leadership creates greater success
Technical difficulties, social disturbance, size were not statistically linked to performance All projects had turbulent events
Technical excellence could not save a socially unacceptable project
Process definition is important but not sufficient.
Value of Systems Engineering; Summary Report 1/04 10
Impact of SE at NASA (SECOE 02-02)
Survey research within NASA Form with 38 questions, answers on graded scale Typical questions:
• On your most recent project, what percent of your total project cost was spent on Systems Engineering tasks?
• On your most recent project, did systems engineering shorten or lengthen the planned schedule?
Aggressive survey pursuit with management push NASA: 250 sent, 54% valid response INCOSE: 650 sent, 38% valid response
Engineering of Complex Systems – The Impact of Systems Engineering at NASA, A.K.P.Kludze, Jr. doctoral dissertation George Washington Univ. 2003
Value of Systems Engineering; Summary Report 1/04 11
Response Demographics
Participating Organization
NASA 136INCOSE 243Total 379
Work Experience
0-10 41%11-20 37%21+ 22%
Age Groups
20-39 29%40-59 61%60+ 10%
Education Level
Bachelor 34%Master 55%Doctor 11%
Job Titles
SE 56%PM 17%Other 27%
...significant differences by organization
Value of Systems Engineering; Summary Report 1/04 12
Key Survey Results - Cost
Percent Spent on SE
0-5% 6-10% 11-15% 16% +
• Respondents marked bracket to show percent of total cost spent on SE on last project.
• Mode at 6-10% of project• Few projects spent 11-15%• Unexplained bimodal response >16%
(perhaps interpretation of “project”)
Cost Benefit of SE
Very Poor Poor Fair Good Excellent
• Respondents believe strongly in cost benefit of SE
• In secondary question, few respondents could quantify
Value of Systems Engineering; Summary Report 1/04 13
Key Survey Results - Schedule
At What Stage is SE Most Effective?
Very Early Midway End No Matter No Need
• Vast majority of respondents believe that SE is most effective very early in a project.
Impact of SE on Schedule
Shorten Lengthen Don’t Know
• INCOSE respondents believe SE shortened schedule on most recent project
• NASA respondents uncertain• Secondary questions uncertain
in the quantification.
...other results available in dissertation
Value of Systems Engineering; Summary Report 1/04 14
Impact of Systems Engineering on Quality and Schedule
Empirical evidence obtained from three parallel (same time) projects Each developed a complex, robotic Universal
Holding Fixture (UHF) Each used a different level of SE Results are compared
Trait UHF1 UHF2 UHF3
Size 10’ x 40’ 8’ x 50’ 6’ x 14’
Accuracy ±0.005” ±0.003” ±0.003”
Contact Sensors None 57 108
Vacuum Sensors 1 70 108
Real-time checks No Yes Yes
Probe contours No Yes Yes
NC interface No Yes Yes
Impact of Systems Engineering on Quality and Schedule – Empirical Evidence, W. Forrest Frantz, Boeing Corp. 1995
Value of Systems Engineering; Summary Report 1/04 15
Project Differences
Project Trait UHF1 UHF2 UHF3
Sys mgmt experience Low Low-Medium Low-Medium
Subcontract approach Design Reviews Full-time SE on site
Access to SE support Low High, but not used
High, used
Requirements approach Token req’s Complete, detailed, integrated req’s
Design approach HW/SW specs, multi-org approach
Functional specs fully address HW/SW processes and interfaces
Functional adherence Design docs took precedence; specs updated per design
Specs followed, CCB control
Design reviews Weekly team reviews
Formal internal; little external
Formal internal and external
Integration approach Patterned after design
Drive by functional specs; defined early in life cycle
Acceptance testing Defined in high-level plan
Formal tests based on Req’s and Functional specs
Value of Systems Engineering; Summary Report 1/04 16
Impacts
Use of better SE reduced Overall cycle time Time to create req’s Time to design/produce Time to test
...even in the face of more complex, higher quality systems!
0 50 100
UHF3
UHF2
UHF1
Overall Development Time (weeks)
0 10 20 30
UHF3
UHF2
UHF1
Requirements to RFP (weeks)
0 20 40 60
UHF3
UHF2
UHF1
Design to Production (weeks)
Value of Systems Engineering; Summary Report 1/04 17
Systems Engineering Effectiveness
Study of 8 software product development projects during upgrade of SE processes Determining Systems Engineering Effectiveness, Bruce Barker, IBM Commercial Products, Conference on Systems Integration,
Stevens Institute 2003
Evaluation by cost and schedule against a standard estimating method.
Identifyaffected
components
EvaluateImpact,
Complexity
Convertto
“points”
EstimateCost,
Schedule
Historical Database, Cost per “Point”
NewProductConcept
ProductLine Architecture
Costing method applies only to project management, business management, systems engineering, system integration, and delivery into production. Application development costs are not included.
© Copyright IBM Corp 2003Used With Permission
Value of Systems Engineering; Summary Report 1/04 18
Project Data
$K/Point Averages
Without SE
With SE
$1,350/pt
$944/pt
200020012002
$1,454/pt$1,142/pt
$818/pt
2000
2000
2001
2001
2001
2002
2002
2002
Project 1
Project 2
Project 3
Project 4
Project 5
Project 6
Project 7
Project 8
12,934
1,223
10,209
8,707
4,678
5,743
14,417
929
18,191
2,400
11,596
10,266
5,099
5,626
10,026
1,600
0
0
9.2
0
10.7
14.4
10.2
16.0
1,406
1,962
1,136
1,179
1,090
980
695
1,739
Year Project “Points” Cost ($K)
SE Costs (%)
$K/Point
© Copyright IBM Corp 2003Used With Permission
Value of Systems Engineering; Summary Report 1/04 19
Timeline of Projects
1999 2001 2002
500
1000
1500
2000
2500
2000
SE organization created
SE Process documented
SE Formal Training Started
Project w/o SE
Project With SE
As the Systems Engineering process has been enabled and integrated through the organization, productivity has increased
Yearly Avg
Cost per “Point”
© Copyright IBM Corp 2003Used With Permission
Value of Systems Engineering; Summary Report 1/04 20
Systems Engineering Effectiveness
Significant Findings: Impact and complexity provide an effective
method to perform parametric costing.Early parametric costing works.
© Copyright IBM Corp 2003Used With Permission
Preliminary data indicates that the use of Systems Engineering will improve project productivity when effectively combined with the Project Management and Test Processes.
Systems engineering improves productivity.
$K/Point Averages
Without SE
With SE
$1,350/pt
$944/pt
200020012002
$1,454/pt$1,142/pt
$818/pt
Value of Systems Engineering; Summary Report 1/04 21
“Value of SE” (SECOE 01-03)
Multi-year effort to obtain statistical data Correlate amount/quality of SE with project
quality/success
SE Effort
Parameterized by• Technical “Size”• Technical Complexity• Risk Level
Development Quality
(function ofTechnical Value,
Cost, Schedule,
Risk)6-10% ?
Value of Systems Engineering; Summary Report 1/04 22
Respondent Data
43 respondents 1 project not completed Values $1.1M - $5.6B SE Cost 0.3% - 26%
0
2
4
6
8
10
0 5 10 15 20 25
SE Cost %
Nu
mb
er
of
Pro
jec
ts
Cost, schedule, quality correlate better with “Systems Engineering Effort”:
SEE = SE Qual * (SE Cost %)
0
2
4
6
8
10
0 5 10 15 20 25
SE Effort = SE Quality * (SE Cost %)
Nu
mb
er
of
Pro
jec
ts
Value of Systems Engineering; Summary Report 1/04 23
0.6
1.0
1.4
1.8
2.2
2.6
3.0
0% 4% 8% 12% 16% 20% 24% 28%
SE Effort = SE Quality * SE Cost/Actual Cost
Act
ual
/Pla
nn
ed C
ost
Cost Overrun vs. SE Effort
Source: SECOE 01-03INCOSE 2003
Average Cost Overrun
90% Assurance (1.6)
Value of Systems Engineering; Summary Report 1/04 24
0.6
1.0
1.4
1.8
2.2
2.6
3.0
0% 4% 8% 12% 16% 20% 24%
SE Effort = SE Quality * SE Cost/Actual Cost
Act
ual
/Pla
nn
ed S
ched
ule
Schedule Overrun vs. SE Effort
Source: SECOE 01-03INCOSE 2003
Average Schedule Overrun
90% Assurance (1.6)
Value of Systems Engineering; Summary Report 1/04 25
0.4
0.6
0.8
1.0
1.2
0% 4% 8% 12% 16% 20% 24%
SE Effort
Dev
elo
pm
ent
Qu
alit
y(C
ost
/Sch
edu
le B
ased
)
Test Hypothesis: Quality
Source: SECOE 01-03INCOSE 2003
SE Effort
Qu
ality Hypothesis
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
0% 4% 8% 12% 16% 20% 24%
SE Effort
Co
mp
arat
ive
Su
cces
s
Value of Systems Engineering; Summary Report 1/04 26
Conclusions “Value of SE”
SE effort improves development quality Cost, schedule, subjective Hypothesis is supported by the data
Optimum SE effort is 10-15% or more Cost, schedule overruns are minimized However, note wide dispersion of data Also note few data points at this level; most
projects spent far less Quality of the SE effort matters
Lower quality SE reduces effectiveness
Value of Systems Engineering; Summary Report 1/04 27
Company Participation
Project benchmarking service funded by participants Aggregated data shared among participants Raw data protected by data blinding
SECOEBlinded
Raw DataProject
Benchmarkingdata
Statistical dataValue of SE
Reportedresults
ParticipatingCompanies
Public
1 month
ParticipatingCompanies(all data)
Quarterly
Every two years
Value of Systems Engineering; Summary Report 1/04 28
Company Participation
Data gathering Select 4-6 programs One day session per participating company
every 4 months 1½-hour sessions with PM+SE of each program Data gathered by two SECOE researchers Forms & notes do not identify programs
Reports Benchmark report within 30 days of each
quarterly session, compares to all prior data 4-month reports to all participants with
aggregated results from all data, all sources Participation price $??K for each year
Value of Systems Engineering; Summary Report 1/04 29
Questions?
Eric Honour
INCOSE Director for Sponsored ResearchPensacola, FL, USA+1 (850) 479-1985 [email protected]