Embed Size (px)
Citation preview
Dr. Ricardo Valerdi es profesor asociado en la Universidad de Arizona en el departamento de Ingeniería de Sistemas e Industriales. Anteriormente fue profesor investigador en el grupo de sistemas en el Instituto Tecnológico de Massachusetts. El Dr. Valerdi fue ganador del premio Frank Freiman, el máximo reconocimiento de la Asociación Internacional de Estimación de Costos y Análisis, además de haber creado y liderado el programa “la Ciencia del Béisbol” como un medio para la enseñanza de la ciencia en educación pre-universitaria. Tiene como clientes a cinco equipos de béisbol profesional para cuales ha desarrollado programas educacionales en Los Angeles, San Diego, Phoenix, Denver y Washington. También ha ganado premios de mejor artículos en la revistas Systems Engineering Journal y Defense Acquisition Review Journal y mejor presentación en varios congresos internacionales. Ha conseguido más de siete millones de dólares en fondos de investigación de la marina, armada, y fuerza aérea americana para desarrollar proyectos de estimación de costo, modelaje de sistemas, y estudios de contusiones del cerebro. En el ámbito gremial, es fundador de una revista técnica y editor en jefe del Journal of Cost Analysis and Parametrics. Ha sido miembro del Consejo del International Council on Systems Engineering y recibió el reconocimiento de Visiting Fellow de la Real Academia de Ingeniería del Reino Unido. El Dr. Valerdi obtuvo su doctorado en ingeniería de sistemas en la Universidad del Sur de California y estudió psicología en Harvard.
1
Cost Estimation of Complex Systems
Prof. Ricardo Valerdi Systems & Industrial Engineering University of Arizona
Nov 5, 2015
Academia de Ingeniería Palacio de Minería
México DF
Take-Aways
1. Major constraints to product size are economic
2. Cost ≈ f(Effort) ≈ f(Size) ≈ f(Complexity) 3. Requirements understanding and “ilities”
are the most influential on cost
2
3
My Research Creating cost models that help stakeholders reason about the economic impacts of their decisions of the systems/products they want to build and operate Principle #1 A solution that is too expensive, is not a solution. Principle #2 Cost of products are a function of complexity and size.
The Iron Triangle
4 4
Performance
Cost Schedule
© N
orth
rop
Gru
mm
an M
issi
on S
yste
ms
Theoretical Lenses
6
Software Engineering Economics
(Boehm 1980s)
Biases and heuristics
(Kahneman and Tversky1970s)
Bounded Rationality (Simon 1990s)
Sociotechnical Systems
(Cherns 1970)
Cybernetics (von Neumann,1940s)
Total Cost of Ownership
(Smith, 1700s)
7
Cost Commitment on Projects
Detail Designand
Development
100
25
50
75
Conceptual-Preliminary
Design
Constructionand/or
ProductionSystem Use, Phaseout,
and Disposal
NEED
% Commitment to Technology,Configuration, Performance, Cost, etc.
Cost Incurred
System-Specific Knowledge
Ease of Change
Blanchard, B., Fabrycky, W., Systems Engineering & Analysis, Prentice Hall, 2010.
8
How is Systems Engineering Defined? • Acquisition and Supply
– Supply Process – Acquisition Process
• Technical Management – Planning Process – Assessment Process – Control Process
• System Design – Requirements Definition Process – Solution Definition Process
• Product Realization – Implementation Process – Transition to Use Process
• Technical Evaluation – Systems Analysis Process – Requirements Validation Process – System Verification Process – End Products Validation Process
EIA/ANSI 632, Processes for Engineering a System, 1999.
Systems Engineering Effort vs. Program Cost
9
Gruhl, W. “Lessons Learned, Cost/Schedule Assessment Guide,” NASA Comptroller’s Office, 1992.
10
COSYSMO Data Sources (2002-present) Boeing Integrated Defense Systems (Seal Beach, CA) Raytheon Intelligence & Information Systems (Garland, TX)
Missile Systems (Tucson, AZ) Northrop Grumman Mission Systems (Redondo Beach, CA) Lockheed Martin Transportation & Security Solutions (Rockville, MD)
Integrated Systems & Solutions (Valley Forge, PA) Systems Integration (Owego, NY) Aeronautics (Marietta, GA) Maritime Systems & Sensors (Manassas, VA; Baltimore, MD; Syracuse, NY)
General Dynamics Maritime Digital Systems/AIS (Pittsfield, MA) Surveillance & Reconnaissance Systems/AIS (Bloomington, MN)
BAE Systems National Security Solutions/ISS (San Diego, CA) Information & Electronic Warfare Systems (Nashua, NH)
SAIC Army Transformation (Orlando, FL) Integrated Data Solutions & Analysis (McLean, VA)
L-3 Communications Greenville, TX
12
Modeling Methodology
Valerdi (2005)
13
Results of Bayesian Update: Using Prior and Sampling Information
1.06
Literature,behavioral analysis
A-prioriExperts’ Delphi
Noisy data analysis
A-posteriori Bayesian update
Produc tivity R ange =Highes t R ating /L owes t R ating
1.451.51
1.41
Valerdi (2005)
14
COSYSMO Scope• Addresses first four phases of the system
engineering lifecycle (per ISO/IEC 15288)
• Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632)
Conceptualize Develop Oper Test & Eval
Transition to Operation
Operate, Maintain, or Enhance
Replace orDismantle
EIA/ANSI 632, Processes for Engineering a System, 1999.
ISO/IEC 15288, System Life Cycle Processes, 2008
15
COSYSMO
SizeDrivers
EffortMultipliers
Effort
Calibration
# Requirements# Interfaces# Scenarios# Algorithms
+3 Adj. Factors
- Application factors- 8 factors
- Team factors- 6 factors
COSYSMO Operational Concept
Valerdi (2005)
16
Software Cost Estimating Relationship
cSaMM e ⋅⋅=
cKDSIMM ⋅⋅= 05.1)(4.2
Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.
MM = Man months a = calibration constant S = size driver E = scale factor c = cost driver(s) KDSI = thousands of delivered source instructions
17
COSYSMO Model Form
∏∑=
⋅⎟⎟⎠
⎞⎜⎜⎝
⎛Φ+Φ+Φ⋅=
14
1,,,,,, )(
jj
E
kkdkdknknkekeNS EMwwwAPM
Where: PMNS = effort in Person Months (Nominal Schedule)A = calibration constant derived from historical project data k = {REQ, IF, ALG, SCN}wx = weight for “easy”, “nominal”, or “difficult” size driver = quantity of “k” size driverE = represents diseconomies of scaleEM = effort multiplier for the jth cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort.
xΦ
Valerdi (2005, 2008)
18
UNDERSTANDING FACTORS– Requirements understanding – Architecture understanding– Stakeholder team cohesion – Personnel experience/continuity
COMPLEXITY FACTORS– Level of service requirements– Technology Risk– # of Recursive Levels in the Design– Documentation Match to Life Cycle Needs
OPERATIONS FACTORS– # and Diversity of Installations/Platforms– Migration complexity
PEOPLE FACTORS– Personnel/team capability – Process capability
ENVIRONMENT FACTORS– Multisite coordination – Tool support
Cost Driver Clusters
Valerdi (2005)
19
Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team.
1.5 1.22 1.00 0.81 0.65
Viewpoint Very Low Low Nominal High Very High
Culture § Stakeholders with diverse expertise, task nature, language, culture, infrastructure § Highly heterogeneous stakeholder communities
§ Heterogeneous stakeholder community § Some similarities in language and culture
§ Shared project culture
§ Strong team cohesion and project culture § Multiple similarities in language and expertise
§ Virtually homogeneous stakeholder communities § Institutionalized project culture
Compatibility § Highly conflicting organizational objectives
§ Converging organizational objectives
§ Compatible organizational objectives
§ Clear roles & responsibilities
§ Strong mutual advantage to collaboration
Familiarity and trust
§ Lack of trust § Willing to collaborate, little experience
§ Some familiarity and trust
§ Extensive successful collaboration
§ Very high level of familiarity and trust
Valerdi (2005)
Technology Risk The maturity, readiness, and obsolescence of the technology being implemented. Immature or obsolescent technology will require more Systems Engineering effort. Viewpoint Very Low Low Nominal High Very High
Lack of Maturity
Technology proven and widely used throughout industry
Proven through actual use and ready for widespread adoption
Proven on pilot projects and ready to roll-out for production jobs
Ready for pilot use
Still in the laboratory
Lack of Readiness
Mission proven (TRL 9)
Concept qualified (TRL 8)
Concept has been demonstrated (TRL 7)
Proof of concept validated (TRL 5 & 6)
Concept defined (TRL 3 & 4)
Obsolescence
- Technology is the state-of-the-practice - Emerging technology could compete in future
- Technology is stale - New and better technology is on the horizon in the near-term
- Technology is outdated and use should be avoided in new systems - Spare parts supply is scarce
20
Valerdi (2005)
Migration complexity This cost driver rates the extent to which the legacy system affects the migration complexity, if any. Legacy system components, databases, workflows, environments, etc., may affect the new system implementation due to new technology introductions, planned upgrades, increased performance, business process reengineering, etc.
Viewpoint Nominal High Very High Extra High
Legacy contractor
Self; legacy system is well documented. Original team largely available
Self; original development team not available; most documentation available
Different contractor; limited documentation
Original contractor out of business; no documentation available
Effect of legacy system on new system
Everything is new; legacy system is completely replaced or non-existent
Migration is restricted to integration only
Migration is related to integration and development
Migration is related to integration, development, architecture and design
21
Valerdi (2005)
22
Cost Drivers Ordered by Effort Multiplier Ratio (EMR)
Valerdi (2005)
ISO/IEC 15288
Conceptualize Develop Transition to
Operation
Acquisition & Supply
Technical Management
System Design
Product Realization
Technical Evaluation
Operational Test &
Evaluation A
NSI
/EIA
632
Effort Profiling
23
Valerdi, et al. (2007)
Benefits of Local Calibration
Before local calibration
After local calibration
Syst
ems
Engi
neer
ing
Effo
rt (S
E H
ours
)
System Size (eReq)
Syst
ems
Engi
neer
ing
Effo
rt (S
E H
ours
)
System Size (eReq)
24
Wang, G., Valerdi, R. and Fortune, J., “Reuse in Systems Engineering,” IEEE Systems Journal, 4(3), 376-384, 2010.
25
Prediction Accuracy PRED(30)
PRED(25)
PRED(20)
PRED(30) = 100% PRED(25) = 57%
Valerdi (2005)
Limitations of Parametric Modeling1. Dataset shift2. Calibration is biased by successful projects
because successful projects share data, bad ones don’t
3. Model does not work outside of calibrated range
4. A fool with a tool is still a fool
26
Valerdi, R., “Heuristics for Systems Engineering Cost Estimation,” IEEE Systems Journal, 5(1), 91-98, 2011.
Academic prototype
Commercial Implementations
Proprietary Implementations
COSYSMO-R
SECOST
SEEMaP
ImpactAcademic Curricula
Intelligence Community Sheppard Mullin, LLC
Policy & Contracts
Model
∏∑=
⋅⎟⎟⎠
⎞⎜⎜⎝
⎛Φ+Φ+Φ⋅=
14
1,,,,,, )(
jj
E
kkdkdknknkekeNS EMwwwAPM
COSYSMO
Take-Aways
1. Major constraints to product size are economic
2. Cost ≈ f(Effort) ≈ f(Size) ≈ f(Complexity) 3. Requirements understanding and “ilities”
are the most influential on cost
28
Why is the ER so slow? (http://www.healthcarereborn.com)
Tied for 1st place in Stevens Institute of Technology Experience Accelerator Competition (www.experience--accelerator.org)
29
RobCup
31
Speed = 90 mph Angular velocity = 400 degrees/sec Weight = 9 oz Temp = 78 degrees ERA = 2.53
Pitch count = 54 Height = 6’ 2” Weight = 225 lbs
Batting Avg. = 245 Balls = 2 Strikes = 2 HR = 5
Concussion: To violently shake Symptoms (cognitive, somatic, emotional) • Headache or a feeling of pressure in the head • Temporary loss of consciousness • Confusion or feeling as if in a fog • Amnesia surrounding the traumatic event • Dizziness or "seeing stars" • Ringing in the ears • Nausea • Vomiting • Slurred speech • Delayed response to questions • Appearing dazed • Fatigue • Concentration and memory complaints • Irritability and other personality changes • Sensitivity to light and noise • Sleep disturbances • Psychological adjustment problems and depression • Disorders of taste and smell
32
