March 6, 2012 Mauricio E. Peña

University of Southern California

Center for Systems and Software Engineering USC CSSE 2012 Annual Research Review

1

March 6, 2012

Mauricio E. Peña

Mathematical Formulation and Validation of the Impact of Requirements Volatility on

Systems Engineering Effort


Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review

2

Outline

• Motivation and introduction

• Research methods

• Observations from the research

• Model Development

• Results– Model Calibration

– Evaluation of Model Performance

– Sensitivity Analysis

– Cross-Validation

• Conclusion



3

Importance of Understanding Requirements Volatility

• Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1

• Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2

• The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3

System developers often lack effective methods and tools to account for and manage requirements volatility

Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393



4

Requirements Volatility is Expected

• Changes to requirements are a part of our increasingly complex systems & dynamic business environment

– Stakeholders needs evolve rapidly

– The customer may not be able to fully specify the system requirements up front

– New requirements may emerge as knowledge of the system evolves

– Requirements often change during the early phases of the project as a result of trades and negotiations

Sources: Kotonya and Sommerville (1995); Reifer (2000)

Requirements volatility must be anticipated and managed



5

CSSE Parametric Cost Models

• The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates

• COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort

• Built on experience from COCOMO 1981, COCOMO II

• During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers

Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm



COSYSMO Operational Concept

Source: Valerdi (2005)

Volatility Factor

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

- Application factors-8 factors

- Team factors-6 factors

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

- Application factors-8 factors

- Team factors-6 factors

6



7

Research Methods

8

USC

C S E University of Southern CaliforniaCenter for Software Engineering

USC-CSE Annual Research Review – 3/17/03

Analyze Existingliterature

1

2

3

4

5

6

7

PerformBehavioral Analysis

Identify RelativeSignificance

Perform Expert-Judgement, DelphiAssessment

Gather Project Data

Determine BayesianA-Posteriori Update

Gather more data;refine model

A-PRIORI MODEL+

SAMPLING DATA=

A-POSTERIORI MODEL

Source: Boehm et al (2000)

Literature Review & 6 Workshops completed

Data gathered from 25 industry projects

We are here



Organizations that Participated in the Research

• The Aerospace Corporation

• Northrop Grumman Corporation

• The Boeing Company

• Raytheon

• United Launch Alliance

• BAE

• TI Metricas Ltda.

• IBM

• Distributed Management

• MIT

• USC

• Lockheed Martin

• Ericsson España

• Samsung SDS

• Rolls Royce

• Softstar

• Texas Tech

• The US Army

• The US Navy

• The US Air force

• The Australian Department of Defense

8



9

Requirements Volatility Observations

1. Requirements volatility is caused by an identifiable set of project and organizational factors

2. The level of requirements volatility is a function of the system life cycle phase

3. Requirements volatility leads to an increase in project size and cost

4. The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle

5. The impact of requirements volatility may vary depending on the type of change: added, deleted, or modified



Model Form

• Incorporated volatility effects through a scale factor (SF) added to the diseconomies of scale exponent (E)

• Similar approach used to model volatility effects in Ada COCOMO1

• Prior research points to the compounding or exponential effect of project factors with variable life cycle impact2

10

n

ii

SFE EMSIZEAPM1

)(

Source: 1: Boehm, B. and Royce, W. (1989); 2: Wang, G. et al., (2008)



Volatility Scale Factor

• Where– REVL = The % of the baseline requirements that is expected to change over

the system lifecycle

– wvl = aggregate lifecycle phase volatility weighting factor

• And:– wl = weighting factor for each life cycle phase1

– Θl = % of total requirements changes per life cycle phase

11

vlw wREVL

SF100

05.0

l

llv ww 1

Expected REVL is rated as Very Low, Low, Moderate, High and Very High

1Life Cycle Phases: Conceptualize, Development, Operational Test and Evaluation, and Transition to Operation



12

Data Collection

• Collected expert judgment on REVL and weighting factors through surveys and workshops– Software and Systems Engineers with 20+ years of

experience

– Variety of industries represented with an emphasis on Aerospace and Defense

• Historical Data Collection– Data were collected from 25 projects from the Aerospace

and Defense application domain

– Collected COSYSMO size, effort, and cost driver data

– Collected volatility data: added, modified, and deleted requirements over time



13

Model Calibration

Historical data Mean and Variance

Expert-judgment estimates

Bayesian Analysis

Source: Boehm et al. (2000); Nguyen (2010)

Initial parameter mean and variance

A Priori Model Data-determined ModelRegression Analysis; OLS

Formally combines expert judgmentWith sample data

A Posteriori Model

Optimized combinationof data sources to increase precision

Updated parameter mean

and variance



Requirements Volatility (REVL) Rating Levels

14

Characteristic Rating Very Low Low Moderate High Very High

System requirements baselined and agreed to by key stakeholders

Fully Mostly Generally Somewhat No

agreement

Level of uncertainty in key customer requirements, mission objectives, and stakeholder needs

Very Low Low Moderate High Very High

Number of co-dependent systems with influence on system requirements

Very Low Low Moderate High Very High

Strength of you organization’s requirements development process and level of change control rigor

Very High High Moderate Low Very Low

Precedentedness of the system , use of mature technology


Stability of the customer’s organization and business environment


Experience level of the systems engineering team in requirements analysis and development


Developed based on surveys of experienced S/W and Systems Engineers (N =38)



Requirements Volatility Profile

15

Very Low

Low

Moderate

High

Very High

Conceptualize Development Operational Test & Eval

Transition to Operation

% v

ola

tilit

y p

er li

fe c

ycle

ph

ase

mean + 1σ

mean + 2σ

mean

mean - 1σ

30%

25%

20%

15%

10%

5%

0%

% V

ola

tilit

y p

er L

ife

Cyc

le P

has

e

Very Low

Low

Moderate

High

Very High

Conceptualize Development Operational Test & Eval


% v

ola

tilit

y p

er li

fe c

ycle

ph

ase

mean + 1σ

mean + 2σ

mean

mean - 1σ

30%

25%

20%

15%

10%

5%

0%

% V

ola

tilit

y p

er L

ife

Cyc

le P

has

e

Based on expert judgment collected through three workshops (N = 36) and historical project data (N = 25)



Regression and Bayesian Calibration

• COSYSMO can be described as multiple regression model

• The model is linearized using a logarithmic transformation

• The data-determined coefficient β1 is combined with the a-priori expert judgment to develop the Bayesian calibrated model

1.96

2.01

2.09

Data Analysis A priori Expert Judgment

A posteriori Bayesian Update

)ln()ln()ln(06.1ln)ln( 1 EMSIZESFSIZEASEhrs



17

Model Performance Comparison

• Model performance evaluated using the baseline model and a model with local calibration

• The performance of COSYSMO improves by including the requirements volatility factor

Model Estimation Accuracy

PRED(.15) PRED(.20) PRED(.30) MMRE

Academic COSYMO 48% 52% 80% 21%

Req. Volatility Model 72% 80% 88% 16%

Calibrated Model*

Academic COSYMO* 52% 64% 84% 20%

Req. Volatility Model* 76% 84% 88% 15%



18

Coefficient of Determination (n=25)

* Due to proprietary reasons only the analysis of the model accuracy is shown, not the data itself

Academic COSYSMO COSYSMO with Requirements Volatility Factor

Estimated Systems Engineering Size (with diseconomies of scale) Estimated Systems Engineering Size (with diseconomies of scale)



Sensitivity Analysis

• Evaluated the sensitivity of the model’s results to the variability of key parameters

• Standard deviation in the volatility life cycle profile (Θl )

• Scenarios 1 and 2

• Standard deviation in the requirements volatility weighting factor (wl)

• Scenarios 3 and 4

• Combination of cases:

• Scenario 1-3, 1-4

• Scenario 2-3, 2-419

DevelopmentConceptualize Operational Test & Eval

Transition to Operations

DevelopmentConceptualize Operational Test & Eval

Transition to Operations

Scenario 2

Scenario 1

% V

olati

lity

per L

ife C

ycle

Pha

se%

Vol

atilit

y pe

r Life

Cyc

le P

hase

Source: Guidelines for the Economic Analysis of Projects (1997)



20

Cross-Validation Results• The 25 projects were randomly divided into K=6 subsets

• One of the subsets is excluded from the data set from which the model is built

• The resulting model is used to predict effort for the excluded cases

Calibrated ModelEstimation Accuracy

PRED(.15) PRED(.20) PRED(.30) MMRE

Academic COSYMO 52% 64% 84% 20%

Req. Volatility Model 76% 84% 88% 15%

Cross-Validation 76% 80% 88% 16%

Cross-Val Scenario 1 72% 80% 88% 16%




Cross-Val Scenario 1-3 72% 76% 88% 17%




Improvement holds across the scenarios



21

Conclusions

• Observations from the literature and workshop surveys were used to develop a mathematical framework for quantifying the impact of requirements volatility on systems engineering effort

• An evaluation of the model was performed by comparing its prediction accuracy against Academic COSYSMO

• The results indicate an improvement in effort prediction accuracy and MMRE

• Cross validation and sensitivity analysis were performed to demonstrate the accuracy of the model in predicting effort for new projects



Future Work

• Obtain additional project data from other organizations

– The external validity of this study will be limited by the engineering organizations that contribute data and the background of the industry experts that participate in the research

• Evaluate the effect of reuse on the results and its potential interaction with requirements volatility

• Further work is needed to complete the characterization of the model performance depending on the type of change: added, modified or deleted

22



23

References• B. Boehm and W. Royce. Ada COCOMO and the Ada Process Model, TRW Defense Systems Group, 1989

• B. Boehm, Software risk management: Principles and practices, IEEE Software 1 (1991), 32-41.

• B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E., Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, New York, NY, 2000

• Economics and Development Resource Center .1997. Guidelines for the Economic Analysis of Projects.

• S. Conte, H. Dunsmore, and V. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, 1986.

• General Accounting Office, "Stronger management practices are needed to improve DoD’s software-intensive weapon acquisitions (GAO-04-393)," 2004.

• ISO/IEC, "ISO/IEC 15288:2002 (e) systems engineering - system life cycle processes," 2002.

• G. Kotonya and I. Sommerville, Requirements engineering: Processes and techniques, John Wiley & Sons, New York, NY, 1998.

• MIL-STD-498, "Software development and documentation," 1994.

• D. Reifer, Requirements management: The search for nirvana, IEEE Software 17(3) (2000), 45-47

• D. Rhodes, R. Valerdi and G. Roedler, Systems engineering leading indicators for assessing program and technical effectiveness, Systems Engineering 12(1) (2009), 21-35.

• Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.

• Wang, G., Boehm, B., Valerdi, R., and Shernoff, A. (2008). “Proposed Modification to COSYSMO Estimating Relationship.” Technical Report. University of Southern California, Center for Systems and Software Engineering.


Center for Systems and Software Engineering USC CSSE 2012 Annual Research Review

Back-up

24



25

COSYSMO Cost Estimating Relationships (CERs)

SE_Hrs = Systems Engineering effort (hours)

A = calibration constant derived from historical project data

SIZE = measure of functional size of the system (number of

requirements, interfaces, algorithms, operational scenarios)

n = number of cost drivers (14)

EM = effort multiplier for the ith cost driver

The exponent (E) accounts for diseconomies of scale

Source: Valerdi (2005)

n

ii

E EMSIZEAHrsSE1

)(_



26

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

11.0

Expert JudgmentLife Cycle Weighting Factors (n = 27)

Data collected from two workshops: 25th Annual USC CSSE COCOMO and the 2011 USC-CSSE ARR

Sys

tem

s E

ng

ine

erin

g E

ffo

rt P

en

alty

Du

e t

o V

ola

tility

Conceptualize Development Operational Test & Evaluation




27

Evaluation of Model Performance

• The cost estimation accuracy of COSYSMO was compared to the accuracy of the model with the requirements volatility factor

• The model was evaluated using predictive accuracy levels, Mean Magnitude of Relative Error (MMRE) and the coefficient of determination (R2)

• The prediction accuracy level is defined as:

Wherek = number of projects in the set whose Magnitude of Relative Error is ≤ l

n

klPRED )(

Source: Conte et al., (1986)



28

Model Comparison Test (F-ratio)

• The F-test is typically used to compare regression models and determine whether the null hypothesis is supported or refuted

• In this case, the null hypothesis is that the simpler model (without a requirements volatility factor) is correct

• Values of the F ratio near one (1) support the null hypothesis, while larger values favor the alternative hypothesis

• The F-value was calculated to be 7.62 with > 95% confidence level: Supports the alternative hypothesis

AH

AHNHAHNH

MSE

dfdfRSSRSSF

)/()(

Documents

March 6, 2012 Mauricio E. Peña