A BBN as it Would be Developed and Used in the QUELCE Method

2013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University

A BBN as it Would be Developed and Used in the QUELCE MethodSoftware Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

Robert W. StoddardJim McCurley

24 October 2013


Introduction

QUELCE (Quantifying Uncertainty for Early Lifecycle Cost Estimation) is a multi-year research project led by the Software Engineering Measurement and Analysis (SEMA) team within the SEI Software Solutions Division.

Research team membership comprises SEI technical staff with cost estimation background in collaboration with several external faculty (Dr. Ricardo Valerdi, Univ of Arizona, & Dr. Eduardo Miranda, CMU).

This research is motivated by (1) the WSARA Act requiring cost estimates pre-Milestone A and (2) DoD’s need for more accurate cost estimation methods that provide continuous monitoring of changing assumptions and constraints.


Modeling Uncertainty

Complexity Reduction

1. Use QUELCE

Repository to Populate Driver State

Matrix

3. Develop BBN Model and

Assign Conditional

Probabilities to BBN Model

4. Calculate Cost Factor

Distributions for Program Execution

Scenarios

5. Monte Carlo Simulation to Compute Cost

Distribution

2. Evaluate Cause and Effect

Relationships and Reduce Explosion via Dependency Structure Matrix

Overview of QUELCE

Legend:

QUELCE Change Repository

Queries of Historical MDAP Experience and

Context

Change Driver Nominal State Alternative States

Scope Definition

Stable Users added Additional (foreign) customer

Additional deliverable (e.g. training & manuals)

Production downsized

Scope Reduction (funding reduction)

Mission / CONOPS defined New condition New mission New echelon Program

becomes Joint

Capability Definition

Stable Addition Subtraction Variance Trade-offs [performance vs affordaility, etc.]

Funding Schedule

Established Funding delays tie up resources {e.g. operational test}

FFRDC ceiling issue

Funding change for end of year

Funding spread out

Obligated vs. allocated funds shifted

Advocacy Change

Stable Joint service program loses particpant

Senator did not get re-elected

Change in senior pentagon staff

Advocate requires change in mission scope

Service owner different than CONOPS users

Closing Technical Gaps (CBA)

Selected Trade studies are sufficient

Technology does not achieve satisfactory performance

Technology is too expensive

Selected solution cannot achieve desired outcome

Technology not performing as expected

New technology not testing well

● ~~~~ ~~~~ ~~~~ ● ~~~~ ~~~~ ~~~~ ~~~~ ● ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~

1. Driver State Matrix

Change Drivers - Cause & Effects Matrix

Mis

sion

/ C

ON

OPS

Cha

nge

in S

trat

egic

Vis

ion

Cap

abilit

y D

efin

ition

Adv

ocac

y C

hang

e

Clo

sing

Tec

hnic

al G

aps

(CBA

)

Build

ing

Tech

nica

l Cap

abilit

y &

Cap

acity

(CB

A)

Inte

rope

rabi

lity

Sys

tem

s D

esig

n

Inte

rdep

ende

ncy

Func

tiona

l Mea

sure

s

Sco

pe D

efin

ition

Func

tiona

l Sol

utio

n C

riter

ia (m

easu

re)

Fund

ing

Sche

dule

Acq

uisi

tion

Man

agem

ent

Prog

ram

Mgt

- C

ontr

acto

r Rel

atio

ns

Proj

ect S

ocia

l / D

ev E

nv

Pro

g M

gt S

truc

ture

Man

ning

at p

rogr

am o

ffice

Mission / CONOPS 3 3 0 6 0Change in Strategic Vision 3 3 3 2 2 2 2 2 3 2 3 2 29 0Capability Definition 3 0 2 1 1 0 0 2 2 2 0 1 0 2 0 0 16 0Advocacy Change 2 1 1 1 1 6 0Closing Technical Gaps (CBA) 2 1 3 1 2 2 1 2 2 1 1 2 1 0 2 2 1 1 2 2 1 2 34 0Building Technical Capability & Capacity (CBA) 1 1 2 1 2 2 1 2 3 2 2 1 2 2 1 1 1 27 0Interoperability 1 2 1 1 1 1 1 1 1 1 2 1 1 2 1 1 3 1 2 2 2 29 1Systems Design 1 2 2 2 2 1 1 1 1 1 2 2 3 21 3Interdependency 1 2 2 1 1 1 1 1 1 1 1 1 2 1 2 2 1 1 1 1 1 2 2 3 33 5Functional Measures 2 2 2 1 1 1 1 1 1 1 2 1 16 0Scope Definition 1 1 3 5 0Functional Solution Criteria (measure) 1 2 2 1 1 2 1 10 1Funding Schedule 1 1 2 1 5 0Acquisition Management 1 1 2 3 1 1 2 2 1 1 1 2 1 19 2Program Mgt - Contractor Relations 1 1 2 1 1 1 1 2 2 12 2Project Social / Dev Env 1 1 1 2 2 1 1 2 1 1 1 14 2Prog Mgt Structure 1 2 1 2 6 1Manning at program office 2 1 2 5 2Scope Responsibility 1 1 1 1 1 1 6 5Standards/Certifications 1 1 1 1 1 1 3 1 10 2Supply Chain Vulnerabilities 1 1 1 1 2 1 7 4Information sharing 1 1 1 1 1 1 1 7 3PO Process Performance 2 2 4 0Sustainment Issues 0 0Contract Award 0 0Production Quantity 2 2 0Data Ownership 2 2 0Industry Company Assessment 0 0Cost Estimate 0 0Test & Evaluation 0 0Contractor Performance 2 2 0Size 0 0Project Challenge 0 0Product Challenge 0 0Totals 0 0 6 4 1 9 5 12 8 7 7 13 4 10 15 18 7 7 8 8 14 17 17 15 12 9 10 13 11 20 19 5 5 17 0Below diagonal 0 0 0 1 1 4 4 4 1 2 0 3 1 3 2 2 3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Effects

Causes

2. Dependency Structure Matrix

3. BBN Model

4. Cost Factor Distributions by Scenario of Change

5. Monte Carlo with Cost

Estimation Tools

Drivers XL VL L N H VH XH Product ProjectScale Factors

PREC 6.20 4.96 3.72 2.48 1.24 0.00 <X>FLEX 5.07 4.05 3.04 2.03 1.01 0.00 <X>RESL 7.07 5.65 4.24 2.83 1.41 0.00 <X>TEAM 5.48 4.38 3.29 2.19 1.10 0.00 <X>PMAT 7.80 6.24 4.68 3.12 1.56 0.00 <X>

Effort MultipliersRCPX 0.49 0.60 0.83 1.00 1.33 1.91 2.72 XRUSE 0.95 1.00 1.07 1.15 1.24 XPDIF 0.87 1.00 1.29 1.81 2.61 XPERS 2.12 1.62 1.26 1.00 0.83 0.63 0.50 <X>PREX 1.59 1.33 1.12 1.00 0.87 0.74 0.62 <X>FCIL 1.43 1.30 1.10 1.00 0.87 0.73 0.62 <X>SCED 1.43 1.14 1.00 1.00 1.00 <X>

SRDR SARs


What is a Bayesian Belief Network (BBN)?

A probabilistic model shown in a graphical form consisting of nodes and arrows

Nodes represent factors

Arrows between factors represent cause-effect relationships (ideally), or correlated relationships (minimally)

Factors may be set at “observed” levels representing observed “evidence”

BBNs use traditional conditional probability and Bayesian calculations to update all “unknown” factors based on the latest “evidence”


Burglar Alarm Example of a BBN - 1

Reproduced from Jensen 1996, as seen at http://www.agenarisk.com/resources/example_models.shtml

The following example of a BBN was derived from an example model shown on the AgenaRisk tool vendor website.

This example helps to concisely articulate the operation and use of a BBN.



(Re-produced from Jensen 1996, as seen at http://www.agenarisk.com/resources/example_models.shtml)

The probabilities of this baseline model reflect both historical data and expert belief.




“Mr. Holmes is working at his office when he receives a telephone call from Watson who tells him that Holmes’ burglar alarm has gone off.”

Convinced that a burglar has broken into his house (alarm sounds -> burglary), Holmes rushes into his car and heads for home.”




“On his way he listens to the radio, and in the news it is reported that there has been a small earthquake in the area (radio report -> earthquake). Knowing that the

earthquake has a tendency to turn the burglar alarm on (earthquake -> alarm sounds), he returns to his work leaving his neighbors the pleasures of the noise.”


Example QUELCE Bayesian Belief Network


Benefits of Using a BBN Model for QUELCE – 1

Models the uncertainty of program change drivers and their relationships using probability distributions.•No longer use single point estimates• Instead, we use ranges and distributions reflecting uncertainty

Provides continuous measurement and ability to update and re-estimate based on changes in program execution.•Once created, simple to run scenarios as new “evidence” is observed•Readily used to update cost estimates based on changing program conditions

Translates the net effect of program change driver uncertainty to the input factors of cost estimation models.•We use Monte Carlo simulation to translate BBN output distributions into

distributions for input parameters of CERs



Enables the use of both objective (hard) information and subjective (soft) information as evidence to update our forecast.•Evidence can be factual observation, e.g. something has happened •Evidence can be subjective in terms of a person’s anticipation of an event

occurring

Enhances ability to conduct “what-if” analysis in context of change drivers.•A scenario in the BBN is a collection of one or more change drivers observed

or anticipated to occur with a specified probability•For each scenario, the BBN then recalculates and produces new outcome

node distributions



Enables analysis and forecast with incomplete information (e.g., no status available for some change drivers).•Traditional statistical analysis requires entries for all modeled factors, e.g. in a

regression equation•BBNs can provide updated assessments of all unknown factors based on

whatever factors are observed

Provides the ability to determine which change drivers are most influential on downstream change drivers or BBN outcome nodes (e.g., project complexity, product complexity).•Any factor may be selected for evaluating sensitivity to any and all other factors•For any factor, a sensitivity “tornado” chart may be created depicting in

descending order all other factors influencing this factor



May explain the most likely state of affairs of upstream change drivers based on current observations of downstream change drivers.•Akin to diagnosing likely causes of today’s observations•Commonly used in medical diagnosis•May be used to diagnose what other change drivers led to the current state of

affairs of known change driver occurrence


Future Work

Need to standardize the output nodes of the QUELCE BBN

Need to provide a method to connect the BBN output nodes to the inputs of commonly used CERs

Need to collect data from retrospectives and ongoing program executions to validate the performance of the QUELCE BBN and method


Summary

The DSM matrix captures the experts’ change probabilities for one change driver affecting another change driver (all possible pairings).

The BBN models the probabilistic relationships so that different change scenarios including cascading change may be evaluated.

For each scenario, the BBN produces probability distributions for the output nodes which will then be used to assign probability distributions to the input factors of the cost estimating relationships/tools.


Contact Information

Robert W. StoddardPrincipal ResearcherSoftware Solutions Division, SEAPTelephone: +1 412-268-1121Email: [email protected]

U.S. MailSoftware Engineering InstituteCustomer Relations4500 Fifth AvenuePittsburgh, PA 15213-2612USA

Webwww.sei.cmu.eduwww.sei.cmu.edu/contact.cfm

Customer RelationsEmail: [email protected]: +1 412-268-5800SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257

Documents

A BBN as it Would be Developed and Used in the QUELCE Method