BAE Systems Risk Opportunity & Uncertainty Modelling - Risk - Alan R Jones_1.pdf · Risk Opportunity & Uncertainty Modelling ... © Copyright BAE Systems. ... Use of Monte Carlo Simulation

10/09/2013 1

BAE Systems

Risk Opportunity & Uncertainty Modelling ACostE North West Region 4th September 2013

© BAE SYSTEMS PLC 2011 All Rights Reserved

The copyright in this document, which contains information of a proprietary nature, is vested in BAE SYSTEMS Public Limited Company. The contents

of this document may not be used for purposes other than that for which it has been supplied and may not be reproduced, either wholly or in part, in any

way whatsoever, nor may it be used by, or it contents divulged to, any person whatsoever without the prior written permission of BAE SYSTEMS Public

Limited Company

ACostE NW Region Sept 2013 10/09/2013 © Copyright BAE Systems. Any use, duplication or disclosure of information contained on

this page is subject to the restrictions on the title slide of this document. 2

Agenda

Part 1

Risk Opportunity and Uncertainty – Definitions

Brief Overview of Monte Carlo Analysis

Shortfalls in Monte Carlo

Part 2

Cost & Schedule Integration

An Overview of the “Slipping and Sliding” Technique

Summary

2



Risk Opportunity & Uncertainty - Definitions

Variability within a project cost or schedule can be summarised as either:

• Uncertainty

– Something that will happen but the exact values/parameters are not known

– E.g. Normal car journey from home to work

• Risk or Opportunity

– Things which may or may not happen

– Risk

• An event or a series of events which, on occurring, would damage a

project or business objective in terms of performance, functionality,

time of delivery, customer acceptance, or cost.

• E.g. An accident on the route ahead causes a hold-up on the journey

– Opportunity

• An event or series of events which, on occurring, would offer benefit to

the project or business in terms of performance, functionality, time of

delivery, customer acceptance, or cost

• E.g. An accident behind us reduces congestion on the journey ahead



ROU Bottom Up Evaluation

In order to do a Bottom-Up assessment of Risk Opportunity and Uncertainty

(ROU), it is generally necessary to:

– Define a work package in an appropriate level of detail (tasks)

– Assign a range of likely cost and/or schedule outcomes for each task

– Link tasks that have an underlying relationship in terms of outcome in terms

of cost (or time)

e.g. Design overrun leads to increasing cost of Manufacture or Construction

i.e. Partial Correlation – often overlooked in Bottom-up ROU because

it is not properly understood

– Add the tasks together

– Review the range of outcomes

(Failure to correlate tasks appropriately will result in too narrow an output

range)

– Make recommendations based on the range of outcomes possible

This approach generally requires the use of a Monte Carlo Simulation toolset



Monte Carlo Simulation – What is it?

A Structured Approach to a Chance Encounter!



Monte Carlo Simulation

What is it? Why do we use it?

• A method of aggregating multiple distributions from independent variables in

a manner which maintains statistical correctness

• Usually not practical to combine distributions manually:

Total Probability =1

Opt = 1 Pess = 6


Opt = 1 Pess = 6

Opt = 2 Pess = 12





Why is that wrong?

• Consider throwing two conventional dice. Each has a uniform distribution.

Add together the “scores”

Die 1 2 3 4 5 6

1 2 3 4 5 6 7

2 3 4 5 6 7 8

3 4 5 6 7 8 9

4 5 6 7 8 9 10

5 6 7 8 9 10 11

6 7 8 9 10 11 12

0

1

2

3

4

5

6

7

1 2 3 4 5 6 7 8 9 10 11 12Score

Oc

cu

rre

nc

es

What is it doing?

• Looking at every possible

combination

• Not feasible in complex multi-

variable environments




Combining distributions mathematically is theoretical possible but extremely

complex and thus impractical in real terms

• For every conceivable permutation we want the product of those

distributions not the sum of them

• Monte Carlo Simulation provides an approximation shortcut to that product

S f(x) S P f(x)




Monte Carlo Simulation is a statistically valid way of adding together a number

of distributions

– Possible outcomes are defined by the user selecting an appropriate

distribution and probability of occurrence for each task

– Monte Carlo Simulation picks a value randomly from within the range of

possible outcomes defined

– The randomly generated outcomes from all the tasks can be combined

together usually by simply adding them together to create a single valid

potential outcome of the overall total (sometimes called a slice)

– This process is repeated many times to generate a distribution of

possible outcomes weighted according to the distributions chosen for

each task

The output distribution generated allows the user to interpret data using

confidence levels to define three-point estimates



Cost

Occurrences

Cost

Probability


Slice 1

Slice 3

Slice 2

-

-

-

• 10,000 Slice view

10,000 Slices

20 Slice view



ROU Bottom Up Evaluation

Baseline Task

Baseline Uncertainty & Risk

Potential Outcomes Cost or Time 0 Lik

elih

ood o

f O

ccurr

ence

Risk Register Based

Potential Outcomes 0 Cost or Time

Baseline Task + Uncertainty

Lik

elih

ood o

f O

ccurr

ence

Baseline Uncertainty

Potential Outcomes

Baseline Task

0 Cost or Time

Lik

elih

ood o

f O

ccurr

ence

S Optimistic S Pessimistic

Uncertainty Only

Model

Risk Opportunity

and Uncertainty

Combined Model

Risk and Opportunity

Only Model


this page is subject to the restrictions on the title slide of this document.


Things to do to give Monte Carlo Simulation a chance of being right

• Try to ensure that the distributions you choose are appropriate

– At least make sure that the basic distribution shape and range are right

– The majority of inputs to Schedule and Cost Monte Carlo Models are likely to be

positively skewed

• Understand the difference between the three principle Measures of Central

Tendency:

– Mean (Average), Mode (Most Likely)and Median (50% Confidence)

• Do you mean “Minimum” and “Maximum” or “Optimistic” and “Pessimistic”?

• Ask the question about the Mode (Most Likely): “In what circumstances…

– ... can the value be less than the Most Likely?”

– ... can the vale be more than the Most Likely?”

• Apply Correlation to tasks

– Very few tasks are totally independent of all others

– Consider a background correlation of between 20-30% (potentially even more for

Concept Development)

– Very few tasks are negatively correlated with others

12



Monte Carlo Simulation: Effect of Correlation

13

The impact of negative correlation is

to push in and up

High values of one variable are associated with low

values of another variable and vice versa

Lik

eli

ho

od

of

Oc

cu

rre

nce

Range of Potential Outcomes

Take any two variables in a Monte Carlo Simulation:

Lik

eli

ho

od

of

Oc

cu

rre

nce

Range of Potential Outcomes

The impact of positive correlation is to

push down and out

High values of one variable are associated with high

values of another variable and vice versa



Monte Carlo Simulation: The Downside

14

May or May Not Occur Will Occur

Risk &

Opportunity Register Defined

Baseline Tasks

Emergent

Baseline Task

Performance

Undefined Risks

or the

“Unknown Unknowns”

Clearly Defined

Undefined

or Unclear

Baseline

Estimate

Bottom-up

Uncertainty

Assessment

Bottom-up

Risk &

Opportunity

Assessment

Gap in Monte

Carlo Analysis

Likelihood of Task Occurrence

Task

Definition

“There are known knowns.

These are things we know

that we know.

There are known unknowns.

That is to say, there are

things that we now know we

don’t know.

But there are also unknown

unknowns. These are things

we do not know we don’t

know.”

Donald Rumsfeld

United States Secretary of Defense

“To know that we know what

we know, and that we do not

know what we do not know,

that is true knowledge.”

Confucius



Cost & Schedule Integration – Things that Dreams are made of

At the risk of being controversial, perhaps we should start by setting the cat

amongst the pigeons ….

15

Note: No real pigeons have been harmed in the making of this presentation



Cost & Schedule Integration – Stuff that Dreams are made of …

Schedule Analysis

Co

st

An

aly

sis

Let’s be honest …

if it was easy we’d all be doing it, but

we’re not

… conclusion: it isn’t easy

Do we even understand it?

We will all have heard the old saying

“Time is Money”, and there is some

truth in that

So we should be looking at

Integrated Cost and Schedule

Analysis … why aren’t we?

It needs investment:

Can we afford to?

Can we afford not to?

Integrated Cost &

Schedule Analysis

Schedule elements may require correlation

Cost elements may require correlation

Both may require cross-correlation



So, how can we fix it?

In line with good estimating practice, we advocate using more than one approach,

method and/or technique to evaluate the potential range of cost outcomes

17

• Top-down approach to pick up the unknowns and avoid duplication

Takes a view of the schedule risk and pro-rata the rate of spend

affected (Time costs money, resources are unlikely to be re-deployed)

Commercial/Financial uplift factors (escalation etc)

• Bottom-up approach based on authorised Risk Registers

Use of Monte Carlo Simulation - statistical technique to allow

multiple variables and probabilities to be modelled

• Balanced view of Top-down and Bottom-up approaches

By its very nature the Top-down approach (‘worst-case’ view) should

be greater than Bottom-up approach If not, or if there is a significant

difference either way, it could indicate that either one or other

approach is too immature, or overly optimistic or pessimistic



Example of a Top-down Approach

18

What is it?

Task

Task

Task

Task

Task

Task

Task

Task

Task Task Task

Baseline Task Top-Down Variability

Cost

Schedule

Schedule Risk

Resources/Cost can be scaled in direct

proportional to the increase in programme

duration

• This implies a “standing army” effect if risks

materialise

Approach can be justified through a number of

general principles:

• Resources utilised by a proposal are scoped

on the baseline programme.

• Resources are generally internally re-deployed

within a contract

• Risks generally manifest themselves as

programme slippage

Uplift Factors

These cover issues not sufficiently covered by

the schedule risk assessment

• Examples include Escalation



Using Monte Carlo to make a Price Recommendation

19

2.8

2.9 3

3.1

3.2

3.3

3.4

3.5

3.6

3.7

3.8

3.9

0%

20%

40%

60%

80%

100%

Min: 2.82119 Max: 3.86365Probability: 100%

2.8

2.9

3

3.1

3.2

3.3

3.4

3.5

3.6

3.7

3.8

3.9

Min: 2.82119 Max: 3.86365 Probability: 100%

Sensible Level of

Confidence for Bid

Our bottom-up approach to estimating

looks at modelling risk, opportunity and

uncertainty around the Most Likely

values

We always look to price based on

values which have a higher level of

Confidence …

Audience Quiz:

Why do we do that?

We don’t trust our own estimates?

We are so totally risk-averse?

We know that our customers will

knock us down?

The schedule may slip?

The sum of the Most Likely Values

is not the most likely value?

Monte Carlo Output is misleading?



Making A Recommendation - Interim

For internal clearance, generally prior to price formulation and submission, we

provide a high level summary of the cost build up:

… maybe we could do better…

Whatever happened to that Top-down Assessment we made?

Confidence Level £ Million Confidence

Optimistic Range 3.105 10%

Pessimistic Range 3.425 90%

Initial Recommendation 3.270 60%

Cost Build Up £ Million % of Total

Baseline Budget Request 3.050 93.3%

Technical Risk Contingency 0.138 4.2%

Management Contingency 0.082 2.5%

Total 3.270 100%

Risk Modelling

50% Confidence

level

Balancing Number

for Total

Bottom-up

Functional Budget

Request



Taking a Balanced View

Risk & Opportunity Register Based

S Optimistic Opportunities

and Uncertainties S Pessimistic Risks

and Uncertainties

Bottom-Up

Confidence Level

Baseline Uncertainty

Baseline Task

Lik

elih

ood o

f O

ccurr

ence

Potential Outcomes 0 Cost

Baseline Task Top-Down Variability

Cost

Bottom-Up Evaluation

Top-Down Evaluation

?

Schedule



Slipping & Sliding - A Pragmatic Aid to “Judgement”

What is it?

• Slipping & Sliding is a simple technique we developed “on the fly” in support of

an Estimating Practitioner Training Course on Risk Opportunity & Uncertainty

• Trainees wanted more guidance on making that “Judgement Call”

• And thus, a training exercise was born that we called “Slipping & Sliding” for

reasons that will become abundantly clear

The rational is:

• We assume that we have made an honest and reasonable assessment of the

Top-down approach to cost variability based on a conservative view of the

schedule risk (higher Confidence Level)

– We know that the approach is inherently pessimistic

• We assume that we have made an honest and reasonable assessment of the

Bottom-up approach to Risk, Opportunity and Uncertainty using Monte Carlo

– We know that it does not include any explicit provision for the Rumsfeld Factor,


– The approach is inherently optimistic

• Reality lies somewhere between the optimistic and pessimistic view of life

22




23

2500 2700 2900 3100 3300 3500 3700 3900 4100

100

90

80

70

60

50

40

30

20

10

0

Bottom-up

Uncertainty Bottom-up Risk, Opportunity & Uncertainty

(for statistical validity)

Top-down Assessment (80%SRA)

Functional Requests

Baseline Programme

0 200 400

Bottom-up

Risk &

Opportunity

First Pass: 70% Confidence on Uncertainty

50% Confidence on Risk & Opportunity

So, in this particular case:

Uncertainty @ 70% Confidence Level

+ Risk/Opportunity @ 50% Confidence Level

Risk Opportunity & Uncertainty @ 60%

Confidence level

… which all fits nicely inside the Top-down

assessment, which is inherently pessimistic




24

2500 2700 2900 3100 3300 3500 3700 3900 4100

100

90

80

70

60

50

40

30

20

10

0

Bottom-up

Uncertainty


Functional Requests

Baseline Programme

0 200 400

Bottom-up

Risk &

Opportunity

Bottom-up Risk, Opportunity & Uncertainty


So, in this particular case:

Our Bottom-up Risk Opportunity &

Uncertainty @ 60% Confidence level

… May only be equivalent to around the 40%

Confidence Level of our Top-down Approach




25

2500 2700 2900 3100 3300 3500 3700 3900 4100

100

90

80

70

60

50

40

30

20

10

0

Bottom-up

Uncertainty


Functional Requests

Baseline Programme

0 200 400

Bottom-up

Risk &

Opportunity



… whereas our Top-down Approach @ 50%

Confidence level

… may be equivalent to around the 72%

Confidence Level on our Bottom-up

Approach




26

2500 2700 2900 3100 3300 3500 3700 3900 4100

100

90

80

70

60

50

40

30

20

10

0

Bottom-up

Uncertainty


Functional Requests

Baseline Programme

0 200 400

Bottom-up

Risk &

Opportunity



Adjustment for inherent

Optimism Bias

in the bottom-up approach } Ratio of Top-down to Bottom-up Approaches:

Greater than but close to one

Conclusion?

The two approaches are reasonably

consistent with one another

Case 1




27

2500 2700 2900 3100 3300 3500 3700 3900 4100

100

90

80

70

60

50

40

30

20

10

0

Bottom-up

Uncertainty


Functional Requests

Baseline Programme

0 200 400

Bottom-up

Risk &

Opportunity



Ratio of Top-down to Bottom-up

Approaches:

Significantly greater than one

Conclusion?

The two approaches are not consistent

with one another, either:

• the top-down is overly pessimistic

• or, the bottom-up is too immature

Case 2




28

2500 2700 2900 3100 3300 3500 3700 3900 4100

100

90

80

70

60

50

40

30

20

10

0

Bottom-up

Uncertainty


Functional Requests

Baseline Programme

0 200 400

Bottom-up

Risk &

Opportunity



Ratio of Top-down to Bottom-up

Approaches:

Less than one

Case 3

Conclusion?

The two approaches are not consistent with

one another – an optimistic value cannot be

greater than a pessimistic value. Either:

• the top-down is overly optimistic

• or, the bottom-up is too immature



Making A Recommendation - Revisited

Following consideration of the inherent Optimism Bias in the Bottom-up Approach,

we can modify our high level summary of the cost build up:

… the inherent adjustment is held within Management Reserve

or a higher Technical Risk Contingency

Confidence Level £ Million Confidence

Optimistic Range 3.105 10%

Pessimistic Range 3.425 90%

Revised Recommendation 3.305 72%

Cost Build Up £ Million % of Total

Baseline Budget Request 3.050 92.3%

Technical Risk Contingency 0.138 4.2%

Management Contingency 0.117 3.5%

Total 3.305 100%

Risk Modelling

50% Confidence

level

Balancing Number

for Total

Bottom-up

Functional

Request




Does it work in Practice?

• We do not claim it to be a perfect substitute for holistic Cost and Schedule Integration

• The technique is not a perfect solution – but none are

• We’re looking for reasonable accuracy not unreasonable precision

• It does not replace estimating judgement … it can guide the thought process in making a judgement

– To narrow the range or eliminate the extremes

– To reject or rework a particular approach

• It provides a degree of Quality Control in our approach to generating three-point estimates

• Proof of the Pudding as they say is in the eating …

• Let’s look at an example

30



Example – Comparison Based on “Real” Data


31

£0

£5,000

£10,000

£15,000

£20,000

£25,000

£30,000

- 10.00 20.00 30.00 40.00 50.00 60.00 70.00 80.00 90.00

Co

st

Inclu

din

g R

isk (

£K

)

Duration (Months)





£24,000

£25,000

£26,000

£27,000

£28,000

£29,000

£30,000

£31,000

60.00 65.00 70.00 75.00 80.00 85.00 90.00 95.00

Co

st

Inclu

din

g R

isk (

£K

)

Duration (Months)





33

£24,000

£25,000

£26,000

£27,000

£28,000

£29,000

£30,000

£31,000

60.00 65.00 70.00 75.00 80.00 85.00 90.00 95.00

Co

st

Inclu

din

g R

isk (

£K

)

Duration (Months)



Slipping & Sliding - A Pragmatic “Judgement” Aide

Confidence Level of the Final Agreement – a Reality Check

• Confidence levels and intervals are calculated as outputs from the Monte

Carlo modelling exercise.

• However, it should be remembered that the Confidence Level generated

through Monte Carlo Analysis is overstated

– Reality is that the true value at any Confidence Level above the Mode will be

greater than calculated, because …

– The value returned by Monte Carlo is only based on the analysis of the inputs

– It excludes any consideration of things that have not been considered –


– It assumes the baseline programme

– Failure to correlate tasks will narrow the output range too much

• The Top-down Method is inherently pessimistic in nature

– It assumes that if the one task slips, everything slips - there is no recovery

– However, it does create “headroom” for those “unknown unknowns”

• Reality is likely to be somewhere between the two views

34

Optimism Bias

Pessimism Bias



Slipping & Sliding - A Pragmatic “Judgement” Aide

35

Thank you for listening

Any questions?

Documents

BAE Systems Risk Opportunity & Uncertainty Modelling - Risk - Alan R Jones_1.pdf · Risk Opportunity & Uncertainty Modelling ... © Copyright BAE Systems. ... Use of Monte Carlo Simulation