Risk And Relevance 20080414ppt

®

IBM Software Group

© 2008 IBM Corporation

Risk and Relevance

Dr. Murray CantorDistinguished Engineer, IBM Rational [email protected]

IBM Software Group | Rational software

22

Value is being created by software/IT

Transportation Equipment

Hospitals

Chemical Manufacturing

Software/Information

Food Manufacturing

Telecommunications

Computers/Electronics

$0.00 $100.00 $200.00 $300.00 $400.00 $500.00 $600.00 $700.00 $800.00

US Industry Revenue in $B

US Census Bureau


33

Software provides huge value for all the other industries


US Census Bureau


Hospitals



Food Manufacturing

Telecommunications


$0.00 $100.00 $200.00 $300.00 $400.00 $500.00 $600.00 $700.00 $800.00


44


US Census Bureau


Hospitals



Food Manufacturing

Telecommunications


$0.00 $100.00 $200.00 $300.00 $400.00 $500.00 $600.00 $700.00 $800.00

Software provides huge value for all the other industries

This does not include value created by

back office/operations support


55

The concern is we are not creating value efficiently

For example, in a European Services Strategy Unit 2007 report studied105 outsourced public sector ICT projects with significant cost overruns, delays and terminations, total value of contracts is £29.5 billion

Results Cost overruns totaled £9.0 billion

57% of contracts experienced cost overruns

The average percentage cost overrun is 30.5%

33% of contracts suffered major delays

30% of contracts were terminated

12.5% of Strategic Service Delivery Partnerships have failed

How much value was deliveredby all the successful efforts?


66

How should we meet the challenge of efficiently delivering value?

Focusing on avoiding overruns? Leads to risk avoidance

limits opportunity for creating value, to be relevant

Rather, lets discuss how to apply engineering discipline to creating value


77

History has shown we do not get the estimates ‘right’Boehm COCOMO Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

100 1000 10000 100000

Size (SLOC)

Eff

ort

(S

taff

-mo

nth

s)

Jones Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

1.00 10.00 100.00 1000.00 10000.00

Size (FP)

Eff

ort (

LM)

Basili & Bailey Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

100 1000 10000 100000 1000000

Size (SLOC)

Eff

ort

(L

M)

ISBSG Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

1.00 10.00 100.00 1000.00 10000.00

Size (FP)

Effo

rt (L

M)

Rone Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

1.00 10.00 100.00 1000.00 10000.00

Size (SLOC)

Eff

ort

(S

M)

From George Stark, Paul Oman, “A comparison of parametric Software Estimation Models using real project data”, in press


88

History has shown we do not get the estimates ‘right’Boehm COCOMO Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

100 1000 10000 100000

Size (SLOC)

Eff

ort

(S

taff

-mo

nth

s)

Jones Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

1.00 10.00 100.00 1000.00 10000.00

Size (FP)

Eff

ort (

LM)

Basili & Bailey Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

100 1000 10000 100000 1000000

Size (SLOC)

Eff

ort

(L

M)

ISBSG Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

1.00 10.00 100.00 1000.00 10000.00

Size (FP)

Effo

rt (L

M)

Rone Effort Estimation Accuracy

1.00

10.00

100.00

1000.00

1.00 10.00 100.00 1000.00 10000.00

Size (SLOC)

Eff

ort

(S

M)

From George Stark, Paul Oman, “A comparison of parametric Software Estimation Models using real project data”, in press, 2007

Variance dominates

Maybe there is a reason


99

“Make everything as simple as possible, but not simpler.”

- Albert Einstein

“We should be guided by theory, not by numbers.”

- W. Edward Deming

Eternal wisdom

“To measure is to know. If you can not measure it, you can not improve it.”

- Lord Kelvin

“There are risks and costs to a program of action. But they are far less than the long-range risks and costs of comfortable inaction.”

- John F. Kennedy


1010

The value conversation can vary

Future business efficiency Investing now to receive future savings, capacity, responsiveness

Operational risk avoidance Investing now to avoid future business/IT risks, e.g., security, privacy, continuity …

Business impact Investing now to affect future top line

Treacy Framework

Mission capabilities

ProductInnovation

OperationalEfficiency

ClientIntimacy

Your Business?


1111

The value conversation can vary

ProductInnovation

OperationalEfficiency

ClientIntimacy

Your Business?

Future business efficiency Investing now to receive future savings, capacity, responsiveness

Operational risk avoidance Investing now to avoid future business/IT risks, e.g., security, privacy, continuity …

Business impact Investing now to affect future top line

Treacy Framework

Mission capabilities

All can be monetized


1212

Just enough theory

Money is how one measures value – need to monetize

The future cost, value are uncertain

Cost and value are random variables

Need to consider current Net Present Value of program

Results need only to be accurate enough to drive desired behavior: Assuming managed risk to deliver value


1313

How much is an incomplete development program worth?

Conventional wisdom is the accountants answer. provides no opportunity for ongoing value management

Can only discuss cost, not value

0 6 Ship date

Conventional WisdomV

alu

e

Little opportunity to show relevance


1414

Imagine (if you will) you could sell your program

The buyer would spend money now to get the right to invest in completing the program – like a call option

How would one reason about the fair price? The buyer, reasoning like an investor, would like to know

Expected cost to complete

Expected value received

The riskThis is what the economists call“incomplete market reasoning”


1515

The Net Present Value of a development program

RI = revenue, benefits stream

Ej = development expense stream

Mk = maintenance expenses stream

The rR, rM, rE are discount rates accounting for the time value of money

NPV, Ri, Ej, Mk are all random variables

0111 )1()1()1(

Er

E

r

M

r

RNPV

m

jj

E

jp

kk

M

kn

ii

R

i


1616

The Net Present Value of a development program

RI = revenue, benefits stream

Ej = development expense stream

Mk = maintenance expenses stream

0111 )1()1()1(

Er

E

r

M

r

RNPV

m

jj

E

jp

kk

m

kn

ii

R

i

NPV, Ri, Ej, Mk are all random variables

So, how can value be created?


1717

Expense risk: Imagine you have 12 months to deliver a business critical system

Your estimators tell you it will be done in 11 months

What do you do with the information? Rest easy, believing there is no risk?

0 6 12


1818

Maybe you realize that program parameters (cost, schedule, effort, quality, …) are random variables Area under curve describes probability of measurement

falling in range

0.80

1.20

1.60

2.00

2.40

2.80

3.20

3.60

4.00

4.40

4.80

5.20

5.60

6.00

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0

0.00

0.40

Likelihood of actual value falling in range

is area under curve


1919

Imagine you have 12 months to deliver a business critical systems

So you ask for the distribution and discover there is some uncertainty

0 6 12


2020

Imagine you have 12 months to deliver a business critical systems

In fact there is less than 50% chance of making the date

48%

0 6 12


2121

0 6 12

Then what?

Move out the date to improve likelihood of shipping?

15

95%


2222

0 6 12

Then what?

Or move in the estimate by sacrificing quality or content?

95%


2323

The estimate variance reflects current state of understanding Source

Lack of knowledge

Lack of confidence

Reduction of variance reflects Increased knowledge about

Client needs

Technology

Team capability

Good decisions 0.80

1.20

1.60

2.00

2.40

2.80

3.20

3.60

4.00

4.40

4.80

5.20

5.60

6.00

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0

0.00

0.40


2424

Then what?

Determine the source of the variance

Over the project lifecycle, reduce the variance to improve likelihood of shipping

90%

0 6 12


2525

0 6 12

Then what?

Over the lifecycle, reduce the variance further to improve likelihood of shipping

95%


2626

To summarize so far

Cost, schedule are random variables

Variance is the measure of risk

Progress measured by reduction of risk

0111 )1()1()1(

Er

E

r

M

r

RNPV

m

jj

E

jp

kk

m

kn

ii

R

i

Medium Variance

Time

Var

ian

ce i

n C

ost

, S

ched

ule

…

Project Lifecycle Phases

Inception Elaboration Construction Transition/Maintenance

High Variance Low Variance

80% reduction96% reduction


2727

Benefits: Delivered value is also a distribution

0111 )1()1()1(

Er

E

r

M

r

RNPV

m

jj

E

jp

kk

m

kn

ii

R

i

Shortfall in stakeholder needs

Value is improved by increasing upside variance0 Planned $Value

Reuse opportunities


2828

Simple inputs

Expert opinion Just guess using expert opinion

Use optimistic, pessimistic, nominal assumptions in other models

Function point, Use case point, Cocomo

Enter into triangular distributions

Historical data Use other distributions if you have them


2929

Model output

In this case Expected NPV = $4,324K

No chance > $14,924K, < ($4,837k)

Mean 4,324.45

Standard Deviation 2,738.59

Variance 7,499,853.36

0% -4,836.65

10% 818.52

20% 1,959.10

30% 2,804.66

40% 3,541.42

50% 4,249.81

60% 4,972.52

70% 5,739.09

80% 6,661.06

90% 7,928.97

100% 14,923.77


3030

No initial risk – little opportunity to add value, stay relevant

NPV increases when you invest in Improving likelihood of delivery (reduce variance of costs)

Improving range of value at delivery by reuse of architecture, services … (increase upside variance of benefits)

0 6 Ship date

Conventional WisdomV

alu

e

NPV/Options Theory

0 Planned $Value0 Planned $Value

Medium Variance

Time

Varia

nce

in C

ost,

Sche

dule

…

Project Lifecycle Phases



80% reduction96% reduction


3131

So, what can you do?

Steer your project to success Measure, analyze and embrace estimation variance

Tailor governance to your risk/value context Determine if governance approach is paying off

Getting more or less value from agile, non-agile projects?

Invest with discipline Detect early when risks are not paying off – abandon projects that are not delivering

Have improved stakeholder conversations Risk and value with funders

Conveys more complete information for better collaboration Inputs/assumptions with program staff

Asking experts for likely and ranges elicits more complete information

– Forces needed discipline

– Wide ranges engenders discussion on where to focus efforts


3232

Actual Path

Steer your project to success

Uncertainty in Stakeholder

Satisfaction Space

Initial Planned Content

Initial Planned Path

Uncertainty in delivery

Each iteration should reduce variance and increase NPV


3333

Tailor the governance solution

Kind of value for organization processes

Kind of projects Full variance – foster discovery and learning

Medium Variance – architecture alignment, lean methods

Low variance – focus on automation, cost

Medium Variance

Little discovery

Var

ianc

e in

Cos

t, S

ched

ule… Inception Elaboration Construction Transition/Maintenance

HighVariance

LowVariance

New platform architecture

Service Delivery

Code Maintenance

New capability

with existing architecture

Some discovery

Much discovery


3434

Tailoring governance is a process

Ass

ess

Imp

lemen

t

Manage

Plan

Governance Lifecycle Dep

loy

ControlMonitor

An

alyz

eE

valu

ate

Des

ign

Set Objective

Agree with stakeholders on value metrics

Choose internal measures (klocs, churn …) that contribute to business value and manage risk

Collect measures Compute

- Value- Productivity: Value per

programmer months

Reassign responsibilities, authorities to improve productivity

Tailor organization decision rights, responsibilities to risk areas


3535

Adjust Governance Solution for each risk area

Medium Variance

Time

Var

ian

ce i

n C

ost

, S

ched

ule

…

Adjust Governance Solution for each risk area



New Platform

New Capability on existing

platform

Maintenance and small change

requests

Focusing on artifacts, not process.

Constrain behavior creating uncertainty

Architectural alignment reduces

uncertainty due to diseconomy

of scaleAutomate, enforce

processes to reduce

uncertainty


3636

Disciplined Investment: Apply quantitative portfolio analysis, not only scorecards

-200

0

200

400

600

800

1000

1200

-2 0 2 4 6 8 10 12

Liklihood of Success

Val

ue

Project 1

Project 2

Project 3

Project 4

Project 5

Project 7

Project 8

Project 9

Project 6

Project 10

Cost


3838

Some final thoughts

Staying relevant requires smart risk taking

Value can only be reliably delivered if it measured and managed

Value delayed is value denied

In the end, these techniques do allow us to “Take risks and add value”


39

For more information

Contact Clay Williams, manager of the Governance Science Research Group at IBM Watson Research

[email protected], 914-784-7457


4040

© Copyright IBM Corporation 2008. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, the on-demand business logo, Rational, the Rational logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

Learn more at:

IBM Rational software

IBM Rational Software Delivery Platform

Process and portfolio management

Change and release management

Quality management

Architecture management

Rational trial downloads

Leading Innovation Website

IBM Rational TV

IBM Rational Business Partners