Global Sensitivity Analysis of Predictor Models in Software Engineering

Dr. Stefan Wagner, TU Munchen PROMISE – May 20, 2007 – 1 / 27

Global Sensitivity Analysis of Predictor

Models in Software Engineering

Stefan WagnerSoftware & Systems EngineeringTechnische Universitat Munchen

[email protected]

May 20, 2007

[email protected]

Introduction

IntroductionPredictor Models inSoftwareEngineering

ProblemExample:Fischer-WagnerModelQuestions About theModel

Overview

Global SensitivityAnalysis

Approach for SE

Summary


Predictor Models in Software Engineering



Overview


Approach for SE

Summary


■ Predictor models describe situations in software engineering■ Various types, aims, . . .■ Examples

◆ COCOMO (costs)◆ Musa-Okumoto (reliability)◆ Fault-proneness

Problem



Overview


Approach for SE

Summary


■ The models themselves can be complex■ Their use needs a lot of effort■ How can I analyse those models themeselves?■ How can I simplify them?■ How do I improve their predictive power?

Example: Fischer-Wagner Model



Overview


Approach for SE

Summary


■ Reliability model used at Siemens COM■ Two parameters estimated by failure data■ Failure probability of a fault: pa = p1 · d

(a−1)

■ Geometrical distribution : Fa(t) = 1 − (1 − pa)t

■ Cummulated failures up to t:µ(t) =

∑∞

a=1 1 − (1 − p1 · d(a−1))

■ Input parameters

◆ p1: Highest failure probability of a fault◆ d: Complexity of the system ]0; 1[◆ t: Execution time (incidents)◆ inf : approximation of ∞

■ Output: µ(t)

More details: S. Wagner, H. Fischer. Ada-Europe, 2006

Questions About the Model



Overview


Approach for SE

Summary


■ How do the input parameters influence the output?■ Which parameter(s) do I have to estimate best to get a good

prediction?■ Are there insignificant parameters that I can remove?

Overview



Overview


Approach for SE

Summary


Introduction

Global Sensitivity Analysis

Approach for SE

Summary

Global Sensitivity Analysis

Introduction


Definition

MethodVariance-basedMethods

Main Effect

Higher-Order Effects

Total Effect

Approach for SE

Summary


Definition

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


■ Saltelli (2000): “Sensitivity analysis studies the relationshipsbetween information flowing in and out of the model.”

■ How do the input parameters influence the output?■ How can this influence by quantified?■ Global properties

◆ Inclusion of influence of shape and scale◆ Multidimensional averaging

Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

x1

x2

x3

xn

Yf(x)

(Based on slide of S. Tarantola)

Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Method

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Input Model Output

Yf(x)

x1

x2

x3

xn


Variance-based Methods

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


■ Global analysis usually variance-based■ “model-free”■ Analysis on the basis of sensitivity indices

◆ Main or first-order effect

◆ Higher-order effects

◆ Total effect

■ Decomposition in main effects and interactions

y = f(x) = f0 +k∑

i=1

fi(xi) +∑

i

∑

j>i

fij(xi, xj)

+ . . . + f1,2,...,k(x1, x2, . . . , xk)

Variance-based Methods

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


■ Global analysis usually variance-based■ “model-free”■ Analysis on the basis of sensitivity indices

◆ Main or first-order effect

◆ Higher-order effects

◆ Total effect

■ Decomposition in main effects and interactions

For example k = 3:

f(x) = f0 + f1(x1) + f2(x2) + f3(x3) + f12(x1, x2)

+f13(x1, x3) + f23(x2, x3) + f123(x1, x2, x3)

Main Effect

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Decomposition of the variance V of f(x)

Var(y) = V =k∑

i=1

Vi +∑

i

∑

j

Vij

∑

i

∑

j

∑

k

Vijk + . . . + V1,2,...k

First-order sensitivity coefficient

Si =Vi

V

(Factors Priorisation)


Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


Higher-order indices analog

Si1...io = Vi1...io/V

With k = 4:2. order: S12, S13, S14, S23, S24, S34

3. order: S123, S124, S134, S234

4. order: S1234

Total Effect

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary


■ Sum of the first- and higher-order effects of xi

■ Removal of the non-relevant parts

S1 + S2 + S3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 +

S123 + S124 + S134 + S234 + S1234 = 1

(Factors Fixing)

Total Effect

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary




S1 + S2/// + S3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 +

S123 + S124 + S134 + S234 + S1234 = 1

(Factors Fixing)

Total Effect

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary




S1 + S2/// + S3/// + S4 + S12 + S13 + S14 + S23 + S24 + S34 +

S123 + S124 + S134 + S234 + S1234 = 1

(Factors Fixing)

Total Effect

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary




S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23 + S24 + S34 +

S123 + S124 + S134 + S234 + S1234 = 1

(Factors Fixing)

Total Effect

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary




S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23//// + S24//// + S34//// +

S123 + S124 + S134 + S234 + S1234 = 1

(Factors Fixing)

Total Effect

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary




S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23//// + S24//// + S34//// +

S123 + S124 + S134 + S234//////+ S1234 = 1

(Factors Fixing)

Total Effect

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary




S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23//// + S24//// + S34//// +

S123 + S124 + S134 + S234//////+ S1234 = 1

ST1 = S1 + S12 + S13 + S14 + S123 + S124 + S134 + S1234

(Factors Fixing)

Total Effect

Introduction


Definition


Main Effect


Total Effect

Approach for SE

Summary




S1 + S2/// + S3/// + S4/// + S12 + S13 + S14 + S23//// + S24//// + S34//// +

S123 + S124 + S134 + S234//////+ S1234 = 1

ST1 = S1 + S12 + S13 + S14 + S123 + S124 + S134 + S1234

ST2 = S2 + S12 + S23 + S24 + S123 + S124 + S234 + S1234

(Factors Fixing)

Approach for SE

Introduction


Approach for SE

Overview of theApproach

DeterminingDistributionsExample:DistributionsVisualising UsingScatterplots

p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


Overview of the Approach

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


Inputdistributions

.045

.003

.002

.375

Sensitivityindices

Determiningdistributions

Visualisingusing

scatterplots

Globalsensitivityanalyses

Scatterplots

Determining Distributions

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


■ Distribution of the values of the input parameteters■ Derived from

◆ scientific literature◆ physical boundaries◆ expert opinion◆ surveys◆ experiments

Example: Distributions

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


■ Based on expert opinion■ General parameter

◆ p1 uniformly distributed between 0 and 1◆ d uniformly distributed between 0.9 and 1

■ Dependent on application

◆ inf e.g.. uniformly distributed between 50 and 500◆ For t e.g. t ∼ N (1000, 200)

Visualising Using Scatterplots

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


■ Sampling using the distributions■ Pairwise relationship between factors■ Detection of errors in the model■ Indications of influences

p1 and µ

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0

100

200

300

400

500

600

700

800

900

1

p1

µ

d and µ

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 0

100

200

300

400

500

600

700

800

900

µ

d

t and µ

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


99200 99400 99600 99800 100000 100200 100400 100600 100800 0

100

200

300

400

500

600

700

800

900

µ

t

inf and µ

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


900

800

700

600

500

400

300

200

100

0 400 450 500 550 600 650 700 750 800 850 900

µ

inf

Applying Global SA

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


■ Various methods for calculating indices■ FAST (Fourier Amplitude Sensitivity Test) gives quantitative

results■ Uses sampled data■ Tool-support: Simlab

Indizes

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


Main Effect

t 6.88e-5p1 0.0090d 0.9026inf 0.0158

Indizes

Introduction


Approach for SE



p1 and µ

d and µ

t and µ

inf and µ

Applying Global SA

Indizes

Summary


Main Effect

t 6.88e-5p1 0.0090d 0.9026inf 0.0158

Total Effect

t 0.005788p1 0.022365d 0.975155inf 0.086735

Summary

Introduction


Approach for SE

Summary

Conclusions


Conclusions

Introduction


Approach for SE

Summary

Conclusions


■ Sensitivity analysis useful tool for predictor models■ SA can help to

◆ find errors in the models◆ simplify the models◆ identify interactions between input parameters◆ identify parameters that should be investigated more◆ get more robust predictions

■ Experience with

◆ reliability model◆ QA economics model◆ process model◆ expert system for IT tools

■ Good tool-support available (Simlab:http://simlab.jrc.cec.eu.int/)

http://simlab.jrc.cec.eu.int/

Technology

Global Sensitivity Analysis of Predictor Models in Software Engineering