LaQuSo is an activity of Technische Universiteit Eindhoven Confidence intervals in software reliability testing Confidence intervals in software reliability

LaQuSo is an activity of Technische Universiteit Eindhoven

Confidence intervals in Confidence intervals in software reliability testingsoftware reliability testing

Alessandro Di Bucchianico (LaQuSo, Eindhoven University of Technology)

Ed Brandt and Rob Henzen (Refis, Netherlands)

ENBIS-5 Newcastle

September 15, 2005

ENBIS-5, September 15, 2005 2

Goals of this talk

show how to obtain confidence intervals for software reliability predictions from NHPP models

apply results to case study


Overview of this talk

Introduction of LaQuSo and RefisCase Dutch Ministry of Transport, Public Works

and Water Management Software reliability modelsConfidence intervals for NHPP models:

asymptotics simulation goodness-of-fit tests

Conclusions


LaQuSo: Laboratory for Quality Software

university based laboratory started at the Eindhoven University of Technology Radboud University (Nijmegen) has recently joined as partner statistics and probability group in math department at TU/e is

one of the participating groups

started in January 2004: 10 fte; will grow to 50 ftecase-study driven in cooperation with industrystatistics will be integrated part of testing and verification

activitiesmore information: www.laquso.com


Refis

consultancy company in Bilthoven, the Netherlands

activities include: software reliability assessments measurements systems for IT sector test audits

for more information, see www.refis.nl


Context of case

2/3 of the Netherlands is below sea levelprotection against sea and rivers by

dunes dikes dams sluices …

hardware reliability of sluices is well understood and documented

control of sluices by huge software systems (reliability??)


Sluice (1)


Sluice (2)


Goals of case

Case is project of Dutch Ministry of Transport, Public Works and Water Management (see www.rws.nl for general information)

Obtain information on reliability of software system

Registration system for defect detection and repair Predict system reliability with confidence bounds


Available data

Data available from three tests: plant acceptation test site acceptation test site acceptation retest

Defect counts grouped data severity index repair status …

Data was collected manually and checked on consistency etc.


Data assumptions

Assumptions are results from intensive discussions with project and test engineers all test intervals have same effort every test period corresponds to 219 days of actual use immediate correction of errors (gaps between testing periods allowed for this) no new error introduced by correction actions


Data (severity 1 FAT)

2 4 6 8 10 12 14period

2468

101214

cumulative errors


Software reliability models

Main differences with hardware reliability: no wear no burn-in exact reproducibility of errors

Hundreds of reliability growth models availableDedicated software for software reliability exists

(not always reliable, though): Casre Smerfs …


Initial Model Selection

Models available in standard software reliability packages (Smerfs, Casre) were judged on several criteria (assumptions or properties), including: upper bound on number of errors interval data length of test intervals distribution of errors shape of failure intensity …

The list of selected models included two NHPP models (Goel-Okumoto and Yamada S-shaped)


Nonhomogeneous Poisson process

( )( )!( ( ) )k

ttkP N t k e

!

( )

( ( ) )kttk

t t

P N t k e

This is a Type II model (cf. Langberg/Singpurwalla (1985)) that in general cannot be described easily in terms of time between failures.

Special case: Poisson process

0

T1 T2 T3 T4

t

N(t )=4


NHPP models

Several choices for have been introduced:

time

expected number

of failures)1()( atentL Goel-Okumoto, Musa

))1(1()( ateatntL delayed S-shaped

bt

at

ye

entL

1

)1()( inflection S-shaped

m

i

tai

ientL1

)1()(

)1ln(1

)( abta

tL

hyperexponential

logarithmic


NHPP models: inference for grouped data

data consists of counts in time intervals:

ni = # detected failures in time interval (ti-1,ti]

likelihood function (t0=0):

(t) = cumulative hazard rate at time t = expected number of failures at time t

if has parametric form, then maximizing L yields ML estimates for parameters

(t) = d/dt (t) = hazard rate at time t

n

in

i

yii

nn tLy

tLtLtLttyLyyL

i

1

12121 ))(exp(

!

))()((),,,;,,,(


NHPP models with 2 parameters: inference for parameters

))(,(: âAsymVaraNâ

eea

eaaEF

222

222

/ln/ln

/ln/ln

))(,(: êAsymCoveNê

Assume depends on 2 parameters a and e

ML-estimators have no closed form

asymptotic distribution through Fisher information:

)(),(

),()(1

êAsymVarêâAsymCov

êâAsymCovâAsymVarFV


NHPP models with 2 parameters: inference for function of parameters

assume depends on two parameters a and basymptotic distribution of functions of a and b

through Fisher information and delta method:

examples of functions of parameters include: probability of no failure in certain time period failure intensity at t=t0

),()(2)()()()(),( ,

222

êâAsymCovea

fêAsymVar

e

fâAsymVar

a

feaVar êeâaêeâa


Simulation NHPP process

Conditional on the event N(t)=n, the T1,…,Tn are distributed as the order statistics of a sample of size n from a distribution with density (t) / (t).

Hence, simulating a sample from a distribution with density (t) / (t) can be used to simulate an NHPP process with intensity (t)

( )( )!( ( ) )k

ttkP N t k e

0

T1 T2 T3 T4

t

N(t )=4


Goodness-of-fit NHPP process

Conditional on the event N(t)=n, the T1,…,Tn are distributed as the order statistics of a sample of size n from a distribution with density (t) / (t).

Hence, the Kolmogorov goodness-of-fit test based on the empirical distribution function may be used to perform a GOF test.

( )( )!( ( ) )k

ttkP N t k e

0

T1 T2 T3 T4

t

N(t )=4


Back to case study

parameter estimates and 95% confidence intervals for Goel-Okumoto model (a(1-exp(b t)):

a : ( 13.2 , 19.5989 ) b = ( 0.000818358 , 0.00318164 )

goodness-of-fit: OK at 5% levelimportant question from Dutch politics: 95%

confidence interval for probability of no failure in 1 year:

( 0.799462 , 1 ) (thus confirmation of suspicion by Ministry officials that defect system is not good enough for required probabilities)


Conclusions

asymptotic confidence intervals for functions of parameters in NHPP models may obtained from Fisher information

testing registration of Dutch water works not sufficient to obtain high-precision estimates of software reliability


Literature

Rijkswaterstaat report (confidential)Systematic description of software reliability

models, manuscript in progress (ADiB + Refis)Xie and Hong (2001), Handbook of statistics 20

(Advances in Reliability), 707-731.

Documents

LaQuSo is an activity of Technische Universiteit Eindhoven Confidence intervals in software reliability testing Confidence intervals in software reliability