Statistical Tools for

Supporting Software Testing

Ed Brandt (Refis)Alessandro Di Bucchianico (TU/e)

12th Dutch Testing DayNovember 17, 2006

2© TU/e - Refis

OutlineBasics of Reliability Analysis

Case of Rijkswaterstaat (Dutch Directorate for Public Works and Water Management )

How to Apply Reliability modelsStatistical Best PracticesTool by LaQuSo and Refis

3© TU/e - Refis

4© TU/e - Refis

The weakest link?

5© TU/e - Refis

The main question:How to determine the failure probability of software?

6© TU/e - Refis

Determine failure probability byRisk analysisCode analysisTestingReliability analysis

7© TU/e - Refis

Test results200 bugs found, of which

198 solved1 work around1 minor bug still open

1000 bugs found,of which

998 solved1 work around1 minor bug still open

8© TU/e - Refis

Definition of “Reliability”

"The probability that an item will perform a required function without failure under stated conditions for a stated period of time”

"The probability that an item will perform a required function without failure under stated conditions for a stated period of time”

9© TU/e - Refis

Reliability growth

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100105110115120125130135140145150155160165170175180185190195200

time

relia

bilit

y

10© TU/e - Refis

Reliability growth

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100105110115120125130135140145150155160165170175180185190195200

time

relia

bilit

y

11© TU/e - Refis

Reliability growth curve

0

0.1

0.2

0.3

0.4

0.5

.,6

0.7

0.8

0.9

1

1 2 3 4 5 6 7

interval

relia

bilit

y

period = 5

12© TU/e - Refis

13© TU/e - Refis

ResultsReliability for 1 year for defects with severity =1

0.00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

.,80

0.90

1.00

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71testinterval

relia

bilit

y

Schneidewind: all

Reliability for 1 year= 0.9014

Reliability for 1 year= 0.9014

14© TU/e - Refis

ResultsCumulative number of defects

0

10

20

30

40

50

60

70

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71

Test interval

Num

ber o

f def

ects

Generalized Poisson Schneidewind: all Act. cumulative number of failures

Expected number of defectsin production = 5 or 6

Expected number of defectsin production = 5 or 6

15© TU/e - Refis

How to Apply Reliability Models

16© TU/e - Refis

Step 1: Data collectiondocument data sourcesrecord data collection procedures clean-up data

17© TU/e - Refis

Step 2: Determine Growth

formal tests (Laplace, MIL-HB, LRT,…)

time

cumulativenumber of

defects

18© TU/e - Refis

Step 3: Initial Model Selection

19© TU/e - Refis

Step 4: Estimate Models

parameters

cumulativenumber of

defectslikelihood

ML LS

time

20© TU/e - Refis

Step 5: Validate Model

goodness-of-fit plots (TTT-plot, u-plot, QQ-plot, residual plots)goodness-of-fit tests (Kolmogorov, Cramer-von Mises,…)

time

cumulativenumber of

defects

maximalabsolutedeviation

21© TU/e - Refis

Step 6: Interpret Model

time

cumulativenumber of

defects

use confidence bounds to learn about precision

22© TU/e - Refis

Refis LaQuSo Tool

23© TU/e - Refis

Conclusionsstatistics can be a useful additional method to evaluate software reliabilitya tool is being developed for software reliability analyses that incorporates best practices (“coding standards”) from statistics

24© TU/e - Refis