GQM and data analysisExample from a Norwegian

company

Tor StålhaneIDI / NTNU

The ProblemThe company in question develops hardware and

software. They have two software groups, each with circa 15 developers.

Part of the system is developed in SDL. In order to focus their V&V work better they needed to know which SDL module characteristics that caused errors.

Possible candidates were number of module states, number of input signals etc.

What to measure?

In order to get a consistent and efficient data collection and analysis, we started with a GQM process.

• A half-day basic course in GQM.• A half-day workshop where they identified

– the questions that needed answers.– which metrics they needed in order to answer

these questions.

The Metrics

Some of the metrics collected for each SDL module:

• Number of errors found.• Subjective - perceived - complexity.• Number of pages of SDL description.• Number of SDL states• Number of signals in• Number of signals out

GQM-metrikker

Metrikk PMECMD PMECON CAMCON HPOTBL TAMCON TAMTST TTRMOT TTRTST SUMM1: Antall feil 8 1 0 1 3 1 2 0 16M2: Sum kost *) 16 0 0 3 8,5 2 5,5 0 35M2_1: Kost, koding 0 0 0 0 0 0 0 0 0M2_2: Kost, Tidsvindu 0 0 0 0 3,5 0 5,5 0 9M2_3, Kost, Ressurskonflikter 0 0 0 0 0 0 0 0 0M2_4, Kost, Manglende Mottak 3 0 0 0 3 0 0 0 6M2_5, Kost, HW/Mekanikk 0 0 0 0 2 0 0 0 2M2_6, Kost, Logiske 13 0 0 3 0 2 0 0 18M2_7, Kost, kjappe løsninger 0 0 0 0 0 0 0 0 0M2_8, Kost, Diverse 0 0 0 0 0 0 0 0 0M3_1: Feil, koding 0 0 0 0 0 0 0 0 0M3_2: Feil, Tidsvindu 2 0 0 0 1 0 2 0 5M3_3, Feil, Ressurskonflikter 0 0 0 0 0 0 0 0 0M3_4, Feil, Manglende Mottak 1 0 0 0 1 0 0 0 2M3_5, Feil, HW/Mekanikk 0 0 0 0 1 0 0 0 1M3_6, Feil, Logiske 5 1 0 1 0 1 0 0 8M3_7, Feil, kjappe løsninger 0 0 0 0 0 0 0 0 0M3_8, Feil, Diverse 0 0 0 0 0 0 0 0 0M4_a, Feil pga andre modulerM4_b, Feil pga andre produkterM5: Antall tilstander 52 56 32 21 25 10 6 3 205M6a: Antall signaler inn, eksterne 80 55 23 48 36 13 9 10 274M6b: Antall signaler som mottas 882 1032 136 125 431 43 42 7 2698M7: Antall goto 117 55 0 0 6 0 3 0 181M8: Antall variabler 103 37 26 45 31 13 14 7 276M9: Antall signaler ut, eksterne 87 43 24 45 38 15 10 11 273M10: Sider SDL 101 54 33 72 50 14 12 5 341M11: Kompleksitet av omgivelser 3 3 2 2 3 2 2 1 18M12: Kompleksitet av blokk 3 2 2 3 3 2 1 1 17M13: Linjer PR-kode 4923 1643 1380 1493 2652 397 665 106 13259Antall if-statements 38 4 13 0 55

Qc – block complexity When we defined the Qc question, we decided

to use a Kiviat diagram to display the metrics included in this question.

We show the following data:• Non-modified metrics. Just to show that it is

not a good idea • Normalized metrics values• Mean metrics values for each complexity class

Non-modified metrics

The large value of M6b reduces everything else to “noise”

Normalized metrics values – 1 M5

PMECDM 52PMECOM 56CAMCON 32HPOTBL 21TAMCON 25TAMTST 10TTRMOT 6TTRTST 3

M5PMECDM 0,93PMECOM 1,00CAMCON 0,57HPOTBL 0,38TAMCON 0,45TAMTST 0,18TTRMOT 0,11TTRTST 0,05

}max{/' Xxx ii

Normalized metrics values – 2

High complexity Low complexity

Be ware different scales on the axis in the two diagrams

Mean metrics values – 1

Be ware differentscales on the axisin the three diagrams

Mean metrics values – 2

0

2

4

6

8

10

12

14

16

Dag

er

PMECMD PMECON CAMCON HPOTBL TAMCON TAMTST TTRMOT TTRTST

Blokk

Q1_2a: Fordeling av rettekostnader over blokker

M2: Sum kost *)

What makes it complex - 1

We went through all the hypothesis put forward by the developers during the GQM session. We will look at three of them:

• Number of states - M5• Number of signals out - M9• Number of pages in the SDL description - M10

What makes it complex - 2The data for the three metrics M5, M9 and M10were sorted according to the complexity scores(High, Medium and Low).

An ANOVA analyses was then performed for each data set. We decided to require a p-value better than 10%

ANOVA results - 1

Number of states – M5:

Source of VariationSS df MS P-value Between Group 1190,04 2 595,02 0,25Within Groups 1631,83 5 326,37

Number of states does not contribute significantly to the complexity as perceived by the developers.

ANOVA results - 2

Number of signals out – M9:

Source of VariationSS df MS P-valueBetween Group 2779,0 42 1389,521 0,098Within Groups 1813,83 5 362,77

Number of signals out contribute significantly to the complexity as perceived by the developers

ANOVA results - 3Number of pages in the SDL description – M10: Source of Variation SS df MS P-valueBetween Groups 5586,04 2 2793,02 0,04Within Groups 2133,83 5 426,77

Number of pages in the SDL description contribute significantly to the complexity as perceived by the developers

Summary - 1

SDL module complexity as perceived by thedevelopers depends on two factors:• Number of signals out• Number of pages in the SDL description

The other suspected factors identified duringthe GQM process did not give a significantcontribution.

What about Errors

We now have some ideas on what makes a module look complex to the developers.

The next step is to see if there is any connection between module complexity and the number of errors in the modules.

The ANOVA can give us an answer.

Complexity and Errors - 1Errors and complexity

Source of Variation SS df MS P-valueBetween Groups 1646,83 2 823,42 0,06 Within Groups 770,67 5 154,13

It is reasonable to assume that complex modules have more errors.

Complexity and Errors - 2If we look at the ANOVA summary table, we see

that the differences are quite large:

Groups Count Sum Average VarianceColumn 1 3 102 34 343Column 2 3 22 7,33 41,33Column 3 2 2 1 2

Due to few observations for each complexity level, the variances are large. Thus, we should not be too categorical in our conclusions.

ConclusionsWith all the necessary caveats in mind the

company decided as follows:

In order to reduce the number of errors we need to single out modules with :

• Large descriptions - more than 35 pages of SDL description.

• Many signals out - more than 30.

The limiting values are the average values from the ANOVA summary tables.