Significance of Test Coverage

Sep 2011

2 © Virtusa Corporation ● Confidential

Contents

Meaning of test coverage

Business importance for test coverage

Types of test coverage

Tools for test coverage

Test coverage in Agile

Test coverage in Risk based testing

Test coverage and Regression test decisions

Interpretation of test coverage numbers

3 © Virtusa Corporation ● Confidential

Meaning of test coverage

Total number of requirements tested against the total number of requirements in scope

Req ID Test case reference Test status

REQ 1 TC x to z Pass/Fail

REQ 2 TC x to z Pass/Fail

REQ 3 TC x to z Pass/Fail

REQ 4 TC x to z Pass/Fail

REQ ….n TC x to z Pass/Fail

4 © Virtusa Corporation ● Confidential

Business importance of test coverage

• Test coverage is the direct indicator of the unknown risks• Sometimes it is a contractual requirement to show 100% test

coverage• Effectiveness of other defect metrics depends on test coverage• Many times 100% test coverage from SIT is the entry criteria for

UAT and for Go-live

5 © Virtusa Corporation ● Confidential

Types of test coverage

Test coverage

ReleaseProject

Design Execution

6 © Virtusa Corporation ● Confidential

Tools for test coverageRequirement traceability matrix

Req ID UC # Unit test case #

Test Scenario #

SIT Test case #

UITTest case #

Test Status

Defect ID

7 © Virtusa Corporation ● Confidential

Test coverage in Agile

Sprint 1 Sprint 2 Sprint 3 Sprint n

Release X

Reg Test coverage

0%

Reg Test coverage

65%

Reg Test coverage

70%

RegTest coverage

85%

Reg Test coverage

100%

8 © Virtusa Corporation ● Confidential

Test coverage in Risk based testing

• Risk based testing is conscious decision to reduce testing time• In this case test coverage cannot be 100%• For such situations, we need to calculate & ensure “Critical”

and “High” complexity scenario coverage separately• Need to intimate customer upfront about the test coverage

9 © Virtusa Corporation ● Confidential

Test coverage and Regression test decisions

Req ID

UC #

Unit test case #

Test Scenario #

SIT Test case #

UITTest case #

Test Status

Defect ID

• We need two matrices to decide regression test scenarios• 100% regression testing for all cycles is impractical• We need functional dependency matrix to do impact analysis• Further we need Req Traceability matrix to decide the test

scenarios

Requirement Traceability MatrixFunctional Dependency Matrix

Functions Fun 1 Fun 2 Fun 3 Fun..n

Fun1

Fun2

Fun3

Req ID Req 1 Req 2 Req 3 Req n

10 © Virtusa Corporation ● Confidential

Interpretation of test coverageDSI trend Vs Test coverage

Project A – DSI increases along with test coverage in cycle 3Project B – DSI decreases over cycle after cycle even if the coverage is increased in the regression test

11 © Virtusa Corporation ● Confidential

Thank You