Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville

Chapter 13: Regression Testing

Omar Meqdadi

SE 3860 Lecture 13Department of Computer Science and Software Engineering

University of Wisconsin-Platteville

2

Topic Covered

What is Regression Testing? Why Regression testing? Regression Testing Process Object-Oriented Regression Testing

3

What is Regression Testing?

Testing activities occur after software changes During software maintenance phase

Execution of a set of test cases on a program in order to ensure that implemented changes do not produce unintended faults

Regression Testing Levels

Unit Testing Integration Testing Acceptance Testing

Who Perform Regression Testing

Developers: regression testing at the unit level or integration

Test Engineers : regression testing at the function level

Quality Engineers: regression testing at the system level

Why Regression Testing?

Validate and Verify the modified software Validate and verify the new features Ensure existing features are still working(change impact)

Small changes in one part of a program may have indirect undesired effects in other unrelated parts of the program

Fixing could introduce new bugs

Why Regression Testing?

8

Regression Testing Process

Input: Given program P, its modified version P’, and a test set T

that was used to previously test P Goal:

Find a way to utilize T to gain sufficient confidence in the correctness of P’

Regression Testing Process

Activities: Regression Test Selection

Select T’ ⊆ T, a set of test cases to execute on P’ Test Suite Execution

Test P’ with T’, establishing correctness of P’

Regression Test Selection

Selecting a subset of the existing test suite based on information about: Original Version (P) Modified Version (P’) Original Test Suite (T)

Approaches: Select All Modification-Based Test Selection Coverage-Based Test Selection

Test Set Minimization Test Case Prioritization

Select All

Traditional approach Select all the test cases in T to test P’ Advantage:

Safe Problem:

Too expensive Note: the new and the old programs could produce

different outputs

Modification-Based Test Selection

Select set of test cases T that covers only modified or affected portions of P

Based on Module-Based Firewall A changed software module and a closure of all possible affected

modules and related integration links in a program based on the static and dynamic analyses (e.g.: a control-flow graph).

We can come out different kinds of firewalls based on various test models such as: Control Firewall Data Firewall Function (method) Calling Firewall Class Firewall (Object-Oriented Regression Testing)


Test selection concept based om Module Firewall re-test of the changed module and its affected modules

within the firewall re-test all of related integration links in the firewall

Main

M1 M3M2

M7M6M5M4

M8


: Control Flow

Mi : Module

Main

M1 M3M2

M7M6M5M4

M8

Changed Module


Main

M1 M3M2

M7M6M5M4

M8

Changed Module


A control based module firewall: M5, M1, M8 , and Main

Main

M1 M3M2

M7M6M5M4

M8

1

2

3

Changed Module


A control based module firewall: M5, M1, M8 , and MainRe-testing at the unit level: M5

Re-testing at the integration level: 1, 2, 3


Problems: Still too many test cases

Need to further reduce the number of regression tests Coverage: the risk of missing test cases that are able to

reveal defect not yet discovered

Test Set Minimization

The test suite is reduced to a minimal subset that maintains the same coverage as the original test suite with respect to a given coverage criterion

Goal: Find a minimal subset T’of T such that each testable entity (function, statement,…) is covered by at least one test in T’

What we need ? Coverage (Execution Flow Trace) per test case


Steps: Step1: From old version P : determine the coverage for

each test t ⊆ T, determine which of the k testable entities is covered.

Step2: Find a minimal subset T’ of T such that each testable entity (function, statement,…) is covered by at least one test in T’by using Test Case Prioritization

Note: T’is not unique

Test Case Prioritization

Tests are prioritized based on some criteria. Highest priority test are executed earlier Test Prioritization Approaches:

Total statement coverage # of statements executed

Additional statement coverage((Heuristic #1) A higher priority is assigned to a test that covers the higher

number of not yet executed statements Total function coverage

# of functions executed Additional function coverage


Example1: Test prioritization based on total function coverage Suppose

P contains two functions: main and foo suppose that P is tested using test cases t1 and t2 During testing it was observed that :

t1 causes the execution of main but not of foo t2 causes the execution of both main and foo

Now suppose that P’ is obtained from P by making some modification

Which of the two test cases should be included in the regression test suite?

regression test suite consists of only t2 (minimization)


Example2: Test prioritization based on total statement coverage Assume P contains two functions main and f1, where the

flow charts are as follow


Example2: Assume the following test cases

Test case Coverage

T1 main: {1, 2, 3} f1: {1, 3}T2 main: {1, 3} f1: {1, 3}T3 main: {1, 3} f1: {1, 2, 3}

A minimal test set for regression testing is {t1, t3}

Test Case Prioritization

Additional statement coverage((Heuristic #1) A higher priority is assigned to a test that covers the higher

number of not yet executed statements

Additional Statement Coverage

Executed statements for each test t1: S1, S2, S3

t2 : S1, S5, S8, S9

t3 : S1, S5, S7

t4 : S1, S5, S3, S4

t5 : S1, S2, S7

t6 : S1, S2

t7 : S1, S2, S4

t8 : S1, S2, S3, S4, S7

t9 : S1, S6

t10 : S1, S2



t2 : S1, S5, S8, S9

t3 : S1, S5, S7

t4 : S1, S5, S3, S4

t5 : S1, S2, S7

t6 : S1, S2

t7 : S1, S2, S4

t8 : S1, S2, S3, S4, S7

t9 : S1, S6

t10 : S1, S2



t2 : S1, S5, S8, S9

t3 : S1, S5, S7

t4 : S1, S5, S3, S4

t5 : S1, S2, S7

t6 : S1, S2

t7 : S1, S2, S4

t8 : S1, S2, S3, S4, S7

t9 : S1, S6

t10 : S1, S2

S: t8


Executed statements for each test t1: S1, S2, S3 Covered statements t2 : S1, S5, S8, S9 S1, S2, S3, S4, S7

t3 : S1, S5, S7

t4 : S1, S5, S3, S4

t5 : S1, S2, S7

t6 : S1, S2

t7 : S1, S2, S4

t9 : S1, S6

t10 : S1, S2

S: t8



t3 : S1, S5, S7

t4 : S1, S5, S3, S4

t5 : S1, S2, S7

t6 : S1, S2

t7 : S1, S2, S4

t9 : S1, S6

t10 : S1, S2

S: t8



t3 : S1, S5, S7

t4 : S1, S5, S3, S4

t5 : S1, S2, S7

t6 : S1, S2

t7 : S1, S2, S4

t9 : S1, S6

t10 : S1, S2

S: t8, t2


Executed statements for each test t1: S1, S2, S3 Covered statements

S1, S2, S3, S4, S5, S7, S8, S9

t3 : S1, S5, S7

t4 : S1, S5, S3, S4

t5 : S1, S2, S7

t6 : S1, S2

t7 : S1, S2, S4

t9 : S1, S6

t10 : S1, S2

S: t8, t2


Executed statements for each test t1: S1, S2, S3 Covered statements S1, S2, S3, S4, S5, S7, S8, S9

t3 : S1, S5, S7

t4 : S1, S5, S3, S4

t5 : S1, S2, S7

t6 : S1, S2

t7 : S1, S2, S4

t9 : S1, S6

t10 : S1, S2

S: t8, t2


Executed statements for each test t1: S1, S2, S3 Covered statements

S1, S2, S3, S4, S5, S7, S8, S9

t3 : S1, S5, S7

t4 : S1, S5, S3, S4

t5 : S1, S2, S7

t6 : S1, S2

t7 : S1, S2, S4

t9 : S1, S6

t10 : S1, S2

S: t8, t2 , t9

Test Set Minimization/ Prioritization

Problem:

Test minimization discards test cases.

There is a chance that, if P’ were executed against a

discarded test case, it would reveal an error in the

modification made

Object-Oriented Regression Testing

Typical Cases Inheritance:

Getting a new subclass working without exercising the superclass is usually easy

No need to re-test the superclass We assume that inherited features just work

Component based development (Re-use) OO technologies offer strong support for reuse but they

cannot prevent regression bugs No unit-testing We need integration testing


Dependency: Dependencies occur for many reasons and so regression

faults can occur in many ways Assume a dependency is between B and D

D has a side effect on B: B fails because a new action of D is inconsistent with B’s requirements and assumptions with respect to D

D is a client of B: D sends a message that violates B’s invariant/precondition. B is not defensive and accepts the incoming message, leading to failure


Dependency (contd.): D is a server of B: B sends a message to D. D has been

changed (or contain bugs). D returns a value that causes a violation of B’s invariant . B fails or returns an invalid value to another baseline component C. C fails


OO- Regression Testing Concepts Each class must be tested when it is developed Each class needs a regression test Regression tests need to have standard interfaces Thus, we can build the regression test when building the

class and have a better, more stable product for less work


OO- Regression Testing Strategy Based on the Class Firewall Concept

Identify changed classes Identify affected classes (using dependency relation) Apply the Class Test Order strategy to perform class re-

testing at the unit level Use the Class Test Order to re-integrate classes together. Select, reuse, and define test cases

Example: A Changed Class in Class Diagram

X

Y

D

T

S

I

U

A C

B

M

N

R V

L

X

Y

D

T

S

I

U

A C

B

M

N

R V

L

Example: A Class Firewall in Class Diagram

X

Y

D

T

S

I

U

A C

B

M

N

R V

L

1

1.1

1.2

1.3

1.42.1

2

2.2

3 3.1

3.2

Example: Class Test Order for The Class Firewall

4.1

4

Documents

Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville