Firewalls for regression testing

Spring 2013

Use of the word “Firewalls”

The way the authors White, Jaber, Robinson and Rajlich use the term “Firewall” in this work is special for their work on regression testing and must not be mixed with the common use of the term “firewall” which is part of networks and security.

Why firewalls in regression testingMost regression tests are large and time-consuming to run. The concept of a “firewall” is used to reduce the set of classes – or components – that need to be tested. A firewall in regression testing separates the classes that depend on the class that is changed from the rest of the classes. There are two central concepts – dependency and encoding

Dependency The main concept to understand when we apply the firewall idea is dependency. The main question is:Is the component dealing with the plain values of the inputorMust the component account for how those inputs were generated

Definitions – 1 Let C be a component with •inputs {Ii}

•outputs {Oj}

•post-conditions {Qk}

If Q {Qk} and I {Ii} so that Q depends on I and on a value generated by a component C1 that is connected to I through a data flow through one or more intermediary component C2,…Cm then

•I is an encoded input•C is an external component in scope of C2 and visa versa

Definitions – 2 No such component C1 => I is a plain input, i.e. not encodedAll inputs of C are plain => C is an I/O component C is external in scope of D and there is no message sent from C to D => there is a hidden dependency between C and D.

c c1 cc2c1

I I

I is a plain inputI is an encoded input

cm

…

Dependency example – 1 Component 1 generates the key k, which is converted to an integer in component 2. Component 3 converts the integer to a hash address A. Component 4 uses the hash address to do a table look-up to generate the value f(k).

If component 2 or 3 is changed, the result r may change.

Dependency example – 2

The post condition of component 4 depends on k, generated by component 1 => input A is encoded by component 2.Component 4 is external in scope of component 2 and visa versa. r = f(k) but k is changed by component 2There is a hidden dependency between 2 and 4

Example of hidden dependency

A’s post-condition is dependent on the value “colour” that is –Generated in class I–Encoded as an integer in class A (0 is red, 1 is yellow etc.)–Decoded in class B (receives colour code, decodes and paints the screen)

Thus class B is external in scope of A. The dependency between A and B is hidden since no message is sent between A and B

Global variables

The output v3 from component 3 is used to modify the global variable g which is used as input to component 4.

Component 4 is external and dependent on variable v1 and is activated by a message from component 3 to component 4 which has no data associated with it

Activation path:1, 2, 3, 4Messages path:1, 2, 3, g, 4

Simple firewall example: a TFW

TFW does not examine dependencies further away than one level from changed component

Process for determining TFW in OO systems1. Given two successive versions of an oo-system, find

the difference of the two versions and identify those classes that have changed

2. If any of the changed classes are in an inheritance hierarchy, also consider descendants of the changed class as changed

3. For each changed class, identify all the classes that send messages to the changed class or receive messages from the changed class and include them in the TFW

• See previous slide for a graph

Extended Testing Firewall - ETW

• EFW extends the TFW by taking into account the data flows, external components and hidden dependencies (see previous slides)

• EFW is obtained by following the process:– First apply steps 1-3 in order to obtain TFW4. Identify all data paths from and to a modified class5. Find all external classes in scope of the modified class (E in

the example on next slide). Include all such classes in EFW

Extended firewall – example

Difference between TFW and EFW

Empirical Study 1: TFW vs. EFW• In order to compare the concepts of Simple

(TFW) and Extended Firewall (EFW), the authors tested several builds of a TelComm system. – 66 classes, 11000 lines of code, C++

• This system had already been tested by the developers using their standard test suite. – Common – found using both TFW and EFW: 8

errors– Found only using EFW: 4 errors. Two of these had

not been found by the company’s test suite.

TelComm system- results

TelComm system: Severity of faults

• All faults in previous slide were of severity 3 except for the EFW fault in build 1.1 which is of severity 2

• No faults of severity 1 or 4

TelComm system: Extent of extra effort

• In the table on next slide, total time for testing is the sum of – Analysis time – identifying the firewall– Testing time is the time needed for

• test setup• test execution• analysis of test results.

– EFW testing requires approximately• 20 – 30% more tests• 40 – 50% more person hours

TelComm system: Extent of extra effort

• TFW: 46 person hours, 8 errors => 5.7 person hours per error found

• EFW: 58 person hours,12 errors => 4.8 person hours per error found

TelComm system: Reused vs. new tests

For TFW some testes were reused from the company’s original test suiteAll extra tests needed for EFW were written from scratch.

Empirical Study 2 TFW vs. ETF• An empirical study was also performed on a

very large system – more than 1 mill lines of code– thousands of classes– C++ code– Real time system– Windows environment

Real time system: TFW vs. EFW

• 13 common faults and 3 faults found only by EFW • Version 1: Common faults: 3 at sev. level 4, 1 at level 3, 1 at level 2 (EFW)• Version 2: Common faults 8 (6 at lev. 4, 2 at lev. 2), • EFW faults, one at level 3 one at level 2

Real time system: number of tests and effort

• TFW: 93 person hours, 13 errors => 7.2 person hours per error found

• EFW: 120 person hours, 16 errors => 7.5 person hours per error found

Conclusion • Extended Firewall testing (EFW) finds more

faults than Simple Fire Wall (TFW) testing– 50% in the first study, 23 % in the second study

• The cost in person hours per fault found is approximately the same: 5 - 8 person hours per fault

• EFW and TFW – should only be used to test updates and changes– assumes that the initial test suite exists and has a

high quality

Reference

Most of the information on these slides are from the paper (available from It’s learning)

“Extended firewall for regression testing: an experience report”

By: Lee White, Khaled Jaber, Brian Robinson and Václac RajlichFrom: Journal of Software Maintenance and Evolution: Research and Practice, 2008, 20: 419-433