(OO) Regression Testing

2006-2007 INT-Evry (Masters IT– Soft Eng) RegressionTesting.1

(OO) Regression Testing

Regression testing is the execution of a set oftest cases on a program in order to ensure that its revisiondoes not produce unintended faults, does not "regress" -that is, become less effective than it has been in the past

Informally -



Given program P, its modified version P’, and a test set Tthat was used to previously test P, find a way to utilize T to gain sufficient confidence in the correctness of P’

More Formally



What About Changing Requirements?

Test case revalidation problem

Identify and modify/remove the obsolete test cases from T if specifications have changed.

Question: What about adding new test cases ?



Test Suite Maintenance – What Order?

Select T’ T, a set of test cases to execute on P’⊆Regression test selection problem

Test P’ with T’, establishing correctness of P’ w.r.t. T’Test suite execution problem

If necessary, create T’’, a set of new functional or structural test cases for P’Coverage identification problem

Test P’ with T’’, establishing correctness of P’ w.r.t. T’’Test suite execution problem

Create T’’’, a new test suite and test execution profile for P’, from T, T’, and T’’.Test suite maintenance problem



Test Suite Maintenance – Discarding baseline test cases

• Broken test cases: fails to run• Obselete test cases: no longer match requirements• Uncontrollable test cases: may not be repeatable (see later)• Redundant test cases: where 2 or more test cases check the

same thing (this is quite a complex concept)



Test Suite Maintenance – Reduction

Inclusiveness – percentage of baseline tests that may show regression faults (safe => 100% inclusiveness). This requires formal dependency analysis and is a whitebox technique sometimes called retest within firewall.

Precision – percentage of baseline tests in a reduced test suite that cannot reveal regression faults and that are not selected for the reduced test suite.

Note: one can argue that no test suite can be both safe and 100 precise as there is no way to decide exactly which tests will pass or fail for the changed system

Note: passing a “safe” regression suite does not guarantee absence of regression bugs – it only guarantees that all baseline test cases that could reveal a regression bug have been exercised



Test Suite Maintenance – Unsafe Reduction

Systematic Sampling

Random Sampling

Coverage-based filtering

By profiling (use)

By Intuition

None of the following are safe because they are not based on dependency analysis:



Can We Automate The Process?

Resulti = Resulti’, forall i



Partial regression testing?

If the time (and money …) interval allocated to regression testing is limited, only a fraction of the test suite can be executed, with the risk of missing test cases that are able to reveal defect not yet discovered



Partial regression testing: techniques

[Regression test selection:] the cost of regression testing is reduced by selecting a subset of the existing test suite based on information about the program, modified version and test suite.

[Test suite minimization:] the test suite is reduced to a minimal subset that maintains the same coverage as the original test suite with respect to a given coverage criterion.

[Test case prioritization:] test cases are ordered so that those with the highest priority are executed earlier, for example with the objective of achieving code coverage at the fastest possible rate, or of exercising the modules according to their propensity to fail.



RT and Software Maintenance

Corrective Maintenance – changes made to debug a system after a failure is observed

Adaptive Maintenance – changes made to achieve continuing compatibility with the target environment

Perfective Maintenance – changes designed/made to improve or add capabilities

Preventive Maintenance – changes made to improve quality of software

RT is required for each of these: to reveal side effects and bad fixes



Bad Fixes

Bad fix injection rates are normally between 2 and 20%

Iterative approach (with OO) demands much more care with bad fixes – as it focuses on changing code

RT must be done in parallel with new code development – otherwise regression faults will go unnoticed



OO testing: re-use techniques – inheritance problem

Getting a new subclass working without exercising the superclass is usually easy

It is tempting to assume that inherited features just work



OO testing: component based development

An adequate regression test suite that can be rerun by a consumer provides a convincing demonstration that a component works out of the box.

RT does not reduce the need to develop and execute tests of new and changed capabilities

OO technologies offer strong support for reuse but they cannot prevent regression bugs.



Regression Faults

When a stable baseline system B and a delta component D pass individually adequate tests, but fail when used together

D can cause some component in B to fail only if there is some dependency between them.

Dependencies occur for many reasons and so regression faults can occur in many ways. Question: how many can you think of?

There is no practical means to develop a test suite guaranteed to reveal all regression faults



Regression Faults: typical cases

D has a side effect on B:

B fails because a new action of D is inconsistent with B’s requirements, assumptions, or contract with respect to D




D is a client of B:

D sends a message that violates B’s invariant/precondition. B is not defensive and accepts the incoming message, leading to failure




D is a server of B:

B sends a message to D. D’s postconsitions have changed (or contain bugs). D returns a value that causes a violation of B’s invariant . B fails or returns an invalid value to another baseline component C. C fails




D is a revised superclass:

B is a client of E, a subclass of D. The interface of D has not changed but its behaviour has: and so the effective behaviour of E is different. This behaviour may introduce timing or nondeterministic problems which can lead to failure of B.




D is a new subclass of a polymorphic class hierarchy that is not LSP (Liskov substitution principle ) compliant:

B is a client of D and expects the new subtype to be compatible. D does not comply leading to an inconsistent state and so when B attempts to use this object it fails.




An undesirable feature interaction occurs:

B’s functionality extended by change to/addition of D1B’s functionality extended by change to/addition of D2

B+D1 passes regression testsB+D2 passes regression tests

But B+D1+D2 fails regression tests

Note: this is a very abstract view of feature composition (+)



Regression and Unit Testing: test frameworks

A test framework provides reusable test functionality which:A test framework provides reusable test functionality which:

•Is easier to use (e.g. don’t have to write the same code for Is easier to use (e.g. don’t have to write the same code for each class)each class)•Is standardized and reusableIs standardized and reusable•Provides a base for regression testsProvides a base for regression tests

A unit testing framework is unit & regression testing A unit testing framework is unit & regression testing on steroidson steroids



Regression and Unit Testing: unit test frameworks

Each class must be tested when it is developedEach class must be tested when it is developed

Each class needs a regression testEach class needs a regression test

Regression tests need to have standard interfacesRegression tests need to have standard interfaces

Thus, we can build the regression test when building the Thus, we can build the regression test when building the class and have a better, more stable product for less workclass and have a better, more stable product for less work



Automating with unit test frameworks

The real power of regression tests happens when they are The real power of regression tests happens when they are automated: this requires they report pass/fail results in a automated: this requires they report pass/fail results in a standardized waystandardized way

Can set up jobs to:Can set up jobs to:

•Clean & check out latest build treeClean & check out latest build tree•Run testsRun tests•Put results on a web page & send mail (if tests fail)Put results on a web page & send mail (if tests fail)

JUnit & ant have code to do all of this (You have already JUnit & ant have code to do all of this (You have already seen these?)seen these?)




Effective automated RG requires:

•Version control•Modular structure•Smart Means of comparing baseline and delta results•Shuffling•Assertions for built-in test




Environmental factors can lead to less-than-identical test configuration:

•Different OSs•Different content in persistant storage•SUT uses nondeterministic objects•SUT uses non-repeatable pseudorandom generation •SUT is sensitive to timing differences•SUT is sensitive to non-controllable system input•SUT uses time sensitive code



Lets learn about regression testing by looking at a practical example:

A simple array of integer pairs is required that can be initialised (with a specified size) and can be randomized with random values within a certain range. It must also provide a method to report if the values are strictly increasing in both dimensions.

Check out the Source Code at the web site

IntPair.java, Compare.java, IntPairArray.java,

TestIntPairArray.java

Is the following test suitable for regression testing? How can we improve things?



Lets learn about regression testing by looking at a practical example:

Now, experiment by extending the system and deliberately inserting typical regression faults (see earlier)

How well does the regression test suite do in finding faults?

How should we maintain the test suite?

Documents

(OO) Regression Testing