(OO) Regression TestingRegression testing is the execution of a set oftest cases on a program in order to ensure that its revisiondoes not produce unintended faults, does not "regress" -that is, become less effective than it has been in the pastInformally -
(OO) Regression TestingGiven program P, its modified version P, and a test set Tthat was used to previously test P, find a way to utilize T to gain sufficient confidence in the correctness of PMore Formally
(OO) Regression TestingWhat About Changing Requirements?Test case revalidation problem
Identify and modify/remove the obsolete test cases from T if specifications have changed.
Question: What about adding new test cases ?
(OO) Regression TestingTest Suite Maintenance What Order?Select T T, a set of test cases to execute on PRegression test selection problem
Test P with T, establishing correctness of P w.r.t. TTest suite execution problem
If necessary, create T, a set of new functional or structural test cases for PCoverage identification problem
Test P with T, establishing correctness of P w.r.t. TTest suite execution problem
Create T, a new test suite and test execution profile for P, from T, T, and T.Test suite maintenance problem
(OO) Regression TestingTest Suite Maintenance Discarding baseline test casesBroken test cases: fails to runObselete test cases: no longer match requirementsUncontrollable test cases: may not be repeatable (see later)Redundant test cases: where 2 or more test cases check the same thing (this is quite a complex concept)
(OO) Regression TestingTest Suite Maintenance ReductionInclusiveness percentage of baseline tests that may show regression faults (safe => 100% inclusiveness). This requires formal dependency analysis and is a whitebox technique sometimes called retest within firewall.
Precision percentage of baseline tests in a reduced test suite that cannot reveal regression faults and that are not selected for the reduced test suite.
Note: one can argue that no test suite can be both safe and 100 precise as there is no way to decide exactly which tests will pass or fail for the changed system
Note: passing a safe regression suite does not guarantee absence of regression bugs it only guarantees that all baseline test cases that could reveal a regression bug have been exercised
(OO) Regression TestingTest Suite Maintenance Unsafe Reduction Systematic Sampling
By profiling (use)
By IntuitionNone of the following are safe because they are not based on dependency analysis:
(OO) Regression TestingCan We Automate The Process?Resulti = Resulti, forall i
(OO) Regression TestingPartial regression testing?If the time (and money ) interval allocated to regression testing is limited, only a fraction of the test suite can be executed, with the risk of missing test cases that are able to reveal defect not yet discovered
(OO) Regression TestingPartial regression testing: techniques[Regression test selection:] the cost of regression testing is reduced by selecting a subset of the existing test suite based on information about the program, modified version and test suite.
[Test suite minimization:] the test suite is reduced to a minimal subset that maintains the same coverage as the original test suite with respect to a given coverage criterion.
[Test case prioritization:] test cases are ordered so that those with the highest priority are executed earlier, for example with the objective of achieving code coverage at the fastest possible rate, or of exercising the modules according to their propensity to fail.
(OO) Regression TestingRT and Software MaintenanceCorrective Maintenance changes made to debug a system after a failure is observed
Adaptive Maintenance changes made to achieve continuing compatibility with the target environment
Perfective Maintenance changes designed/made to improve or add capabilities
Preventive Maintenance changes made to improve quality of softwareRT is required for each of these: to reveal side effects and bad fixes
(OO) Regression TestingBad FixesBad fix injection rates are normally between 2 and 20%
Iterative approach (with OO) demands much more care with bad fixes as it focuses on changing code
RT must be done in parallel with new code development otherwise regression faults will go unnoticed
(OO) Regression TestingOO testing: re-use techniques inheritance problemGetting a new subclass working without exercising the superclass is usually easy
It is tempting to assume that inherited features just work
(OO) Regression TestingOO testing: component based developmentAn adequate regression test suite that can be rerun by a consumer provides a convincing demonstration that a component works out of the box.
RT does not reduce the need to develop and execute tests of new and changed capabilities
OO technologies offer strong support for reuse but they cannot prevent regression bugs.
(OO) Regression TestingRegression FaultsWhen a stable baseline system B and a delta component D pass individually adequate tests, but fail when used together
D can cause some component in B to fail only if there is some dependency between them.
Dependencies occur for many reasons and so regression faults can occur in many ways. Question: how many can you think of?
There is no practical means to develop a test suite guaranteed to reveal all regression faults
(OO) Regression TestingRegression Faults: typical casesD has a side effect on B:
B fails because a new action of D is inconsistent with Bs requirements, assumptions, or contract with respect to D
(OO) Regression TestingRegression Faults: typical casesD is a client of B:
D sends a message that violates Bs invariant/precondition. B is not defensive and accepts the incoming message, leading to failure
(OO) Regression TestingRegression Faults: typical casesD is a server of B:
B sends a message to D. Ds postconsitions have changed (or contain bugs). D returns a value that causes a violation of Bs invariant . B fails or returns an invalid value to another baseline component C. C fails
(OO) Regression TestingRegression Faults: typical casesD is a revised superclass:
B is a client of E, a subclass of D. The interface of D has not changed but its behaviour has: and so the effective behaviour of E is different. This behaviour may introduce timing or nondeterministic problems which can lead to failure of B.
(OO) Regression TestingRegression Faults: typical casesD is a new subclass of a polymorphic class hierarchy that is not LSP (Liskov substitution principle ) compliant:
B is a client of D and expects the new subtype to be compatible. D does not comply leading to an inconsistent state and so when B attempts to use this object it fails.
(OO) Regression TestingRegression Faults: typical casesAn undesirable feature interaction occurs:
Bs functionality extended by change to/addition of D1Bs functionality extended by change to/addition of D2
B+D1 passes regression testsB+D2 passes regression tests
But B+D1+D2 fails regression testsNote: this is a very abstract view of feature composition (+)
(OO) Regression TestingRegression and Unit Testing: test frameworksA test framework provides reusable test functionality which:
Is easier to use (e.g. dont have to write the same code for each class)Is standardized and reusableProvides a base for regression testsA unit testing framework is unit & regression testing on steroids
(OO) Regression TestingRegression and Unit Testing: unit test frameworksEach class must be tested when it is developed
Each class needs a regression test
Regression tests need to have standard interfaces
Thus, we can build the regression test when building the class and have a better, more stable product for less work
(OO) Regression TestingAutomating with unit test frameworksThe real power of regression tests happens when they are automated: this requires they report pass/fail results in a standardized way
Can set up jobs to:
Clean & check out latest build treeRun testsPut results on a web page & send mail (if tests fail)
JUnit & ant have code to do all of this (You have already seen these?)
(OO) Regression TestingAutomating with unit test frameworksEffective automated RG requires:
Version controlModular structureSmart Means of comparing baseline and delta resultsShufflingAssertions for built-in test
(OO) Regression TestingAutomating with unit test frameworksEnvironmental factors can lead to less-than-identical test configuration:
Different OSsDifferent content in persistant storageSUT uses nondeterministic objectsSUT uses non-repeatable pseudorandom generation SUT is sensitive to timing differencesSUT is sensitive to non-controllable system inputSUT uses time sensitive code
(OO) Regression TestingLets learn about regression testing by looking at a practical example:A simple array of integer pairs is required that can be initialised (with a specified size) and can be randomized with random values within a certain range. It must also provide a method to report if the values are strictly increasing in both dimensions.
Check out the Source Code at the web site
IntPair.java, Compare.java, IntPairArray.java, TestIntPairArray.java
Is the following test suitable for regression testing? How can we improve things?