15. Regression testing

15. Regression testingTom Verheyen, Jelle Slowack, Bart Smets, Glenn Van Loon

2

Outline• Introduction

- What, why, when, how- Regression faults- Test automation- Test suite maintenance- Reducing a test suite

• Patterns- Test All- Test Risky Use Cases- Test by Profile- Test Changed Segments- Test Within Firewall

3

What is regressions testing?• Baseline version: component/system passed test suite• Delta version: component/system not passed regression

test• Delta build: executable configuration SUT containing

baseline and delta components• Regression test case: test case baseline passed, expected

to pass on delta build• Regression test suite: composition of regression test cases• Regression fault: revealed by test case that no longer

passes

4

Why should we regression test? Ariane 5

• First test flight: crash!- Data conversion from 64-bit

floating point to 16-bit signed integer

- Software (Ada) reused from Ariane 4

- No regression tests done• Estimate cost: $350 million -

$2,5 billion

http://en.wikipedia.org/wiki/File:Ariane_5_(mock-up).jpg

5

When can we use regression test? • What kind of errors can be found?

- Delta side effects- Delta - baseline incompatibilities- Undesirable feature interactions baseline - delta- Bad fixes

• IBM: bad fixes injection rates 2% - 20%

6

When should we run a regression test?

• When a new subclass has been developed• When a superclass is changed• When a server class is changed• When a bug fix has been completed• When a new system build has been generated• When a new increment is generated for system

scope integration testing• When a system has stabilized and the final release

build has been generated

7

How to run a regression test?• Basic procedure same for all situation

1. Remove broken test cases2. Choose a regression test suite (full/reduced:

see Regression patterns)3. Set up test configuration4. Run the regression test suite5. Do something with the results

8

Regression faults• Combination of baseline B and delta D fails

- D has side effect on B- C is client/server of B- D is incompatible with B (Ariane 5!)- D’s and B’s contract specifications differ- ...

(Extended list of errors: Binder p. 761-762)• Faults at intercomponent, subsystem and system

scope

9

Test automation• Manual testing is not a good thing

- Person must reenter it and judge result- If number of tests and time to rerun

increases:1. Rerun fewer baseline test cases +

focus on tests to validate final requirements2. Add more people to enter test cases and judge results

(usually not possible)- Time to ship decrease

# regressions tests approaches 0

10

Test automation: requirements• Effective automated RT requires capabilities:

- Version control, compare baseline–delta results, smart comparator, ...

- Want to know more? Binder p.764• Test environment should be controllable

- Some variable factors can result in less-than identical test configurations• Test environment itself, generator with clock as

seed, ...- Extended list of factors: Binder p.765

11

Test suite maintenance• A regression test suite can rapidly grow large• Test decay is inevitable

- Broken testcase, redundant test cases, …

• Example: aerospace manufacturer- Test suite of 165.000 test cases

• +/- 90% test cases: redundant• Documentation inadequate• Some parts not tested: 3000 new test cases

only 18.000 test cases remained

12

Test suite maintenance: procedure1. Run baseline test suite on delta build. Remove broken test

cases.2. Correct relevant bugs.3. Merge delta component scope tests with baseline test suite.4. Use a coverage analyzer.5. Rerun the test suite (to know coverage)6. Analyze tests with same entry-exit paths (still necessary?)7. If code is not tested: develop new test case.8. Rerun test suite again.9. Check revised baseline test suite as new baseline test suit.

13

Reducing a test suite• Even with maintance, tests can become too

large- Use reduced regression test,

with selected tests

• How to “safely” reduce a regression test?- Safe: all tests that could possibly exhibit different

output when run on the delta build

14

Reducing a test suite• 4 criteria

- Inclusiveness: percentage of BL tests that may show regression faults and are in the RT suite

• Safe RT suite is 100% inclusive- Precision: percentage BL tests in a RT that cannot reveal

regression faults and are not selected in the reduced RT suite• 100 test cases: 1 cannot reach changed code

99% precise• Precise RT suite: no tests that cannot produce different

output- Efficiency: cost of identifying a reduced RT suite- Generality: range of application for the selection strategy

15

Test patterns

1. Retest All: default2. Retest Risky Use Cases3. Retest by Profile4. Retest Changed Segments5. Retest Within Firewall

16

1. Retest All• Rerun entire baseline test suite• Context

- Any scope• Fault model

- Failures because of incompatibility, side effects, undesirable feature interaction

17

1. Retest All• Strategy

- Test model: baseline test suite is reused- Test procedure: rerun after removing broken tests- Oracle: should be same results as previous run- Automation: see slides 12-13

• Entry criteria- Delta components pass component scope testing- Suitable baseline test suite exists- Test environment has same configuration as previous

runs (see slide 13)

18

1. Retest All• Exit criteria

- No pass test cases reveal bugs that are acceptable

- Remaining test cases pass• Consequences

- Inclusiveness, precision, efficiency, generality- See further slides

19

2. Retest Risky Use Cases• Use risk-based metrics to select partial RT

suite• Context

- Too few time, personnel or equipement• Fault model

- Failures because of incompatibility, side effects, undesirable feature interaction

20

2. Retest Risky Use Cases• Strategy

- Test model: risk criteria to select test cases• Suspicious use cases: unstable, complex use cases• Critical use cases: necessary for safe, effective

operation- Test procedure: Identify, develop en run test suite- Oracle: should be same results as previous run- Automation: see slides 12-13

• Entry criteria- Same as “Retest All” pattern

21

2. Retest Risky Use Cases• Exit criteria

- Same as “Retest All” pattern • Consequences


22

3. Retest by profileIntent

- Partial regression test- Budget-constrained

Context- Too few time, personnel or equipement- Greatest dependability within budget?- Applicable at any scope

23

3. Retest by profileFault model

- Allocate tests by profile:frequency-based testing vs reliability

Strategy: Test model- # test cases for each use case? => total budget

24

3. Retest by profileExample• Total budget 6000 mins• Run 1 test case 5 mins• Chance on bug reveal 0.5 %• Bug fix requires 200 mins• Baseline test suite 20 000 test cases

How many of test cases (T) can we run?6000 = 5T + (0.005 x 200) 5=> T ≈ 5000

25

3. Retest by profileAllocation of Regression Test Time by Use Case frequency

Use case Probability Number of testsCash withdrawal 0.50 2500Deposit 0.25 1250Balance inquiry 0.25 1250

TOTAL 1.00 5000

26

3. Retest by profileStrategy: Test procedure

1. Identify, develop, run reduced test suite2. Verify: critical use cases & use case variants

included

Strategy: Oracle- should be same results as previous run

27

3. Retest by profileStrategy: Automation

- Slides 12-13- Automate selection in prev. example:

• 5 000 of 20 000 test cases=> ¼ test random selector(each test 25% chance of being selected)

Entry & exit criteria- Same as “Retest All” pattern

28

3. Retest by profileConsequences


29

4. Retest Changed CodeIntent

- Partial regression test- Code-change analysis

Context- Too few time, personnel or equipement- Applicable at class, cluster or subsystemscope

30

4. Retest Changed CodeFault model

- See pattern “Retest All”

Strategy: Test model- Select all baseline tests that have reached:

• A changed code segment• Or a deleted code segment

-> “Graph walk technique” (Rothermel and Harrold)

31

4. Retest Changed CodeStrategy: Test model (2)

- Basic model does NOT consider:• Inheritance• Dynamic binding• Data flow• Control flow• Other dependencies

arising from state-based behaviour, iteration or recursion

32

4. Retest Changed CodeStrategy: Test procedure

1. Obtain a report from coverage analyzer, that lists codesegments by testcase

Testcase CodesegmentsT1 B1,B3,B6,B7T2 B1,B4,B8T3 B2,B5

33


2. Extract pairs with each segment and testcase

Testcase CodesegmentsT1 B1,B3,B6,B7T2 B1,B4,B8T3 B2,B5

PAIRB1 T1B3 T1B6 T1B7 T1..B5 T3

34


3. Use a version control tool to generate a report on the changes between baseline & delta

Segment ChangeB1 SameB2 DeletedB3 Changed.. ..B9 New

35


4. Concatenate test (step 2) & change file (step 3). Sort!

ConcatB1 SameB1 T1B1 T2B2 DeletedB2 T3...

36


5. Selection rules for tests:• Tests under same skip• Tests under delete include• Tests under changed include• Tests under new skip

ConcatB1 SameB1 T1B1 T2B2 DeletedB2 T3...

37

4. Retest Changed CodeStrategy: Oracle

- Same as “Retest All” patternStrategy: Automation

- see slides 12-13Entry & exit criteria

- Same as “Retest All” patternConsequences

- See further slides

38

5. Retest Within FirewallIntent

- Partial regression test- Code-dependency analysis

Context- Too few time, personnel or equipement- Applicable at class, cluster or subsystemscope

39

5. Retest Within FirewallFault model

- See pattern “Retest All”

Strategy: Test Model- Firewall = set of components whose test cases will be

included in a regression test- Firewall set is identified by analysis of changes to each

component in the SUT and its dependencies.

40

5. Retest Within FirewallStrategy: Test Model (2)• Each pair of components (A,B) is analyzed• Either or both may be changed:

1. Contract changealters external interface and/or externally visible contract(eg. alteration of public methods, ..)

2. Implementation changeother changes that are not visible to clients

41

5. Retest Within FirewallStrategy: Test Model (3)• Dependency relationship between each pair is used to

select test cases

• 4 Relationships:1. B uses A2. B is a subclass of A3. B overrides A4. B is a server of A

42

5. Retest Within FirewallStrategy: Test Model (4)• For each relationship: baseline testcase that apply to A,B

or AB may be reused in one of 4 ways:Level 0 No testcases can be rerunLevel 1 The state setup and test messages can be rerun.

Expected results must be redeveloped.The sequence of testcases may need to be

reworked.Level 2 The state setup, test messages and expected results

can be rerun.The sequence of testcases may need to be

reworked.Level 3 The test cases can be rerun as is.

43

5. Retest Within FirewallStrategy: Test Model (5)• Example

- Class Account (A) and class Money (B)- Account uses a variable amount of type Money- Implementation change to account

- Relationship: Money is a server of Account - ..

44

5. Retest Within FirewallChanges Retesting requirement

A B Dependency A ..... ... ... ... ...Implementation None B is server of A 2... ... ... ... ..

Þ Level 2 reuse of the test cases for Account

(decision table for selecting regression tests 15.7 p 791)

45

5. Retest Within FirewallStrategy: Test procedure1. Develop a dependency matrix2. Apply the decision rules (table 15.7)3. Rerun tests according to the testing levels

46

5. Retest Within FirewallStrategy: Oracle

- Same as “Retest All” patternStrategy: Automation

- see slides 12-13Entry & exit criteria

- Same as “Retest All” patternConsequences

- See next slide!

47

Pattern ComparisonInclusivenessAll: Safe, all tests selectedRisky: Unsafe, selection of testcases not by

analysis / dependenciesProfile: Unsafe, same as Risky Use CaseChanged:Safe, all baseline tests that can produce a

different result are selectedFirewall: Safe

48

Pattern ComparisonPrecisionAll: no tests are skipped, least preciseRisky: some tests that could be skippedProfile: dittoChanged: few tests that could be skipped

most precise of the white box partial regression strategies

Firewall: few tests that could be skipped

49

Pattern ComparisonEfficiencyAll: lowest analysis & setup cost

high run costRisky: time and cost are constrained (budget)

selection based on use cases (<-> implementation)

=> analysis can be done without code analyzers / technical knowledge of SUT

Profile: Same as Risky Use Casebut if the operational profile is established, then

low cost of selection analysis

50

Pattern ComparisonEfficiency (2)

Changed:High setup costRun cost = size of deltasCost can be greater than Retest All: reselection

Firewall: Highest setup costRun cost = size of firewall

51

Pattern ComparisonGeneralityAll: applicable in any circumstances

easy to set upRisky: nearly any circumstances, any scopeProfile: many circumstances,

system scope with accurate operational profile

Changed:applicable at class scoperequires skills in code analysis

Firewall: applicable at cluster scoperequires skills in code analysis

Documents

15. Regression testing