Complex Test Suites Organization

Victor Kuliaminkuliamin@ispras.ruISP RAS, Moscow

2/16

Modern Software Complexity

Modern software systems Are huge and have a

lot of functions Have very complex

interfaces Are developed by

hundreds of people

System Year Size(MLOC)Windows 3.1 1992 3Windows NT 3.1 1993 6Windows NT 3.5 1994 10Windows 95 1995 15Windows NT 4.0 1996 16.5Solaris 7.0 1998 7.5Red Hat Linux 5.2 1998 12Debian 2.0 1998 25Windows 2000 2000 29Red Hat Linux 6.2 2000 17Sun StarOffice 5.2 2000 7.6Debian 2.2 2000 59Red Hat Linux 7.1 2001 30Windows XP 2002 40Red Hat Linux 8.0 2002 50Debian 3.0 2002 105Windows Vista Beta 1 2005 50Fedora Core 4 2005 76Debian 3.1 2005 216Mac OS X 10.4 2006 86

System Year DevTeam SizeWindows NT 3.1 1993 200Windows NT 3.5 1994 300Windows NT 4.0 1996 800Debian 1.2 1996 120 *Debian 2.0 1998 400 *Windows 2000 1999 1400Debian 2.2 2000 450 *Debian 3.0 2002 1000 *Debian 3.1 2005 1500 *

3/16

Test Suites Complexity

Scalability of testing techniques is strongly related

with test suite organization

System Size(MLOC) Test suite size(MLOC)Java 1.5 5 13 (JCK)

System Year TestTeam SizeWindows NT 3.1 1993 140Windows NT 3.5 1994 230Windows NT 4.0 1996 700 (0.9)Windows 2000 1999 1700 (1.2)

System Test CasesMS Word XP 35000Oracle 10i 100000Windows XP >2000000

4/16

Traditional Test Suite Organization

Set of test cases

Suitable for test suite size management Suitable for failure analysis

− Test suites for complex systems became huge and poorly manageable

− Many common parts and low reuse− Hard to modify

Test suite

5/16

Full Set of Problems

Test execution Selection the necessary tests Test suite configuration

Test result analysis Failure diagnostics Test adequacy reporting

Test creation and maintenance Explicit relations with requirements, testing goals, and

components under test Fight with complexity Reuse

6/16

Approaches

Attributing Configuration parameters Modularization

7/16

Attributing

Adding a set of test case attributes Very simple Provides multi-aspect tests

classification Provides links to requirements and

components under tests Can help to measure test

completeness

Test suite

1 ‘x’ 3F7/

tests/usr/t1

2 ‘A’ B41/

tests/sys/t2

3 ‘x’ 9E/

tests/sys/t3

8/16

Configuration Parameters

A set of parameters that control test

execution and test results assessment Provide additional flexibility, portability and

adaptability of test suite Help to check optional requirements May be assigned dynamically− May make test suite too complex

9/16

Modularization

Separation of modules

performing specific tasks Facilitates reuse Enhances maintainability Facilitates requirements

traceability− May complicate test suite− May complicate failure analysis

10/16

“Right” Modules

Test run-time supportSchedulerTimerEvent monitor

Test adapter Test data generator Test oracle

11/16

Test Adapters

Make possible reusable and abstract tests

Test SUTAdapter

12/16

Test Data Generators

Provide data for tests Can contain filters or

selectors Can be composed from

smaller data generators

13/16

Test Oracles

Provide verdict of SUT behavior correctness

Test Data Generator

System under Test

Test OraclePartial oracles

- Event or operation oracle- Data integrity constraint

Enhance reuse Simplify requirements traceability

?

14/16

Oracle Composition

Simple caseAny failure detected by partial oracle is global Complex data integrity constraints

Behavior of distributed systemsInterleaving semantics

✕

15/16

Conclusion

Test suite organization techniques improving testing scalability already exist

The most powerful one is modularization

But usage of these techniques is restricted by existing tools

All the techniques presented need proper support in tools

16/16

Thank you for attention!

Any questions?

kuliamin@ispras.ru