Access the Next Generation of Software Testing with Conformiq Software

Lars Sonckin kaari 16, 02600 Espoo Finland, Tel +358 10 286 6300 Fax +358 10 286 6309 Web www.conformiq.com Conformiq Software Ltd.

Testaus ja testauksen automatisointi

The Next Generation of Testing

Testing is up to 50% of Software Development Cost Today

Developing the System Testing the System

• Sparse tool offering• Lacking integration and

system testing tools• Becoming industry focus

Cost

50%

• Major focus in the 90’s• Multitude of methods

and tool suites• Gave process focus to

SW industry

Definition of QA and Testing

o QA = ”All actions taken to ensure that standards and procedures are adhered to and that delivered products or services meet requirements”

o Testing = ”Testing is the process of executing a program with the intent of finding errors”

Different Types of Testing

o Unit testingo Module testingo Integration testingo System testingo Functional testingo Load testingo Stress testingo Performance testingo Acceptance testing

o White-box testingo Glass-box testing o Black-box testingo Mutation testingo Usability testingo Conformance testingo Protocol testingo Interface testingo GUI testing

IT Trends over the years

o HW & OS intensive programming 60’s & 70’sLow level languagesBatch processing

o Application programming 80’sApplication independent from operating systemStandalone applications

o System development 90’sMore extensive systemsClient-server architecture

o System integration & process automation 00’sSystems become integral part of business processesWeb-based systems, distributed architecturesMultivendor - multifunctionality networks

The Four Eras of Testing

I

II

III

IV

Test Management & Tailored tools

Test Coverage & White-box testing

Automated test generation &Specification Coverage

Regression & Black-box testing

Evolution of Testing Automation

o I Methodology Era Requirement traceabilityInvariant proving & In-house tools

o II White-box EraMcCabe’s static analysisUnit testing tools, test beds & code coverage

o III Regression testing EraGUI testingscript taping & execution environments

o IV Automated test generationSpecification based black-box testingAlgorithmic test generation Specification based coverage metrics

Evolution Analogy from Electronics

o Methodology Era 50’s & 60’sDesign principles

o Measurement / Tester Tool Era 70’sPhysical measurement / tester configurations

o Design Simulator Era 80’sDesign functionality simulation before implementation

o Automated Desing Verification 90’sAutomated formal design correctness proving before chip implementation

Problems in Current Testing Practice

Time consuming & Costly

Maintenance

Relevance

Coverage

Documentation & Reporting

Conformance testing

• Up to 50% of development effort• Bottleneck in time-to-market, delayed product releases• Poor product quality => troubleshooting, customer badwill

• Test scripts are larger the system under test• Maintenance problematic and costly• Difficult to introduce new people into maintenance tasks

• Human testers are not always systematic super professionals• Script writing is error prone• Human brain cannot work out system concurrency

• Tests coverage uncontrolled and low• No transparency for testing effort• No testing metrics related to system functionality

• Test planning and structure poor• Proper test scripts massive to manage / understand• Reporting manual

• There is no reliable means to prove that new component can replace old version in larger system

The Four Eras of Testing

I

II

III

IV

Test Management & Tailored tools

Test Coverage & White-box testing

Regression & Black-box testing

We are hereAutomated test generation &

Specification Coverage

The Challenge in Software Testing

Testing requirement

Number of required testsGap between testingrequirement and current situation

Scalability of currenttesting practices

System ComplexityToday

Normal Practice Today

Test Scripteawef(asdfa, asdf, )

awefaweef

awefafaweaeawef(asdfa, asdf, )

sdasdwe(oas)endsfsdfSdf

Manual Testing

Script Execution

Tape & Play

Automated Test Generation

Manual Testing

TestGenerator

Engine

On-the-Flytest execution

Test Scripteawef(asdfa, asdf, )

awefaweef

awefafaweaeawef(asdfa, asdf, )

sdasdwe(oas)endsfsdfsdf

Conformiq Concept

Industry Model + Generator Tool

Script Execution

The Next Generation Solution

Model maintenance is simpleRetape / Manual script editing

Maintenance /Regression

Can doNo can doConformance

Self documentingRead the scriptsDocumentation / Reporting

Full coverageHope You tape wellCoverage

10-1000 x relevant testsHope You tape wellRelevance

1 model replaces massive scriptsTape scriptsTime & Cost

Conformiq solutionCurrent solutionChallenge

Next generation paradigm

Automatic testing – on-the-fly mode

ON-THE-FLY TESTINGTester

Systemadapter

12 FF 02 44 56

19 AB 5B 00 11

F0 12 11 00 FF

A0 FF B0 15 93

PASS

specification

Testing workstation Implementation under test

Choose system to test Tester connects to the SUTCreate or obtain formal specification … chooses and executes a test runInstall adapter layer, connect … compares behavior to the specificationStart tester … announces test verdict

Automatic testing – batch test suites

BATCH MODE TESTING

Implementation under test

Tester

Scriptrun-time

Script collectionPASS

specification

Testing workstation

Choose system to test Tester generates batch test suiteCreate or obtain formal specification Start test suite executionInstall script run-time Run-time environment executes scripts…Start tester … which compute the test verdict

How to deploy specification-based testing

VERIFY

VERIFY VERIFY

VERIFY

System model

Conformance test

Test

Test

Test Test

Incrementaltest model

Proof of Success Emerging

o ”Ten-fold productivity level improvement in telecom & avionics industries in reaching aimed quality level”

(Rosaria & Robinson, Information and software technology, 2000, Nro: 42)

o ”Test cases revealed numerous defects that were not exposed by traditional approaches”(Dalal & al, Proceedings of ICSE, 1999)

antti.laine@conformiq.comTel +358 40 751 2307

Lars Sonckin kaari 16, 02600 EspooTel +358 10 286 6300Fax +358 10 286 6309www.conformiq.com