1 Jan Tretmans [email protected] Embedded Systems Institute Eindhoven Radboud University Nijmegen Model-Based Testing

1

Jan Tretmans

[email protected]

Embedded Systems Institute

Eindhoven

Radboud UniversityNijmegen

Model-Based Testing

2 © Jan Tretmans

Model Based TestingContents

Model-based testing What and when

Formal model-based testing Testing versus verification Testing as a decision procedure Test assumption, implementation relation, tests

Model-based testing with transition systems Implementation relation ioco Test generation for ioco

Test Tools On-the-fly testing

Conclusions The future, perspectives

3 © Jan Tretmans







4 © Jan Tretmans

Towards Model Based Testing

Increase in complexity, and quest for higher quality software

testing effort grows exponentially with complexity

testing cannot keep pace with development

More abstraction

less detail

model based development

Checking quality

practice: testing - ad hoc, too late, expensive, lot of time

research: formal verification - proofs, model checking, . . . .

with disappointing practical impact

5 © Jan Tretmans

Towards Model Based Testing

Model based testing has potential to combine

practice - testing

theory - formal methods

Model Based Testing :

testing with respect to a (formal) model / specification

state model, pre/post, CSP, Promela, UML, Spec#, . . . .

promises better, faster, cheaper testing:

algorithmic generation of tests and test oracles : tools

formal and unambiguous basis for testing

measuring the completeness of tests

maintenance of tests through model modification

6 © Jan Tretmans

Automated Model Based TestingAutomat

edModel Based

Testing

modelspec

IUT

IUTconftospec

TTCNTTCNtestcase

s

pass fail

testtool

testgeneration

tool

testexecution

tool

IUT passes tests

IUT confto spec

soundexhaustiv

e

7 © Jan Tretmans

A Formal Development Process

informalrequirements

specification

realization

design

code

formalizablevalidation

formalverification

testingformal

informal world

formal world

real world

8 © Jan Tretmans







9 © Jan Tretmans

formal world

real world

Formal Verification

specifications

implementationi

model m

of i

sat

m modcheck s

m sat s

completesound

modelchecker

m modcheck s

m modcheck s

assumption:m is valid model of i

sat

10 © Jan Tretmans

formal world

real world

Formal Testing

specifications

implementationi

confto

IUT

IUT passes Ts

IUT confto s

correctness of

test generation ?

IUT passes Ts

IUT fails Ts

test

execution

test

generation

test suite TS

11 © Jan Tretmans

Correctness of Formal Testing

IUT confto s IUT passes Ts ?

iIUT imp s iIUT passes' Ts iIUT MOD .

Implies completeness of testing as a decision procedureBased on test assumption

sound

exhaustive

12 © Jan Tretmans

Formal Testing : Test Assumption

Test assumption :

IUT . iIUT MOD.

t TEST . IUT passes t iIUT passes t

IUT iIUT

test t test t

13 © Jan Tretmans

Testing Addition

Test a function adding numbers of two dice:

int dadd ( int x, y ) for x,y [1…6]

Is the following a complete test suite?

(1,1) (1,2) . . . . . (1,6)(2,1) (2,2) . . . . . (2,6). . .. . .. . .(6,1) (6,2) . . . . . (6,6)

14 © Jan Tretmans

Testing Addition

Test a function adding numbers of two dice:

int dadd ( int x, y ) for x,y [1…6]

The test suite (1,1) (1,2) . . . . . (1,6)(2,1) (2,2) . . . . . (2,6). . .. . .(6,1) (6,2) . . . . . (6,6)

is sound & exhaustive if

imp is such that it requires correct addition for x,y [1…6] only;

the test assumption is that implementations

can be modelled as functions : MOD = intintint

15 © Jan Tretmans

Formal Testing

passes : TEST MODS

{pass,fail}

gen : SPEC (TEST)

Ts TEST

s SPEC

impProof soundness and exhaustivess:

iMOD . i passes Ts . i imp s

Test hypothesis :

IUTIMP . iIUT MOD .

tTEST. IUT passes tiIUTpasses t

pass / fail

Then:

IUT is confto-correct

IUT passes the

test

iIUT MODS

16 © Jan Tretmans

Approaches to Formal Testing

Finite State Machine

Pre/post-conditions

Labelled Transition Systems

Programs as Functions

Abstract Data Type testing

. . . . . . .



17 © Jan Tretmans

Testing Propertiesof Sequential Input/Output

Programs

Specification: property over x and y property(x,y) = x 0 |y y - x|

Implementation is function i :: X Y

Test set T X Tools like GST and QuickCheck generate thousands of tests

by systematic traversal of all values of type X

But still: what is a "good" set ?

IUT

i(x) = x

x: real

pre: x 0

y: real

post: |y y - x|

18 © Jan Tretmans

Approaches to Formal Testing

Finite State Machine

Pre/post-conditions



Abstract Data Type testing

. . . . . . .



19 © Jan Tretmans







20 © Jan Tretmans

Formal Testing with Transition Systems

exec : TESTS IMPS (OBS)

gen : LTS (TTS)

Ts TTS

s LTS

IUT IMPS

ioco

iIUT IOTS

passes : IOTS TTS

{pass,fail

}

Proof soundness and exhaustiveness:

iIOTS .

( tgen(s) . i passes t )

i ioco s

Test hypothesis :

IUTIMP . iIUT IOTS .

tTTS . IUT passes t iIUT passes t

pass / fail

21 © Jan Tretmans

Specifications:Labelled Transition Systems

Labelled Transition System S, L, T, s0

?coin

?button

!alarm ?button

!coffee

states

actions transitionsT S (L{}) S

initial states0 S

22 © Jan Tretmans

Models of Implementations:Input-Output Transition Systems

LI = { ?dub, ?kwart }

LU = { !coffee, !tea }

Input-Output Transition Systems

IOTS (LI , LU ) LTS (LI LU )

IOTS is LTS with Input-Outputand always enabled inputs:

for all states s,

for all inputs ?a LI :

?dub?kwart

?dub?kwart

?dub?kwart

?dub?kwart

?dub

!coffee

?kwart

!tea

S ?a

23 © Jan Tretmans

i ioco s =def Straces (s) : out (i after ) out (s after )

Implementation Relation ioco

Correctness expressed by ioco IOTS LTS :

p p = !x LU {} . p !x

out ( P ) = { !x LU | p !x , pP } { | p p, pP }

Straces ( s ) = { (L{})* | s }p after = { p’ | p p’ }

24 © Jan Tretmans



Correctness expressed by ioco IOTS LTS :

Intuition:

i ioco-conforms to s, iff

• if i produces output x after trace , then s can produce x after

• if i cannot produce any output after trace , then s cannot produce any output after ( quiescence )

25 © Jan Tretmans

Genealogy of ioco

Labelled Transition SystemsLabelled Transition Systems

IOTS (IOA, IOSM, IOLTS)

IOTS (IOA, IOSM, IOLTS)

Testing Equivalences(Preorders)

Testing Equivalences(Preorders)

Refusal Equivalence(Preorder)

Refusal Equivalence(Preorder)

Canonical Testerconf

Canonical Testerconf

Quiescent Trace PreorderQuiescent Trace Preorder

Repetitive QuiescentTrace Preorder

(Suspension Preorder)

Repetitive QuiescentTrace Preorder

(Suspension Preorder)

iocoioco

ioconfioconf

26 © Jan Tretmans

Implementation Relation iocoout ( i after )

=

out ( i after ?dub )

=

out ( i after ?dub.?dub )

=

out ( i after ?dub.!coffee)

=

out ( i after ?kwart )

=

out ( i after !coffee )

=

out ( i after ?dub.!tea )

=

out ( i after )

=

!coffee

?dub

?dub?kwart

?dub?kwart

i?kwart

{ }

{ !

coffee }

{ !

coffee }

{ }

{ }

{ }


27 © Jan Tretmans

!coffee

?dub

?dub

?dub

i

!coffee

?dub

s

!tea

out (i after ?dub) = { !coffee } out (s after ?dub) = { !coffee, !tea }

ioco



28 © Jan Tretmans

!coffee

?dub

s

?dub

?dub

!coffee

?dub

i

!tea

?dub

out (i after ?dub) = { !coffee, !tea }

out (s after ?dub) = { !coffee}

ioco



29 © Jan Tretmans

ioco

?dub

?dub?kwart

!coffee

?kwarti

!tea !coffee

?dub

s

out (i after ?dub) = { !coffee }

out (i after ?kwart) = { !tea }

out (s after ?dub) = { !coffee }out (s after ?kwart) =

But ?kwart Straces ( s )



30 © Jan Tretmans

!coffee

?dub

i?kwart

?dub?kwart

?dub?kwart

out (s after ?kwart) = { !tea }out (i after ?kwart) = { }

ioco



s?dub

!coffee

?kwart

!tea

31 © Jan Tretmans

out (s after ?dub) = { !coffee }out (i after ?dub) = { , !coffee }

ioco

?dub

?dub

?dub

!coffee

?dubi

?dub

!coffee

?dub

s



32 © Jan Tretmans

out (s after ?dub) = { , !coffee }out (i after ?dub) = { , !coffee }

ioco

!coffee

?dub

s



?dub

?dub

?dub

!coffee

?dubi

?dub

33 © Jan Tretmans

out (i after ?dub.?dub) = out (s after ?dub.?dub) = { !tea, !coffee }

i ioco s


i ioco s =def Straces (s) : out (i after ) out (s after ) i

?dub

?dub

?dub ?dub

!tea

?dub

?dub

!coffee

?dub

s

!coffee

?dub

?dub

?dub ?dub

!tea

?dub

?dub

?dub

?dub

!tea

s ioco i

out (i after ?dub.?dub) = { !coffee } out (s after ?dub..?dub) = { !tea, !coffee }

34 © Jan Tretmans

Test Cases

labels in L { } tree-structured finite, deterministic final states pass and fail

from each state pass, fail : either one input !a

or all outputs ?x and

?coffee

!dub

!kwart

?tea

?coffee?tea

!dub

pass

failfail

failpass

Test case t TTS

TTS - Test Transition System :

35 © Jan Tretmans

ioco Test Generation


out (s after )

= { !x, !y, }

s

!x !y

i

!x

!z

out (i after )

= { !x, !z, }

out (test after )

= LU { }

pass

pass fail

test

?x

?y ?z

pass

36 © Jan Tretmans

Algorithm

To generate a test case from transition system specification s0

compute T(S), with S a set of states, and initially S = s0 after ;

1 end test case

pass

For T(S), apply the following recursively, non-deterministically:

2 supply input

!a

T( S after ?a )

ioco Test Generation Algorithm

allowed outputs or : !x out ( S

)

forbidden outputs or : !y out ( S )

3 observe output

fail

T ( S after !x )

fail

allowed outputsforbidden outputs ?y

?x

37 © Jan Tretmans

?coffee

failpass

?tea?choc

failfail

Test Generation Example

test

?coffee

failfail

?tea

?choc

pass

!dub

s

!tea

?dub

!coffee

38 © Jan Tretmans

Test Execution Example

Two test runs :

t i dub tea pass i'

fail i''t i dub choc i fails t

!choc

?dub

!tea

i

?coffee

failpass

?tea?choc

failfail

test

?coffee

failfail

?tea

?choc

pass

!dub

i' i''

39 © Jan Tretmans

?coffee ?tea

passfail fail

?coffee

passfail

?tea

Test Generation Example

s

?dub

!coffee

?dub

test

!dub

40 © Jan Tretmans

Test Execution Example

Two test runs :

t i dub pass i'

pass i't i dub coffee i passes t

?coffee ?tea

passfail fail

?coffee

passfail

?tea

test

!dub

i?dub

!coffee

?dub

?dub

?dub

?dub

41 © Jan Tretmans

Completenessof ioco Test Generation

For every test t generated with algorithm we have:

Soundness :t will never fail with correct implementation

i ioco s implies i passes t

Exhaustiveness :each incorrect implementation can be detectedwith a generated test t

i ioco s implies t : i fails t

42 © Jan Tretmans

Formal Testing with Transition Systems

exec : TESTS IMPS (OBS)

gen : LTS (TTS)

Ts TTS

s LTS

IUT IMPS

ioco

iIUT IOTS

passes : IOTS TTS

{pass,fail

}

Proof soundness and exhaustiveness:

iIOTS .

( tgen(s) . i passes t )

i ioco s

Test hypothesis :

IUTIMP . iIUT IOTS .

tTTS . IUT passes t iIUT passes t

pass / fail

43 © Jan Tretmans

Compositional TestingComponent Based Testing

i1

i2 s2

s1

ioco

i1 ioco s1

i2 ioco s2

s1 || s2i1 || i2

If s1, s2 input enabled - s1, s2 IOTS - then ioco is preserved !

44 © Jan Tretmans







45 © Jan Tretmans

Some Model BasedTesting Approaches and Tools

Agatha Agedis Autolink Cooper Gst Gotcha Leirios Phact/The Kit QuickCheck RT-Tester SaMsTaG

Spec#/SpecExplorer Statemate MAGNUM ATG STG TestGen (Stirling) TestGen (INT) TestComposer TGV TorX T-Uppaal Tveda . . . . . .

TorX

46 © Jan Tretmans

A Tool for Transition Systems Testing: TorX

On-the-fly test generation and test execution

Implementation relation: ioco Mainly applicable to reactive systems / state based systems;

specification languages: LOTOS, Promela, FSP, Automata

TorX IUTobserveoutput

offerinput

nextinput

specification

checkoutput

passfailinconclusive

user:manualautomatic

47 © Jan Tretmans

TorX Case Study Lessons

model construction difficult: missing or bad specs leads to detection of design errors not supported by integrated environment yet research on test based modelling

adapter/test environment development is cumbersome specific for each system

longer and more flexible tests full automation : test generation + execution + analysis

no notion of test selection or specification coverage yet only random coverage or user guided test purposes

48 © Jan Tretmans

? money

? button1 ? button2

! coffee! tea

test case

fail

! money

! button2

? tea

fail

? coffee

pass

n: int

[ n 35 ] -> [ n 50 ] ->

with data model

and time and hybrid

c := 0

c < 10 c < 15

[ c 5 ] ->

c := 0

d Vt / dt =

3d Vc / dt =

2

Vc := 0

[Vc = 10 ] ->

Vt := 0

[Vt = 15 ] ->

?

coin1

?

coin3

?

coin2and actionrefinement

?

Testing Transition Systems: StatusExtensions

49 © Jan Tretmans

Concluding

Testing can be formal, too (M.-C. Gaudel, TACAS'95)

Testing shall be formal, too

A test generation algorithm is not just another algorithm :

Proof of soundness and exhaustiveness

Definition of test assumption and implementation relation

For labelled transition systems :

ioco for expressing conformance between imp and spec

a sound and exhaustive test generation algorithm

tools generating and executing tests:

TGV, TestGen, Agedis, TorX, . . . .

50 © Jan Tretmans

Model based formal testing can improve the testing process :

model is precise and unambiguous basis for testing

design errors found during validation of model

longer, cheaper, more flexible, and provably correct tests

easier test maintenance and regression testing

Automatic test generation and execution

full automation : test generation + execution + analysis

Extra effort of modelling compensated by better tests

Perspectives

Documents

1 Jan Tretmans [email protected] Embedded Systems Institute Eindhoven Radboud University Nijmegen Model-Based Testing