Automated test pattern Automated test pattern selection from requirementsselection from requirements

« Requested »« Requested »

Yves Le TraonYves Le Traon

Clémentine NebutClémentine Nebut

OutlineOutline

ObjectivesObjectives A contract language for requirementsA contract language for requirements Test generation from enhanced use Test generation from enhanced use

casescases ResultsResults ConclusionConclusion

ObjectivesObjectives

Benefiting from the sequential Benefiting from the sequential dependencies of the use cases …dependencies of the use cases …• Expressing the sequential Expressing the sequential

dependencies: a contract language for dependencies: a contract language for the requirementsthe requirements

… … To generate test objectives/test To generate test objectives/test patterns under the form of patterns under the form of sequences of use casessequences of use cases• Expressing test covering criteriaExpressing test covering criteria

Global methodGlobal methodRequirements

(use cases andcontracts)

Execution model(UCTS)

Interpreter

Simulator

Test objectivegenerator

Testcriterion

(functional)

Testcriterion

(robustness) Test patternsand verdicts

Test case generation« Mapper »

Set of test casesand their verdicts

Useful for model-checking or improving requirements

A contract language for A contract language for requirementsrequirements

Based on Catalysis and Meyer’s Based on Catalysis and Meyer’s «design by contract» approach«design by contract» approach

Contract = pre and post conditionsContract = pre and post conditions Dealing with parameters of the use Dealing with parameters of the use

casecase Easy to learnEasy to learn Making easy the expression of Making easy the expression of

constraints and dependencies on use constraints and dependencies on use casescases

Use case parametersUse case parameters

Actors or Business conceptsActors or Business concepts Used to represent the inputs of the Used to represent the inputs of the

use caseuse case Parameter type: enumerated Parameter type: enumerated

(enumeration only needed for the (enumeration only needed for the simulation)simulation)

Contracts: logical expression on Contracts: logical expression on predicatespredicates

Predicate = a name + set of typed Predicate = a name + set of typed parametersparameters• Ex: created(m:meeting)Ex: created(m:meeting)

manager(u:participant,m:meeting)manager(u:participant,m:meeting) Logical expressions: classical Logical expressions: classical

boolean operators (and, or, implies, boolean operators (and, or, implies, not) + quantifiers (forall, exists)not) + quantifiers (forall, exists)

Contracts: GrammarContracts: Grammar

BOOLEXPR -> DISJONCTIONDISJONCTION -> CONJONCTION (or CONJONCTION)*CONJONCTION -> UNARYEXPR (and UNARYEXPR)*UNARYEXPR -> ( BOOLEXPR ) | NEGATION | FORALL | IMPLIES |EXISTS | PREDICATE | DIFF | EQUALITYPREDICATE -> IDENT ( IDENT (, IDENT)* ) | IDENT() EQUALITY -> IDENT = IDENTDIFF -> IDENT /= IDENTNEGATION -> not BOOLEXPRFORALL -> forall ( LISTFORMALPARAMS ) { BOOLEXPR }EXISTS -> exists ( LISTFORMALPARAMS ) { BOOLEXPR }IMPLIES -> { BOOLEXPR } implies { BOOLEXPR }

Contracts example:the virtual Contracts example:the virtual meeting case studymeeting case study

Contracts: exampleContracts: example

#use case OPENUC open(u:participant;m:mtg) pre created(m) and moderator(u,m) and not closed(m) and not opened(m) and connected(u)post opened(m)

#use case CLOSEUC close(u:participant; m:mtg) pre opened(m) and moderator(u,m)post not opened(m) and closed(m) and forall(v:participant) {not entered(v,m) and not asked(v,m) and not speaker(v,m) }

OPEN(u1,m1);CLOSE(u1,m1) is a correct sequence

Test generation from enhanced Test generation from enhanced use casesuse cases

A use case transition systemA use case transition system• DefinitionDefinition• Building algorithmBuilding algorithm

Coverage criteriaCoverage criteria• 4 structural criteria4 structural criteria• 1 semantical criterion1 semantical criterion

Robustness testsRobustness tests

Use Case Transition System Use Case Transition System (UCTS) : definition(UCTS) : definition

Quadruple M=(Q,qQuadruple M=(Q,q00,A,,A,))• Q = non empty set of statesQ = non empty set of states

State = set of instantiated predicatesState = set of instantiated predicates

• qq0 0 = initial state= initial state

• A = alphabet of actionsA = alphabet of actions action= instantiated use caseaction= instantiated use case

Q x A x Q = transition functionQ x A x Q = transition function

UCTS: exampleUCTS: example

0 ->{moderator(p2, m1), connected(p1), connected(p2), created(m1), manager(p2, m1), opened(m1)}1->{moderator(p2, m1), connected(p1), connected(p2), created(m1), manager(p2, m1), opened(m1), entered(p2, m1)}2->{moderator(p2, m1), connected(p1), connected(p2), created(m1), manager(p2, m1), opened(m1), entered(p1, m1)}3->{moderator(p2, m1), connected(p1), connected(p2), created(m1), manager(p2, m1), closed(m1)}4->{moderator(p2, m1), connected(p1), connected(p2),created(m1), manager(p2, m1), opened(m1), entered(p2, m1), entered(p1, m1)}5->{moderator(p2, m1), connected(p1), connected(p2), created(m1), manager(p2, m1), opened(m1), entered(p2, m1), asked(p2, m1)}6->{moderator(p2, m1), connected(p1), connected(p2), created(m1), manager(p2, m1), opened(m1), entered(p1, m1), asked(p1, m1)}

Building algorithmBuilding algorithmalgorithmalgorithm buildUCLTS paramparam initState: STATE ; useCases : SET[ACTION]varvar result : UCLTS to_visit : STACK[STATE] currentState : STATE newState : STATEinitinit result.initialState := initState to_visit.push(initState)bodybody while (to_visit≠) { currentState := to_visit.pop ∀ uc in useCases | currentState ⇒uc.pre do newState := apply(currentState, uc) if newState ∉ result then result.Q := result.Q ∪ {newState} to_visit.push(newState) fi result. :=result. ∪{(currentState,uc,newState)} } } }

Test criteriaTest criteria

All edges criterionAll edges criterion All vertices criterionAll vertices criterion All instantiated use casesAll instantiated use cases All instantiated use cases and all All instantiated use cases and all

verticesvertices

All precondition termsAll precondition terms

Implementation of test criteriaImplementation of test criteria

Breadth first search of the UCTSBreadth first search of the UCTS To obtain short test objectivesTo obtain short test objectives Prototype: UCTBuilderPrototype: UCTBuilder

Robustness testsRobustness tests

Generate paths leading to an invalid Generate paths leading to an invalid application of the use caseapplication of the use case

Exercize correctly the system, then Exercize correctly the system, then make a non specified actionmake a non specified action

Criterion: similar to all precondition Criterion: similar to all precondition term (all the terms that makes the term (all the terms that makes the precondition fail)precondition fail)

DemonstrationDemonstrationofof

thetheRequestedRequestedprototypeprototype

Is system testing efficient ?Is system testing efficient ?

Dead code9%

Robustness code

(w.r.t. env)18%

Nominal code65%

Robustness code

(w.r.t.spec)8%

ResultsResultsCriterion # generated test

objectivesaverage size of the tests

All edges 13841 11

All vertices 769 10

All instantiated UC 50 5

AV-AIUC 819 10

All precondition terms

15 5

ResultsResults

Code covered by functional test cases

Code covered by robustness test cases

50

55

60

65

70

75

80

85

All edges All vertices All inst. UC All inst. UCand Alledges

Allprecondition

terms

% covered code

ResultsResults

0

4

8

12

16

20

All edges All vertices All inst. UC All inst. UCand Alledges

Allprecondition

terms

# covered satements

# test cases

# covered satements

# test cases

ConclusionConclusion

2 relevant test criteria2 relevant test criteria 1 relevant robustness test criterion1 relevant robustness test criterion Test objectives generated Test objectives generated

automatically from enhenced use automatically from enhenced use casescases

Prototype toolPrototype tool

Future tool supportFuture tool support(in a close future)(in a close future)

Integrated to ObjecteeringIntegrated to Objecteering Using OCL expressions Using OCL expressions Map test objectives into test cases Map test objectives into test cases

using scenarios (with contracts)using scenarios (with contracts)