Department of CIS University of Pennsylvania 1/31/2001 Specification-based Protocol Testing Hyoung Seok Hong Oleg Sokolsky CSE 642

1/31/2001

Department of CISUniversity of Pennsylvania

Specification-based Protocol Testing

Hyoung Seok HongOleg Sokolsky

CSE 642

1/31/2001


Outline

• Introduction– Specifications, implementations– Conformance relations– Tests, testing architectures– Assumptions, etc.

• Formal protocol specifications• Finite-state machine (FSM) testing• Extended FSM (EFSM) testing• Test generation for StateCharts

1/31/2001


Testing problem

• A specification gives a description of the system behavior.

• An implementation should behave according to the specification.

• Is this really true?

c

a

b

a

b

a

b

1/31/2001


Black-box vs. White-box testing

• Black-box: unknown system structure– Testing against reference specification– Use the specification interface for

• test selection, coverage criteria, result analysis

• White-box: known system structure– Testing against reference specification– Use system structure for

• test selection, etc.

1/31/2001


Protocol testing architectures

TesterLocal

architecture

IUT

• IUT – implementation under test

PCO

• PCO – point of control and observation

1/31/2001


Protocol testing architectures

UpperTester

IUT

PCO1

PCO2

underlying service layer

LowerTester

DistributedarchitectureCoordinatedarchitecture

Remotearchitecture

1/31/2001


General testing architecture

IUT

PCO

PCO PCO

Tester

Tester Tester

1/31/2001


Tests and test suites

• A test case defines:– A finite sequence of inputs – A finite sequence of expected outputs

• A test suite is a collection of tests to achieve a certain test coverage

• Testing process– test generation: construct a test suite– test application: execute each test– result evaluation: interpret outputs

1/31/2001


Test structure

• Test purpose– e.g., identify a state of the

implementation

• Test preamble– lead the IUT into known state

• Test body– invoke behavior corresponding to the test

purpose

• Checking of test results

1/31/2001


Modeling approach• Testing criteria on a concrete system

are hard to formulate

implementationdetailed

specification?

conformancerelation

• Abstract models make reasoning easier

abstract modelof specification

abstract modelof implementation

assumptions andtest hypothesis

assumptions andtest hypothesis

1/31/2001


Outline

• Introduction• Formal protocol specifications

– Formal specification languages

• Finite-state machine (FSM) testing• Extended FSM (EFSM) testing• Test generation for StateCharts

1/31/2001


Formal specification languages

• Extended FSM– SDL– Estelle– StateCharts

• Process Algebras– LOTOS

• Algebraic Specification– Z, VDM

1/31/2001


Outline

• Introduction• Formal protocol specifications• Finite-state machine (FSM) testing

– FSM model– FSM fault models– FSM test selection

• Extended FSM (EFSM) testing• Test generation for StateCharts

1/31/2001


FSM model• A set of states• An initial state• Finite sets of input (X) and output (Y) events• Transfer function

x1

x2x3

• Output function/y1

/y2/y3

1/31/2001


FSM-based fault model

• Output faults• Transfer faults• Transfer faults with additional states• Additional or missing transitions• Control and data flow faults?

1/31/2001


Testing based on FSM models

• Testing methods based on control flow– Transition-tour (TT) method

• a single sequence of inputs to traverse all transitions

• simple but weak

– Unique-input-output (UIO) method• a test to identify each state in the specification

– Distinguishing sequence (DS) method• identify each state by the same test body• provides full coverage, but may not have a DS

1/31/2001


UIO testing method• Assumption: for each state, there is an input sequence that produces a unique output• Each test identifies one state

• s1 b/1• s2 a/0 a/0 b/1• s3 b/1 a/1

preamble body

s1 s2

s3

b/1

b/0

b/0

a/1

a/0a/0

1/31/2001


Outline

• Introduction• Formal protocol specifications• Finite-state machine (FSM) testing• Extended FSM (EFSM) testing

– EFSM model– EFSM fault models– EFSM test selection

• Test generation for StateCharts

1/31/2001


EFSM model• A set of states• An initial state• Finite sets of input (X) and output (Y) events

• Transition relationx1

x2

x3

• A finite set of variables (V)

– Guardsb1

b2

b3

{output y1}

{update v2}

{output y3}– Update blocks

1/31/2001


Tests for EFSM models

• EFSM executions depend on input signals and data

• A test consists of:– test sequence– test data

• Executability problem:– Find data to execute the test sequence

• Undecidable, in general

1/31/2001


• Executability problem introduces an additional step in the testing process:– test generation: construct a test suite– test application: execute each test– result evaluation: interpret outputs

Test validation

– test validation: keep only executable tests– test application: execute each test

– result evaluation: interpret outputs

1/31/2001


Control and data flow testing

• EFSM executions are data-dependent• Control flow FSM testing methods are

not adequate for EFSM models• Data flow testing methods account for

data dependencies

1/31/2001


Data flow: definitions and uses

• Data variables are – defined by inputs and updates (def)– used in

• updates (c-use)• guards (p-use)

• Data-flow graph captures data dependencies

1/31/2001


Data flow graph

• Directed graph with nodes labeled with definitions and uses of variables

v=0{input u}

u<0{v:=u+1}

p-use vdef u

p-use uc-use udef v

1/31/2001


Data-flow coverage criteria

• all-def– test suite covers each definition at least

once

• all-use– cover each def-use association at least

once

• all-du-paths– exercise all paths from each definition of

a variable to every its use.

1/31/2001


all-use coverage

• Find at least one definition-free path for every def-use association

def v

use v

no definitons

of v

1/31/2001


Outline

• Introduction• Formal protocol specifications• Finite-state machine (FSM) testing• Extended FSM (EFSM) testing• Test generation for Statecharts

1/31/2001


• EFSMs (FSMs + variables) + concurrency + hierarchy + communication + real-time

• Widely used for specifying real-time embedded HW/SW controllers

• Also used in most of object-oriented methodologies, e.g., UML

Statecharts

1/31/2001


A coffee vending machine

Statecharts = off

power-on/light-on; m:=0

power-off/light-off

Hierarchy +

notempty

emptyinc/m:=0

dec[m=1]/m:=0

inc/m:=m+1

dec[m>1]/m:=m-1

money

EFSMs +

busyidle

coffee

on

done/stop

Concurrency +

coffee[m>0]/start

Real-timetm(coffee,[3,5])/stop

Communication + coffee[m>0]/start;dec

1/31/2001


Test sequences for Statecharts

• The main purpose– What should implementations do?

•{power-on}/{light-on}, {inc}/{}, {coffee}/{start}

– What should not implementations do?•{power-on}/{light-on}, {coffee}/{}

• The main issue– How can we generate a finite and

reasonable number of test sequences?

1/31/2001


State coverage

off


power-off/light-off

notempty

emptyinc/m:=0

dec[m=1]/m:=0

inc/m:=m+1

dec[m>1]/m:=m-1

money

busyidle

coffee

on

done/stop

coffee[m>0]/start

tm(coffee,[3,5])/stop

coffee[m>0]/start;dec

1/31/2001


Transition coverage

off


power-off/light-off

notempty

emptyinc/m:=0

dec[m=1]/m:=0

inc/m:=m+1

dec[m>1]/m:=m-1

money

busyidle

coffee

on

done/stop

coffee[m>0]/start



1/31/2001


Data flow oriented coverage

off


power-off/light-off

notempty

emptyinc/m:=0

dec[m=1]/m:=0

inc/m:=m+1

dec[m>1]/m:=m-1

money

busyidle

coffee

on

done/stop

coffee[m>0]/start



1/31/2001


Test generation from Statecharts (I)

• Basic idea– Transforms Statecharts into EFSMs

• Advantage– Can reuse the existing methods and

tools for EFSMs– Can handle infinite state space

• Disadvantage– Cannot determine the executability of

test sequences

1/31/2001


1/31/2001


Test generation from Statecharts (II)

• Basic Idea– Test generation can be reduced into model

checking– TG: for each state s, is there a path leading to s– MC: a temporal logic formula EFs express this

property

• Advantage– Fully automatic– Generates only executable test sequences

• Disadvantages– Can handle only finite state space

1/31/2001


Another perspective

• Testing classes in object-oriented programs– Can we show an CLASS implementation

conforms to a Statechart specification?– Variables of Statecharts: data members of

classes– Events of Statecharts: member functions of

classes

• Testing inheritance in object-oriented programs– Can we reuse the test sequences of a super-

class for testing its subclasses ?

1/31/2001


A coffee-cocoa vending machine

off


power-off/light-off

busyidle

cocoa

on

notemptyempty

inc/m:=0

dec[m=1]/m:=0

inc/m:=m+1

dec[m>1]/m:=m-1

money


cocoa[m>0]/start;dec

busyidle

coffee



VendingMachine

CoffeeVendingMachine

CocoaVendingMachine

Coffee-Cocoa

VendingMachine

Documents

Department of CIS University of Pennsylvania 1/31/2001 Specification-based Protocol Testing Hyoung Seok Hong Oleg Sokolsky CSE 642