VII. Conformance Testing Methodology€¦ · 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Test Types Behavior tests Capability tests Basic ii Dept. Electrical

521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology

VII. Conformance Testing Methodology

Dept. Electrical & Information Engineering, Computer Engineering Lab 7-1 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-1 Dr. Junzhao Sun


Outline

General conceptsTesting documentsAbstract test methodsAbstract test suitesTest realizationConformance assessment process



Outline




What is Conformance Testing?

The process of testing the extent to which implementations of protocol entities adhere to the requirements stated in the relevant standard or specificationstandard or specification

Concerned with external behavior (black box)Prerequisites for interoperability

The primary objective of conformance testing is to increase the probability that different product implementations actually interoperate.

Testing of implementationsBut NOT with performance, reliability, fault tolerance, efficiency, etc.


y y

ISO 9646:A conforming implementation is one which satisfies both static and dynamic conformance requirements, consistent with the capabilities stated in the Protocol Implementation Conformance Statement (PICS)


Standard

ISO/IEC 9646 (ITU-T X.290)Conformance Testing Methodology and Framework (CTMF) for the Implementations of OSI and ITU ProtocolsImplementations of OSI and ITU Protocols

7 part standardISO 9646-1. General ConceptsISO 9646-2. Abstract Test Suite SpecificationISO 9646-3. Tree and Tabular Combined Notation (TTCN)ISO 9646-4. Test RealizationISO 9646 5 Conformance Assessment Process


ISO 9646-5. Conformance Assessment ProcessISO 9646-6. Protocol Profile Test SpecificationISO 9646-7. Implementation Conformance Statement


Conformance Testing Limitations

Aspects of protocol testing not considered in ISO 9646Interoperability testing, performance testing and robustness testingTh li ti f i i itiThe realization of service primitivesInternal aspects of the system implementationThe system environment and testing toolsProcurement proceduresTest certificates, and accreditation of test labs

Conformance testing will always be limited, for technical and theoretical as well as for economic reasons


theoretical as well as for economic reasonsNo guarantee can be given that an implementation absolutely conforms to the relevant protocol standardsConformance testing with positive test results will increase the degree of confidence in a protocol implementation


Basic Standard vs. Profile

Base standard is a standard relative to a single protocol in a given OSI layerA profile is a set of one or more base standards necessary forA profile is a set of one or more base standards necessary for accomplishing a particular function

a multilayer stack of several adjacent base standardsFunction might be a type of application, or a transport service based on specific telecommunications techniquesA protocol can be a base standard by itself, or can belong to a profileThe purpose of defining a profile is to facilitate interoperability

P fil i t f ld


Profile is twofold Multiplicity: a profile may retain options in a set of base specificationsRefinement: a profile restricts the choices of options in the base specifications. It may also specify additional conformance requirements.

We will mainly concentrate on the conformance testing of protocol


Conformance Testing Process (1/2)

Test generationTo generate the abstract test suites (ATSs) manually or automaticallyDevelopment of proformas for implementation conformance statementsDevelopment of proformas for implementation conformance statements (ICS) and implementation extra information for testing (IXIT)

Test realizationThe realization of test tools and means of testing, capable of executing particular test suites To generate the executable test suites (ETSs) with ATSs

Including both selection and parameterization process

Test execution


Test executionPerform test to an implementation with executable test case selected

Conformance assessment processPerformed at a test laboratory for a client, with a protocol conformance test report (PCTR) as the ultimate result


Conformance Testing Process (2/2)

Abstract test suiteStandard protocol

specification

Test generationTTCN

System Under

Executable test suite

specification

Protocol implementation

Test realization

Test execution


Under Test

IUT Tester Verdict: (result)• accept• failure• inconclusive• error (in test HW)


Test types in testing processBasic interconnection tests (BIT)

To determine whether interconnection is possible at all as an initial stage

Test Types

To determine whether interconnection is possible at all as an initial stageTo establish main features of the protocol are implementedBasic interconnection tests are standardized but may be omitted

Capability testsTo validate that all the the observable external static capabilities of the IUT are valid with respect to the static conformance requirements expressed in the PICSCapabilities tests are standardized and mandatory

Behavior tests


To check dynamic conformance of an IUT with respect to the dynamic conformance requirements specified in the protocol standardBehaviour tests are standardized and mandatory

Conformance resolution testsSpecial tests for purposes beyond ordinary behavior testing e.g. exceptionConformance resolution tests are not standardized and not mandatory


Test Types

Behavior tests

Capability tests

Basic i i


interaction tests Conformance

resolution tests


Test Results

Test resultsPass

Observed test outcome shows that the test purpose has been fulfilled and theObserved test outcome shows that the test purpose has been fulfilled and the IUT has displayed only valid behaviors as specified in protocol standards

FailTest outcome proves that some conformance requirement has been violated

InconclusiveNeither pass nor fail can be assigned to the test outcomeHappens when underlying provider signals problems e.g. disconnect or reset

Documentation of test results


SCTR = system conformance test reportPCTR = protocol conformance test reports

Requirements on test resultsRepeatable, comparable and auditable to a reasonable degree


Outline




Test Generation

TPsSt nd rd ATS

Standardized (Protocol) Implementation Conformance Statement (tables)

Standardized Test Purposes (in English)

TPsStandard

Standardized Abstract Test Suite (in TTCN)

ATS


ICSStandardized (Protocol) Implementation eXtra Information for Testing (tables)

IXIT


Test Specification to Be Produced



Conformance Requirements

Systems are called conforming if their externally visible behavior fulfils conformance requirements stated in protocol specification

M d t i t l h t b b dMandatory requirements, always have to be observedConditional requirements, have to be observed under conditions defined

Dependence on previous choices of parameter values and optionsOptional requirements, can be freely selected

Static conformance requirementsSpecify permitted combinations of capabilities implemented in real systemE g value ranges for size parameters or timers as in ICS


E.g. value ranges for size parameters or timers, as in ICS

Dynamic conformance requirementsSpecify the observable behavior of implementation permitted by standardDefine actual protocol e.g. PDU exchanges, timing, format of PDUs.

Profile is based on base specifications with additional constraints


ICS (1/2)

ICS = Implementation Conformance StatementA checklist of the capabilities supported by the IUT

A ti i l t d b li t t t f i tA questionnaires completed by client to present conformance requirementsProduced by filling out an ICS proforma with static requirementsProvides an overview of the features and options that are implementedUsed to select and parameterise test cases and as an indicator for basicinteroperability between different products

PICS = Protocol Implementation Conformance StatementIUT realizes a single protocol


Profile ICSIUT contains a set of interrelated protocols, e.g. a stack of protocols

ICS is not just for testingUsed by specification writer to clarify the intent of standardsUsed by product implementer for selection of features


ICS (2/2) RL = Requirements List

Standardization organizations Client

Profile ICSBase

specificationBase specificationBase

specificationBase specification

PICS-proformaPICS-

proformaPICS-proformaPICS-

proforma

PICSPICSPICSPICS

Profile ICS


Profile specification

Profile RL

Profile specific ICS proforma

Profile RL

Profile specific ICS


ICS Proforma – Protocol Capability

Status of capability in protocol specificationMandatory, optional , conditional

hibit d l d dprohibited, excludedout-of scope, not applicable

Capabilities can be grouped into categoriesMajor capabilitiesSupported PDUs and fields within PDUsSupported timersE di i ti


Encoding variationsNegotiation mechanismsError procedures


ICS Proforma – Structure

ICS proforma corrigenda (version information)ICS proforma instructions, abbreviations and terms

Identification of the implementation, supplier and client

Major capabilities table

PICS proforma tables

Identification of the implementation, supplier and clientIdentification of the protocol, information object or profile

Protocol, information object or profile corrigenda

Supported negotiation Supported Supported

PDUs TableSupported Supported


Multi-specification dependencies table

Supported protocol error handling table

capabilities tabletimer tables

Supported parameters table in PDUSupported parameters

table in PDUSupported parameters table in PDU

PDUs TablePDUs TableSupported

PDUs Table


ICS Proforma – ISDN ExampleIndex Protocol capability

Does the implementation support…Conditionsfor status

Status Reference Support

M1 3.1kHz audio calls O.1 6.1.2 [ ] Yes[ ][ ] No

M2 Data calls O.1 6.1.1 [ ] Yes[ ] No

M3 Packet mode calls O 6.3 [ ] Yes[ ] No

M4 Interworking with non-ISDN networks M 8.4 [ ] Yes[ ] No


M41 Calling line identification presentation M 8.4 [ ] Yes[ ] No

M421 Calling line identification restriction M1not M1

MN/A

8.4 [ ] Yes[ ] No

M422 Advice of charge M1not M1

MN/A

8.4 [ ] Yes[ ] No


IXIT, SCS

IXIT = Implementation eXtra Information for TestingContains additional information beyond specification as ICS

A IXIT f fill d t b t t l b t d li tAn IXIT proforma filled out by test laboratory and clientWith both protocol IXIT and profile IXIT, as in ICS typesSpecify additional testing information related to an specification

Physical setup, connection of test, and testing environmentSUT hardware, socket, telephone numbers, and tester SAP addressesTest-specific timer or parameter values

SCS = System Conformance StatementLi t f t ifi ti f th IUT l t d ICS l t d t t


List references to specifications of the IUT, related ICSs, related testreports, together with configuration information of SUT, I.e. an index

Conformance of an IUT: An IUT is said to be conforming if itsatisfied the static and dynamic requirements of the relevantstandards referenced in SCS and capabilities in ICSs.


IXIT Proforma – ISDN Example

#PC_BASIC TRUE

#PC POINT TO POINT FALSE#PC_POINT_TO_POINT FALSE

#PC_POINT_TO_MULTIPOINT TRUE

#PC_EN_BLOC_RECEIVING TRUE

#PC_OVERLAP_RECEIVING TRUE

#PX_CDPN_OCTET3 ’81’O


#PX_CPN_PTC1 ‘323830313531’O

#PX_LCPN_PTC2 ‘07’O

#PX_CPN_PTC2 ‘323830313532’O


TSS & TP

TSS = Test Suite StructureTP = Test PurposesTSS & TP are derived from the relevant base standards.

Provide an informal, easy-to-read description of each test caseConcentrate on the meaning of the test rather than detailing how it may beachievedTP is a textual description of a single requirement or a set of relatedrequirements which should be testedTest Purposes are grouped into a logical Test Suite Structure according to


Test Purposes are grouped into a logical Test Suite Structure according tosuitable criteria, e.g., basic interconnection, error handling, functionality etc.

Will be introduced later in ATS section


ATS & ETS

ATS = Abstract Test SuiteThe entire collection of Test Cases (ABSTRACT)E h T t C ifi th d t il d di f th T t P ittEach Test Case specifies the detailed coding of the Test Purposes, writtenin the standardized test specification language TTCNCan be generated manually from protocol and service standardCan be generated automatically from formal specification of protocol

CATG = Computer-Aided Test Case GenerationSome tools are commercially available to generate test suite from e.g. SDL

Will be studied later in more detail


ETS = Executable Test SuiteTest cases for truly test executionCan be quickly and easily implemented from the ATS using the TTCNcompilers available on most modern test tool platforms


Test Suites & Specification

Tests apply to a single system under test (SUT), and one protocol standardsExhaustive test of standards specificationsExhaustive test of standards specifications

Must cover all features – mandatory & optionalMust cover valid, invalid, & inopportune protocol exchanges

Use of standardized test specification mayImprove transparency and increase of the objectiveness for the test processProvide comparability of test resultsAs the criteria for accreditation bodies to assess test laboratories


As the criteria for accreditation bodies to assess test laboratoriesCommon test specifications are the basis for harmonized test servicesProvide a common starting point to suppliers, users, test laboratories and certification bodies

Test specification language – TTCN


Test Specification Development in ETSI



Typical Test Suites

GSM, GPRS, 3G (UMTS in 3GPP)Broadband Radio Access Networks (HiperLAN)DECT(Digital Enhanced Cordless Telecommunications)BluetoothLocal Loop, Digital Subscriber Line (DSL)IN, INAP (Intelligent Network Application Protocol)ISDN, B-ISDNSMS i PSTN/ISDN


SMS in PSTN/ISDNSIP (Session Initiation Protocol), IPv6IP CablecomXML (Extensible Markup Language)


Outline




ATM

ATM = Abstract Test MethodsNOT Automatic Teller MachineNOT Asynchronous Transfer ModeNOT Asynchronous Transfer Mode

A set of ATMs are defined by OSI testing methodology, describing Abstract test configurationPossible types of observation and controlProviding for test coordination

ATMs are determined byConfiguration of IUT with respect to SUT


Entire testing configuration, I.e. test system + SUT + service providerWhere tester can control and observe the IUTDifference degrees of precision of the test coordination procedures

Common test architecture and 4 test methods


OSI System & IUT

Complete system Partial systemApplication relay

systemNetwork relay

system

Transport

Application

PresentationSession

Data link

Network

Transport

Application

PresentationSession

Data link

Network

Transport

Application

PresentationSession

Data link

Network

Transport

Application

PresentationSession

Data link

Network


IUT implement one or a combination of OSI protocols from one layer or several adjacent layers

Data link

Physical

Data link

Physical

Data link

Physical

Data link

Physical


OSI Terminology

Layer S Layer SSP

Session layer

Layer T Layer T

Service provided

Service used

PDU

SP

SPSAP

Transport layer


Layer N Layer N

SP : Service PrimitiveSAP : Service Access PointPDU : Protocol Data Unit

Network layer


Conformance Testing Terminology

Upper Tester

S

IUT

ASP

ASP

ASP

ASP

PCO

PCO

ASP: Abstract Service Primitive

PCO: Point of Control and Observation

IUT: Implementation Under Test

PDU: Protocol Data Unit

PDU


Lower Tester

The PCO has two FIFO queues:• Send (from tester to IUT)• Receive (by tester from IUT )

A PCO maps to a SAP (Service Access Point) in the OSI reference model


Conceptual Test Architecture (1/3)

System Under Test (SUT)Test System (TS)Test operator interface Product operator interface

IUT

ASPs

PCOLower Tester

Test Co-ordinationProcedures (TCP)

PDU

Lower Tester Control Function

Lower TesterLower Tester

TCP

Upper TesterUpper TesterUpper Tester


ASPs

PCO

Lower level service provider

There can be several LTs and UTs being simultaneously used



Testing contextsSingle-party testing is for testing an IUT which communicates with exactly one real open system represented by a single lower testerone real open system, represented by a single lower testerMulti-party testing is for testing an IUT communicating with multiple real open systems, represented by more than one lower tester

Configuration for IUT components can be homogeneous or heterogeneous

Lower tester (LT) controls and observes the IUT’s lower service boundary, indirectly, via the underlying service provider

In single-party testing, behaves as the peer entity to IUT


g p y g, p yIn multi-party testing, act as peer entities working in parallel

Lower tester control function (LTCF) coordinating all LTsAssign the test case verdictsMandatory in multi-party context, inapplicable in single-party context



Upper tester (UT) controls and observes IUT’s upper service boundary, by operator access, API, or hardware interface

I i l t t t UT b h f IUTIn single-party context, UT behaves as a user of IUTIn multi-party context, UTs working in parallel act as users of IUT

Test coordination procedures (TCPs) are used to ensure cooperation between the UTs and LTs

How tester shall respond Passing (preliminary) results S h i ti


SynchronisationTCP is NOT Transport Control Protocol, as in TCP/IP


ATM Classification

ATMs for multi-party testingSeveral parallel upper and lower testersI l it ti t t t l f ti (UTCF) i d dIn complex situation a upper tester control function (UTCF) is neededSpecial cases include only one upper tester, or even no upper tester at all

ATMs for single-party testingLocal Test Method (L)

Upper Tester and Lower Tester in Test SystemDistributed Test Method (D)

Upper Tester in SUT Lower Tester in Test System


Upper Tester in SUT, Lower Tester in Test SystemCo-ordinated Test Method (C)

As above, uses Test Management ProtocolRemote Test Method (R)

Lower Tester in Test System, no Upper Tester


Test Case

Lower Tester IUT

ConnectRequest “CR”

Upper Tester

ConnectIndication “CONind”ConnectConfirm “CC” ConnectResponse “CONresp”


Recall service primitivesRequestIndicationResponseConfirm


Local Test Method

UT

LT

N-layer entityunder test

TCPcoordination

ASPs

ASPs


• Upper Tester is located in Test System• Requires an upper interface on IUT• IUT is “built in” the tester• No ATSs for this method• Good for the testing of a hardware component• Example: Ethernet driver


Local Test Method

System Under Test (SUT) Test System (TS)

Upper Tester

IUT

ASPs

PCO


Lower Tester

ASPs

TCPs

2) CONind

3) CONresp


ASPs

PCO1) CR4) CC


Distributed Test Method

UTTCP

Service provider

IUT

UT

LT ASPs

ASPs

PDUs


Service provider

• UT in SUT, LT remote• Requires synchronisation• Suitable for upper layer protocols / protocols offering API• Example: socket communication


Distributed Test Method


Upper Tester

IUT

PCO


Lower Tester

ASPs

Test Co-ordinationProcedures (TCP)

2) CONind 3) CONresp


PCO1) CR4) CC


Co-ordinated Test Method

UTTM-PDUs

Service provider

IUT

UTLT

ASPs

PDUs


• UT in SUT but no access, LT remote• No assumption of upper interface to the IUT• Use only one PCO below the LT• Uses Test Management Protocol (TMP) embedded in ASPs• Suitable for mid-layer protocols


Co-ordinated Test Method

Upper Tester

IUT


Lower Tester

ASPs

Test ManagementProtocol

2) CONind 3) CONresp


PCO1) CR4) CC


Remote Test Method

TCP

Service provider

IUT

UTLT

ASPs

PDUs


Service provider

• No Upper Tester• Upper Tester can be native application or a user accessible interface• Manual co-ordination• Limited, but easy to use


Remote Test Method


IUT Lower Tester

ASPs


PCO1) CR2) CC


ATMs Put Together



Testing Configuration Examples – MHS

MHS = Message Handling Systems Multi-user in single-party contextRTS R li bl T f S t

UT-2

RTS = Reliable Transfer System

UT-1LT MHS-ASPs

TCP


RT service provider

IUTRTS-ASPs

MHS-PDUs


Testing Configuration Examples – Multiplexing

Multiplexing happens in transport or network layer protocols where more than one connection is multiplexed over a single lower connection e g transport connections over network one

UT-2

lower connection, e.g. transport connections over network one

UT-1LT Transport-

ASPsNW

TCP


Service provider

IUTNetwork-ASPs

NW-PDUs


Testing Configuration Examples – Relay

Relay systems’ testing should be in multi-party context, without using UTs

LT –1PDUs

TCP

LT –2PDUs


Service provider

IUTASPs

Service provider

ASPs


A Example Test System Architecture

Protocol Tester SUT

IUT

Interface

TCEditor

PICS，PIXITEditor

Result Analysis&report

TCTranslator

TestSelect


Service Provider

TCLibrary

TestExecute

Transmit Receive


Outline




ATS Development Process

ATS = Abstract Test Suites1. Determine the testable conformance requirements from the relevant specifications ICS and related documentsspecifications, ICS and related documents2. Determine the test groups & objectives that cover these requirements3. Develop test purposes covering requirements and test group objectives4. Choose testing context (single-/multi-party) and an ATM (configuration)

A test suite is always dedicated to a single test method5. Specify ATS, including test steps, using standardized notation like TTCN6. Specify the selection and parameterisation criteria with respect to the ICS


p y p pand IXIT proformas

ATS should be used to derive ETS by compilation and adaptation to the means of testing (MOT)

MOT is the combination of test devices, equipments and procedures


Recall the Test Generation Process

TPsSt nd rd ATS

Standardized (Protocol) Implementation Conformance Statement (tables)

Standardized Test Purposes (in English)

TPsStandard

Standardized Abstract Test Suite (in TTCN)

ATS


ICSStandardized (Protocol) Implementation eXtra Information for Testing (tables)

IXIT


ATS Structure (1/2)

T t Test group

Test suite

Test group

Test caseTest caseTest case Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest case

Test group

Test groupTest group

Test caseTest caseTest caseTest case

T t t


Test step

Test caseTest caseTest event Test caseTest caseTest caseTest eventTest caseTest caseTest caseTest event

Test step

Test stepTest step

Test caseTest caseTest caseTest event


ATS Structure (2/2)

A test suite consists of test cases with no execution orderTest cases are logically grouped into test group according to test purposes

E h t t h i di id l t t t d t t t i b h iEach test case has an individual test purpose to demonstrate a certain behaviorA test group objective may be provided for each test groupThe nested hierarchy should be created in a top-down fashion

Ensure systematic derivation of test purposes Achieve good coverage of all protocol aspects and conformance requirements

A test case may be modularized into nesting test stepsModularization facilitates test case development and maintenance


Modularization facilitates test case development and maintenanceGroup commonly used test steps together into test step libraries

Economical that they need not be specified repeatedly

Test event is the elementary unit within a test caseUsually the exchange of a PDU or an ASP at a PCO


TSS = Test Suite Structure

“which tests are performed on which aspects of the reference specification”Adopt a top down approach deriving TSS from base specificationAdopt a top down approach deriving TSS from base specificationwhen no technical constraint forbids it, structure TSS as a tree1. The name of base specification should be placed at the top of the TSS2. Test groups related to the "major functions", or "roles" of the base

specification3. Pre-defined groups of tests according to the "nature" of the tests:

Basic interconnection tests; capability tests; V lid b h i t t i lid b h i t t i t b h i t t


Valid behaviour tests; invalid behaviour tests; inopportune behaviour tests;4. Pre-defined groups of tests according to the "functional aspect" tested:

State event transitions; parameter variations; parameter combinations; timers5. Groupings relevant to the base specification

E.g. name of functional units, then of PDUs


TSS FTAM Example

FTAM = File Transfer Access and Management



Test Purposes (1/2)

Each test case with one test purpose for meaning descriptionTest purposes focus on one or more related conformance requirements, derived from OSI specificationTest purposes and their hierarchical grouping are specified in a TSS & TP (Test suite structure and test purposes) standard

Structure chosen depends largely on personal experience and style

Test purposes for a test suite have to cover aspects of protocolCapabilities, in particular mandatory and optional ones


Behavior tests, with the tester displaying valid/invalid behaviorPDUs sent to/received from IUT, and relationship between the PDUsProtocol phases and protocol machine statesTiming, encoding, individual parameters and parameter combinations


Test Purposes (2/2)

The expression of a TP shall not be limited to checking a state or a state transition of a protocol state machine

E l TP "Ch k th t th IUT d COMMON CONTROL ACK ftExample TP: "Check that the IUT sends a COMMON CONTROL ACK after receipt of a COMMON CONTROL message containing ... , as a result of state transition S1 → S2.".

A clear explanation of the relation between TPs and the conformance requirements in the reference specification shall be providedThe TPs shall be designated by the same numbering and


The TPs shall be designated by the same numbering and referencing scheme as the TSS


Combined TPs (CTPs)

The final cost of a test campaign directly depends on the size of the ATS

The objective of CTP is to reduce the number of test cases, but to keep a good coverage



TP Combination Principle



TSS for CTP



Test Purpose Component

Initial condition should define precisely the expected state of the IUT at the beginning of the test

Example: Ensure that the IUT is in state connection established (reception of a connect confirm on the LT on response to a connect request).

Check(s) to be performed


Example: Ensure that the IUT, on reception of a valid SETUP message with the Sending complete information element sends any of a CALL PROCEEDING, ALERTING or CONNECT message.

Verdict criteria should explicitly describe the observable events which will be used to assess the result of the test.

Example: Check that the IUT in state S1, on receipt of a valid connect request, enters state S2 (sends a valid connect confirm).


Testing States

Testing state is a combination of the states of SUT (IUT), test system, and underlying provider which can occur during testing

St bl t t b i t i d b t t t ith t t t ti itiStable state can be maintained between test case without tester activitiesIdle state is that no connection is established, independent of previous tests

Test state is the one that test case is truly performed to check test purposeInitial state the start point of the execution of a test case

End state is the stop state of a testOne test case may stop at several end states



Test Case Scheme

A test case is conceptually composed of 3 partsTest body, the combination of test events for achieving test purpose

St t i i iti l t ti t tStarts in a initial testing stateTest preamble, takes the test from stable states to initial stateTest postamble, takes the test from end states to one stable state

If end state is not unique, it has to be checked by a verification step

Postamble


The outcome for each test case should be assigned a verdictCalculated from verdicts assigned to test events in the outcome

End State (Verification)

End State (Test Body)

Stable State

Test State

Preamble Test Body Verification


Outline




Test Realization

Test realization is the process of producing a means of testing(MOT) for conformance to protocol standards, by reference to an ATS standardATS standard.

MOT is a real test system used to execute a test suite on IUTsCombination of equipment and procedures that derive, select, parameterize and execute the test cases of a given ATS, together with produce conformance log

Does not mean the actual execution of test cases

ETS derivation is to convert the ATS into final ETSSelection of applicable test cases (abstract or executable)


Selection of applicable test cases (abstract or executable)Performed mainly on the basis of ICS (Imple. Conformance Statement)

Parameterization of selected test cases (abstract or executable)Performed mainly on the basis of IXIT (Imple. eXtra Info for Testing)

Derivation process, see next


Derivation Process

One-to-one correspondence between selected abstract &

ATS Derivation ETS

executable test casesCorrespondence in all respects e.g. behavior, test purpose, verdicts, relation to ICS/IXIT entries, selectability, independency, etc.May include some MOT

SATS

Selection

PATS

Parameterization

Selection

Parameterization

PETS

SETSDerivation


yspecific selectionIndependent from and compatible with the selection & parameterization processes Derivation

& execution

PATS

Execution

PETSDerivation


MOT Functions

Realization of configured ATM (abstract test method)Realize the LTs (lower testers) and LTCF (LT control function)

LT’ d /d d li ti i l b t t ffi i tLT’s encoder/decoder realization: simple but not efficientLT’s reference protocol implementation: efficient but difficult

Specify or realize UTs (upper testers)UT can be dedicated software, human operator, existing protocol layers, etc.

Specify or realize TCP (test coordination procedures)

ATS to ETSContain derived executable test cases


Contain derived executable test casesProvide means for selection and parameterizationProvide means for executing PETS

Conformance log generationMust be human readable and permit verification of test verdicts assigned


Outline




Conformance Assessment Process (1/2)

Conformance assessment process is the entire activities that are necessary to validate and document the protocol conformance

T t i i th t l ti f th t t itTest campaign is the actual execution of the test suite

3 phases for conformance assessment process1. Preparation for testing

Production of SCS, ICS & IXITChoice of ATM & ATSPreparation of SUT & MOT

2 Test operations


2. Test operationsStatic conformance review of ICSTest realization (selection & parameterization)Test campaigns

3. Test report productionProvides clients with documentation


Conformance Assessment Process (2/2)

OSI specifications,proformas,

ifi i

Preparation for testing•administrative activities

l i f dtest specifications •completion of documents•choice of test method and ATS•choice of test specifications•SUT and MOT preparation

SCS, IXIT, ICS, PTS, ATS

static conformance

Test Operation•static conformance review•test selection & parameterizationt t i


static conformance review results,

test results, test log

PCTRSCTR

•test campaigns

Test Report Production


Test Laboratories and Test Clients

Test Laboratory is responsible for conducting the conformance assessment of IUTs and can be classified according to their role

Fi t t i t d b th li f IUT I i l t dFirst-party is operated by the supplier of IUT, I.e. implementer, vendorSecond-party is affiliated to the user of IUT who is willing to test itThird-party is independent from either of the two whose business is the testing of products

Clients will frequently be implementers or vendors of protocol implementation who want to have their own products tested and


implementation who want to have their own products tested and certified, but can also be important user of IUT


Preparations for Testing (1/2)

Production of SCS, ICSBy client filling proformas

Production of IXITBy lab & client filling proformas

Choice of ATM & ATSAgreement between client & lab

Preparation of SUTBy client


Preparation of MOTBy lab


Preparations for Testing (2/2)

SUT decs

SUT config.

ICSIXIT MOT

config.

Extra decs


ATSICS pro. IXIT

pro.


Testing Operations (1/3)

Static conformance reviewChecks ICS for consistency with static conformance requirements

Test selectionSelect a suitable subset of test cases from ATS resulting SATS

Test parameterizationParameters are assigned values as specified in ICS & IXITs resulting PATSTransform PATS into PETS (parameterized executable test suite)

Verification of TCPs


To verity MOT & SUT are able to use TCP (Test Coordination Procedure)

Test campaign, see next slideExecution of PETS and production of conformance log



Test campaignAssignment of verdicts = [pass, fail, inconclusive]

T t l b h t i tl t t di t t d t tTest lab has to assign exactly one test verdict per executed test caseAbnormal test termination or error leads to “not run” in report“Inconclusive” verdict have to be repeated at least once

Conformance logOutcome of all executed test cases to document for client

Negotiated exitTime when client and test lab jointly decide to stop the test campaign


No formal test report will be produced


ETS


Selection

Derivation

Parameterization

SUT config.

ICSIXIT MOT

config.

SUT decs

Extra decs

SUT config.SETS


ATSICS pro. IXIT

pro. PETS Execution


Test Report Production (1/2)

ETS Selection

SUT config.

ICSIXIT MOT

config.Derivation

SUT decs

Extra decs

SUT config.SETS

Parameterization


ATSICS pro. IXIT

pro. PETS ExecutionComp. Report


Test Report Production (2/2)

SCRT = System Conformance Test ReportGives an overall summary of the conformance status to the SUTC t i f hi d t t lt & d i i t ti i f tiContains a summary of achieved test results & administrative information

PCTR = Protocol Conformance Test ReportOne for each protocol under testAdministrative and technical informationA summary of all achieved test resultsIndividual detailed presentation of each test case



SCTR Proforma

1. Identification summaryAdministrative information for identification of SCTRI f ti b t t t l b t t t li t SUT fil tInformation about test laboratory, test client, SUT, profile, etc.

2. System report summaryOne section for each tested protocol is requiredDocument reference

IUT definition, protocol specification, ICSs, IXITs, PCTR, ATS, ATM, etc.Conformance status

Whether static and/or dynamic conformance error


Whether static and/or dynamic conformance errorTest cases run

Number of passed, failed, and inconclusive test cases


PCTR Proforma

1. Identification summaryInformation about PCTR, IUT, and testing environment

2 Protocol conformance summary2. Protocol conformance summaryIUT conformance status

“This IUT has/has not been shown by conformance assessment to be non-conforming to the referenced base specification(s)”

Static conformance summary“The ICSs for this IUT is/is not consistent with the static conformance requirements in the referenced base specification”

Dynamic conformance summary


y yThe test campaign did/did not reveal errors in the IUT

3. Protocol conformance reportStatic conformance review reportTest campaign report, for each test case in ATS

ATS reference, selected or not, run or not, verdict, and observations


Test Conducting

Formal tests Done by test laboratories that will issue official test certificate

Informal testsIn-house or private lab product testTests at trade shows, bake-off meetsRemote test – access to tester through Internet (e.g. on Web server)

Quality of TestsConsistency and repeatability


Standard ATSs & approved ETSsStandard of test laboratory operations

Test Report for procurement and regulatory bodiesNational and International recognition of test results


Detailed Testing Process



Whole Information Flow

ETS Selection

SUT config.

ICSIXIT

SETSMOT

config.Derivation

SUT decs

Extra decs Parameterization


ATSICS pro. IXIT

pro. PETSComp. ReportExecution


Summary

Conformance testingBlack-box approach of testing protocol implementations against protocol specificationspecification

Three phases for conformance assessment process1. Preparation for testing

Test generation: from ICS+IXIT & TPs, get ATS• Document proforma

Configuration of ATM architecture• 4 methods


2. Test operationsTest realization: from ATS, through selection & parameterization, get PETSTest execution

3. Test report productionSCTR & PCTRs


References of This Chapter

Bernd Baumgarten and Alfred Giessler, OSI Conformance Testing Methodology and TTCN, Elsevier Science, 1994


Documents

VII. Conformance Testing Methodology€¦ · 521265A Telecommunication Software Ch7 – Conformance Testing Methodology Test Types Behavior tests Capability tests Basic ii Dept. Electrical