Overview of ETSI Testing Methodology Anthony Wiles Manager ETSI Protocol and Testing Competence Centre

Overview of ETSI Testing Methodology

Anthony Wiles

Manager

ETSI Protocol and Testing Competence Centre

ETSI Testing Methodology - SINTESIO Seminar 2006 2

ETSIEuropean Telecommunications

Standards Institute

Sophia Antipolis, France

The home of ICT standardization

http://www.etsi.org


Interoperability is …

The ultimate aim of ICT standardisation The red thread running through the entire standards

development process, it’s not an isolated issue Not something to be somehow fixed at the end

The ability of systems and products to interwork is fundamental

ETSI approach ETSI produces with the required degree of parallelism

• Base Standards and • Test Specifications

Base standards should be designed with interoperability in mind Profiles designed to reduce potential non-interoperability Protocol conformance and Interoperability statements Two complementary forms of testing

• Conformance testing• Interoperability testing (formal IOT)

Testing provides vital feedback into the standardization work


Poor Interoperability is Expensive

Bad publicity in trade magazines Embarrassment for the manufacturer Annoyance of the end customer

In the past, interoperability failures meant:

Today, interoperability failures in the field means:

Front page headlines in the Financial Times Fall in manufacturers stock price Loss of investor confidence Unrecoverable damage to brand name Irretrievable loss of customers

We can no longer afford to get it wrong! We can no longer afford to get it wrong!


Specification and Testing is Part of ETSI’s Strategy (to get it right!)

ETSI membership believes in methodical testing A good example

GSM/UMTS is growing by 1 Million users per day! Can you imagine what would happen if users were

experiencing interoperability problems ? More than 1 Million Euros invested per year for writing

formal test specifications We have experienced what it means to get it right!

A bad example We have not always got it right! For GPRS we did not take a formal approach to testing First market products did not interoperate

We have learnt that testing is not cheap, but it pays in the long run


Technical Committee MTS (Methods for Testing and Specification)Development of methodologies, techniques and languages (including TTCN3)

• http://portal.etsi.org

ETSI’s Protocol and Testing Competence Centre (PTCC)Supports ETSI committees on the application of formal techniques in standards on a daily basisDevelopment of test specifications (conformance and interop)

• http://portal.etsi.org/ptcc/

ETSI Plugtests™ ServiceValidation of standards and prototypes through interoperability events

• http://www.etsi.org/plugtests

ETSI Resources for Interoperability


How Does the PTCC Help?

Assist ETSI Technical Bodies on the use of state of the art techniques for Specification, validation and testing Good working practices (Standards Engineering)

Pragmatic and flexible approach Based on experience

Help to develop usable methodologies For ETSI’s current and emerging needs

Knowledge transfer Quality through Continuity


Who PTCC Supports

Technical Bodies (TB) Technical Committees ETSI Projects Partnership Projects etc.

Chairmen, Rapporteurs, Individuals Working Groups (WG) STFs (Specialist Task Forces)

Experts are seconded from the ETSI membership• 25-30 experts in 2006

Produce test specifications (including validation) Short projects

• e.g., 2 man-months maintenance of VoIP tests Long projects

• e.g., UMTS testing 58 man-months per year over 3-5 years 15 – 20 PTCC STFs per year Responsible for approx. 2,7M€ of expert resource per year


Typical Standards Development Process in ETSI

(Unit) Conformance Testing

Interoperability Testing

Products mature from prototypes to commercial products

ETSI: Development of base standards

Certification

Industry

time

Conformance Test Specifications

Interoperability Test SpecificationsIterative feedback

Iterative feedback


PTCC Support for Specification Techniques

PTCC provides support to the ETSI TBs on the use of modern specification techniques in ETSI deliverables: UML (Unified Modeling Language) ASN.1 (Abstract Syntax Notation) XML (Extensible Markup Language) MSC (Message Sequence Charts) SDL (Specification and Description Language) Etc.

TTCN (Test and Test Control Notation)


Different Kinds of ETSI Test Specifications

Conformance Robustness

PerformanceInteroperability

Network Integration

RF/EMC


Typical ETSI PTCC Areas of Testing

Cellular: GSM and 3G (UMTS) terminals Now includes early IMS

WiFi: HiperMAN, HiperACCESS, WiMax

VoIP: H.323, SIP, SIGTRAN

Service Creation: OSA/Parlay (API, IDL)

IPv6: Core, Security, Mobility, v4-v6

Cordless phones: DECT

Radio communications: TETRA, DMR

Access terminals: FSK, SMS

Broadband: ISDN, DSL

Smartcards: Readers, cards, security modules

Intelligent Transport Systems (ITS): DSRC

Early NGN

Future: More Security, more NGN, GRID ...


Progressive Testing

All engineering disciplines accept that testing should be applied to progressively complex units From individual components to entire systems

This demands a number of different testing solutions There is no silver bullet

In the two extremes Just testing the components is not enough Just testing the system is not enough

Limitations are economic, not technical What can I afford to test? What can I afford not to test?

What should be covered by standardised test specifications What is the right kind of testing for the job

What kind of ‘thing’ is being tested? Device? System? Service? What is the most economic method(s)? What resources are (or must be made) available?

• Tools, test scripts, testbeds etc.


Conformance Testing

Is Black-Box testing Stimulation and Response

Tester Device

Point of Control and Observation (PCO)

Reqs.


Conformance Testing Tests is Usually Layer-byLayer

lat

ii

1 2 3

4 5 67 8 9

* 8 #

latigid

TestSystem

SUT API

MMI

L3

L2

PHY

SUT = System Under Test

IUT = Implementation Under Test

L3

Upper Tester

Lower Tester


Characteristics of Conformance Testing (1)

Is unit testing Tests a single ‘part’ of a device (e.g., a protocol layer)

• Often incrementally – layer-by-layer

Tests against well-specified requirements For conformance to the requirements in a base

specification or profile (standard) Usually limited to one requirement per test

Tests at a 'low' level At the protocol (message/behaviour) level (bits)

• Or service primitive, API, Interface

Tests over standardised interfaces May not be available to ‘normal’ user

Requires a test system (and executable test cases) Can be expensive (e.g., radio-based test system) Tests under ideal conditions


Characteristics of Conformance Testing (2)

High control and observability Means we can explicitly test error behaviour Can provoke and test non-normal (but legitimate)

scenarios Can be extended to include robustness tests

It is usually automated and tests are repeatable Conformance Testing is DEEP and NARROW

Thorough and accurate but limited in scope Gives a high-level of confidence that key components of

a device or system are working as they were specified and designed to do


Limitations of Conformance Testing

Does not prove end-to-end functionality (interoperability) between communicating systems Conformance tested implementations may still not interoperate

• This is often a specification problem rather than a testing problem! Need minimum requirements or profiles

Does not test a complete system Tests individual system components, not the whole

• A system is often greater than the sum of its parts!• Does not test functionality

– Does not test the user’s ‘perception’ of the system

Standardised conformance tests do not include proprietary ‘aspects’ Though this may well be done by a manufacturer with own

conformance tests for proprietary requirements


Conformance Testing Methodology ISO/IEC 9646 (parts 1-7)

Test Suite (Test Cases)

ATS

Req. checklist

ICS

Test Purposes

TSS & TP

testingTest

Report

logging and

analysis

Test System

Base Standard

(or Profile)

implementation

Product

Implementation Under Test

compilation

Executable Test Suite

(e.g., C++)

Industry

validation


Implementation Conformance Statement (ICS)

The ICS is a checklist of the capabilities supported by the Implementation Under Test (IUT)

Provides an overview of the features and options that are implemented in any given product

The ICS can be used to select and parameterise test cases and can be used as an indicator of basic interoperability between different products.

Conditional support e.g., Qn must be supported if Q1 supported then otherwise not.

Q# ICS Question Ref Status Support

Q1 Support of Feature F1 [x] Clause a OPTIONAL

Q2 Support of Feature F2 [x] Clause b MANDATORY

: : : :

Qn Support of Message M1 [y] Clause c CONDITIONAL

: : : :

Qk Support of message Parameter P1 [y] Clause d OPTIONAL


Profile ICS

Q# ICS Question Ref Status Profile Status Support

Q1 Support of feature F1 [x] Clause a OPTIONAL MANDATORY

Q2 Support of feature F2 [x] Clause b MANDATORY MANDATORY

: : : : :

Qn Support of Message M1 [y] Clause c CONDITIONAL MANDATORY

: : : : :

Qk Support of parameter P1 [y] Clause d OPTIONAL N/A

A profile ICS reflects how a (protocol) profile restricts the scope of a base standard by making certain options mandatory or excluding certain options.


Selecting Tests with the ICS

Q# ICS Question Ref Status Support

Q1 Support of Feature F1 [x] Clause a OPTIONAL

Q2 Support of Feature F2 [x] Clause b MANDATORY

: : : :

Qn Support of Message M1 [y] Clause c CONDITIONAL

: : : :

Qk Support of message Parameter P1 [y] Clause d OPTIONAL

A filled-in ICS can be used to select tests. For example, a test for an OPTIONAL feature which is not supported in a given product will be deselected from the test suite.


Parameterizing Tests with eXtra Information for Testing (IXIT)

Q# IXIT Question Ref Allowed Values Value

Q1 Network address [x] Clause a Valid IP address 12.34.56.76

Q2 Value of Timer T1 [x] Clause b 1 .. 7 s 5s

: : : : :

A filled-in IXIT can be used to parameterize tests. The Value column is used to specify explicit IUT or test system dependent values.


The Requirements Catalogue

Each Requirement is categorized as follows (example from IPv6): Requirement type:

• Mandatory (MUST, MUST NOT)

• Recommended (SHOULD, SHOULD NOT)

• Optional (MAY, MAY NOT)

Requirement target:• E.g., Host, Router, Node (Host or Router)

Requirement text Functional groupings

• E.g., Process Fragmented packet, Generate ICMPv6 Error Typetc.

A scalable database containing all requirement elements Web interface Browsing by function User-defined search Links to RFC and related test specification


ETSI IP Testing Library


Conformance Test Purposes

Test Purposes (TP) are natural language, precise descriptions of the purpose of the test in relation to a particular (base standard) requirement

Define the functionality being tested—the WHAT Do not define HOW to test the function Grouped into a logical structure (Test Suite Structure)

TSS & TP

One Requirement may spawn several TPs A conformance TP is written on the protocol level Specified in

Natural language, or ETSI’s Test Purpose Language (TPLan) They are not test code


TPLan Example for Conformance

TP id : TP_COR_0047_01Summary : ‘hop limit of one'RQ Ref : RQ_COR_0047Config : CF_02_CTC Ref : TC_COR_0047_01ensure that {--Stimulus when { IUT receives ‘Ipv6 packet' from ‘Host' containing ‘IPv6 Header' indicating ‘Hop limit' set to ‘1‘ }--Expected response then { IUT sends ‘ICMPv6 Time Exceeded' to ‘Host‘

containing ‘ICMP code' set to ‘ZERO‘ } }


Conformance Test Cases

Detailed TTCN-3 test script that implements test purpose can be compiled and executed

Is the HOW to test not WHAT to test Composition

a preamble test body (i.e., implementation of the Test Purpose) A postamble

Assigns test verdicts Handles unexpected behaviour as well as the behaviour in

the test purpose Can be distributed over parallel test components Can be entirely automated Configurable at run-time, e.g., SUT address


Abstract Test Suite

The Test Suite (ATS) is the collection of all Test Cases Most ETSI Test Suites are written in TTCN

Predominantly in TTCN-2 Progression to TTCN-3 for new work Not RF testing (generally not physical layer)

TTCN-3 Testing and Test Control Notation Allows tests to be specified in detail (test code) that is

independent of the (eventual) real test system on which it may be run

• i.e., TTCN-3 will run on any test system that supports a standardised TTCN-3 execution environment


What is TTCN-3?

Internationally standardized testing language Look and feel of a regular programming language Allows unambiguous implementation of tests Independent of execution environment due to

separate test system interface standards Wide range of applications from mobile (radio)

communications to Internet to software Good tool support Good book:

http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470012242.html


Example TTCN-3 Test Case

testcase TC_COR_0047_01() runs on Ipv6Node system EtherNetAdapter { f_cf02Up(); // Configure test system for HS->RT // No preamble required in this case f_TP_HopsSetToOne(); // Perform test // No postamble required in this case f_cf02Down(); // Return test system to initial state}

function f_TP_HopsSetToOne() runs on Ipv6Node { var Ipv6Packet v_ipPkt; var FncRetCode v_ret := f_echoTimeExceeded( 1, v_ipPkt ); if ( v_ret == e_success and v_ipPkt.icmpCode == 0 ) { setverdict(pass);} else { setverdict(fail); }}

function f_echoTimeExceeded(in UInt8 p_hops, out Ipv6Packet p_ípPkt ) runs on Ipv6Node return FncRetCode { var Ipv6Packet v_ipPacket; var FncRetCode v_ret; ipPort.send( m_echoReqWithHops(p_hops) ); alt { [] ipPort.receive( mw_anyTimeExceeded ) -> value p_ipPkt { return e_success } [] ipPort.receive { return e_error } }}


Executable Test Suites

Executable test suites are compiled from the abstract Either direct to binary, or More commonly to some intermediate programming

language (C++, Java, etc.)

ETSI does not deliver executables, but We try to ensure all code can be compiled And validated by executing the tests against a real

implementation on at least one test system• 5 test systems in the case of UMTS, for example


TTCN-3 Test System

ENCODER

DECODER

(N-protocol specific)

Adaptation Layers

TRITTCN-3 Runtime Interface

TCI TTCN-3 Control Interface

The Industry

Test Suite in TTCN-3

(source)

TTCN-3 Test Suite

(object)

Compilation

PICS etc

(reqs. catalogu

e)

Parameterisation

Selection

Underlying Protocol Stack

(N-1)

Connection to the SUT

Control / Logging

User


Interoperability Testing End-to-End Interoperability of devices

What is being tested? Assignment of verdicts?

Need to distinguish between Formal interoperability testing And informal interoperability events used to validate/develop

technologies

Device1 Devicen

Device2


SUT

API

MMI

L3

L2

PHY

API

MMI

L3

L2

PHY

QE = Qualified

Equipment

EUT = Equipment Under Test

Interoperability Testing Tests Functionality Between Devices

Interoperability Testing(of terminals)

1 2 3

4 5 67 8 9

* 8 #

QE EUT

1 2 3

4 5 67 8 9

* 8 #

Means of Communication (MoC)


Characteristics of Interoperability Testing

Is system testing Tests a complete device or a collection of devices

Shows that (two) devices interoperate Albeit within a limited scenario

Tests at a ‘high’ level (as perceived by users) Tests the ‘whole’, not the parts

• e.g, protocol stacks + applications Tests functionality

• Shows function is accomplished (but not how)

Does not necessarily require a test system Uses existing interfaces (standard/proprietary)

Interoperability Testing is BROAD and SHALLOW Less thorough but wide in scope Gives a high-level of confidence that devices (or components in

a system) will interoperate with other devices (components)


Limitations of Interoperability Testing

Does not prove interoperability with other implementations with which no testing has been done A may interoperate with B and B may interoperate with C. But it

doesn’t necessarily follow that A will interoperate with C. Combinatorial explosion

Does not prove that a device is conformant Interoperable devices may still interoperate even though they

are non-conformant

Cannot explicitly test error behaviour or unusual scenarios Or other conditions that may need to be forced (lack of

controllability) Has limited coverage (does not fully exercise the device)

Not usually automated and may not be repeatable


Interoperability Testing Methodology ETSI TS 102 237

Interop Test Cases

Test Report

logging and

analysis

Base Standard (or Profile)

implementation

Product 1

Qualified Equipment

Product 2

Equipment Under Test

implementation

IFS (functional checklist)

Interop Test Purposes

testing


Example Interoperability Test Description

Identifier:

Step Step

Test Purpose: TP_COR_1100_01 Reference: RQ_COR_1100 Configuration:

TD_COR_1100_01Summary EUT reassembles a fragmented packet of an original length less than 1500 octets

with { 'the MTU on Link1 set to 1400 octets' } ensure that { when { QE is requested to 'send data requiring a packet length greater than 1500 octets' } then { EUT indicates 'receipt of the same data without modification' } }

Pre-Test Conditions:

MTU set to 1400 octets on link1

Verdict

1

2

3

Cause QE to send an Echo Request to EUT with a packet size of 1450 octets and with each octet set to the hexadecimal value "F0"

CF_011_I

Check: Does protocol monitor show that the Echo Request was sent from QE to EUT?

Yes No

Check: Does QE receive an Echo Reply from EUT with the packet length the same as the Echo Request and with each octet containing the hexadecimal value "F0"?

Yes No

Pass Fail

Test Description

Observations


Automated IOP Testing Using TTCN-3

TestDriver

(TTCN-3)

TestDriver

(TTCN-3)

TestController(TTCN-3)

AT Command

s

AT Command

s


IOT with Conformance Verifiction(aka NIT - Network Integration Testing)

NWC1

NWC2

NWC3

Terninal E2E tests driven by human users

Terminal E2E tests over internal product API (automated)

UNI UNI

Network E2E tests over the UNI (automated)

Ra Rb

Conformance verification of reference points


Example NIT for IMS

End-to-End Testing

Mw Mw

UE1

Um Um

UE2

P-CSCF1

S-CSCF1 S-CSCF2

Cx

Mw

Access Network

HSS

I-CSCF P-CSCF2 Access Network


Conformance and Interoperability Testing are Complementary

ETSI experience As you move up a system stack the emphasis should change

from conformance to IOT

Lower layer protocols, infrastructure Emphasis on conformance

Middleware, enablers Combination of Conformance + IOT

Services, applications, systems Emphasis on IOT

Conformance testing as a pre-requisite to IOT

Interop Testing

Conformance Testing

Interoperability


TTCN-3 Standardshttp://www.ttcn-3.org

ES 201 873-1 (Z.140) TTCN-3 Core Language

ES 201 873-2 (Z.141) TTCN-3 Tabular Presentation Format (TFT)

TR 101 873-3 (will eventually be ES 201 873-3) (Z.142) TTCN-3 Graphical Presentation Format (GFT)

ES 201 873-4 (Z.143) TTCN-3 Operational Semantics

ES 201 873-5 TTCN-3 Runtime Interface (TRI)

ES 201 873-6 TTCN-3 Control Interfaces (TCI)

ES 201 873-7 and upwards Using ASN.1, XML, IDL, C/C++, with TTCN-3

Thank You!

Phone +33 (0)4 92 94 43 26

e-mail [email protected]

Website www.etsi.org/ptcc

Documents

Overview of ETSI Testing Methodology Anthony Wiles Manager ETSI Protocol and Testing Competence Centre