Testing of protocols for cooperative ITSAchieving the first ITS test specifications with TTCN-3

Agenda

Why standardized conformance test

specifications? The UMTS example

Introduction to Centre for Testing and

Interoperability (CTI)Interoperability (CTI)

The phases of the first TTCN-3 test

specifications

Cooperative ITS interoperability event

2

UMTS example

Standards Test System Test Lab CertificationTest Specifications

TC ITS

Standardized test specifications

• Are part of the product development process

• Are the basis for certification schemes

• Define clear and unambiguous tests

• Guarantee same test results independent of test systems and test labs3

Standards and

Test Specifications Prototype Test System

TC ITS

CTI

Centre for Testing and Interoperability (CTI)

SPECIFICATION

•Support ETSI Technical Committees on the

application of best practice protocol

specification methods, techniques and tools.

•e.g., ASN.1, UML, MBT (Model Based Testing)

•3GPP

•AERO

•ATTM

•BRAN

•CLOUD

•DECT

•eHEALTH

4

VALIDATION

TESTING

•Support ETSI Technical Committees on the

validation of standards.

•Mainly Plugtests events (organisation and

provision of testing expertise)

•Support ETSI Technical Committees on ALL

testing aspects including the development of

test frameworks, methodologies, test

specifications. Mostly through

participation/leadership of STFs

•e.g., STF 160, ITS, TTCN-3

•eHEALTH

•ERM

ESI

•HF

•INT

•ITS

•LI

•MTS

•PLT

•SCP

•STQ

•TETRA

•TISPAN

The initial phase: study the ITS specific features and provide appropriate guidance• ITS Testing Framework

The second phase: provide formal conformance test specifications• For CAM / DENM / GN (Geonetworking)

The third phase: develop a platform for conformance testing• To validate the conformance test specifications

Different Phases (projects) for testing

• To validate the conformance test specifications

• To give feedback on the conformity of products

Further phases:• Maintenance of test specifications, following updates of the base

specifications

• Development of further test specifications for other base standards,

• Development of test specifications for interoperability events (Plugtests)

• …

Guidance for the application of the two types of test methods commonly used in ETSI:• Conformance testing

• Interoperability testing

The ETSI guide consists of two main parts:• The first part is suitable for non-testing experts to understand how the testing

strategy commonly used at ETSI can be applied to ITS (clauses 4 to 7)

The initial phase (Feb – Oct 2010)

strategy commonly used at ETSI can be applied to ITS (clauses 4 to 7)

• The second part is more dedicated to testing experts, for understanding specific implementation and design principle to be applied for testing ITS systems (clause 8 and annexes A and B)

One deliverable was produced:

• EG 202 798: Intelligent Transport System (ITS); Testing Framework for conformance and interoperability testing

ACHIEVED

Implementation of CAM/DENM/GN test specifications according to the ITS Test Framework• ensuring a harmonized global approach towards testable ITS standards

and related test standards based on a common test framework

For example• The test purpose table proforma, which is providing:

•

The second phase (May – Dec 2010)

• The test purpose table proforma, which is providing:• all relevant fields to describe the expected test behaviour and the test

environment,

• A style guide for writing test behaviour in a clear and unambiguous manner

• Test architecture enabling to apply the standardized formal test methodologies

• ISO 9646 (testing concept, test descriptions, test language)

• ES 201 873 (improved test language)

ACHIEVED

PICS (Protocol Implementation Conformance Statement)• A PICS proforma is a questionnaire or check list that covers all requirements,

optional and conditional functions, procedures, messages, timers and other capabilities identified in the ITS standards.

• PICS proforma are to be completed by the suppliers of the ITS station to be tested

• See ISO 9646-7

Test purpose

Different parts of test specifications

Test purpose• A test purpose is a prose description of a well-defined objective of testing

• It focuses on a single requirement, which shall be followed by ITS equipment to be compliant with the standard

Abstract Test Suite / Abstract Test Case• An abstract test suite is a set of abstract test cases

• A test case is a complete formal specification of the actions and protocol messages required to achieve a given test purpose.

• Uses dedicated “test language”, enabling a direct compilation and then execution of the test cases (e.g. TTCN-3)

• See ISO 9646-1 to -3 and ES 201 873 (specifying the test language TTCN-3)

• See also http://www.ttcn-3.org/StandardSuite.htm

TTCN-3 details of CAM/DENM/GN testing

See ITS presentation in T3UC 2011 in Bled, Slovenia for all

details http://www.ttcn3uc.eu/

Test Purposes

• Use of Framework

• Systematic Approach• Systematic Approach

• Project specific TP Rules

Test Suite Design Decisions

• Port or PTC

• Upper Tester Design

• PCO Placement

• Naming Conventions

• Structure and modularization of the TTCN-3 code

9

Further Phases - Test Platform

Project duration from March 2011 – November 2012

Design, build and validate a TTCN-3 test platform

Validate the CAM/DENM/GN test specification by running test against real implementations

Allow for other protocol tests to be included

Allow for other hardware controller components to be Allow for other hardware controller components to be included

See http://www.etsi.org/WebSite/Technologies/IntelligentTransportSystemsTesting.aspx

Further Phases

Cooperative ITS Interop Event

Idea was approved during the TC ITS meeting in Mainz (Oct.

2010)

To organize an interoperability event with the support of CTI

• Offer the capability to proceed with face to face test sessions, where

vendors can assess the level of interoperability of their vendors can assess the level of interoperability of their

implementations

• Verify their correct understanding of the base standards requirements

• Rather Q1 2012 due to the availability of implementations

• Subscribe to ITS_Plugtests@list.etsi.org at

http://www.etsi.org/WebSite/document/Technologies/ITS%20Call%20

for%20Interest%20ITS%20Interoperability%20Event.pdf

ETSI is willing to cooperate with other ITS players (ERTICO,

C2C CC)

11

Cooperative ITS Interop Event - Scope

Interoperability testing

• V2V and V2I

• Involves OBE, RSE and infrastructure

• Lab conditions or field trial

• The first event could aim to verify communication stacks

• access, networking/transport, facilities

• But application are required to trigger the communication stacks

• Application scenarios

• Synchronization and synergy with ongoing FP7 projects

Pre-conformance testing:

• Provides pre-conformance test services as an output from Test

Platform project:

• Enable providing a compliance check before IOP tests

• A good opportunity to assess the usefulness of conformance testing

• Can also be a part of the STF VJ validation task.12

Making it Happen

Industry (e.g.

Fora)Participants

(Vendors)

CTI

Coordination, Organisation,

Technical Expertise etc.

Technical

Partner

13

Standards

(ETSI, IEEE,

ISO, ...)

How do we Organise Plugtests?

Event Co-ordination

• Event Promotion

• Legal Aspects

• Finance

• Logistics

• Dedicated Event Supervisor

14

Technical co-ordination

• Test plan

• Test Infrastructure

• Test Scheduling

• Test Session Reporting

• Final technical report

Typical Plugtests Events

In operation since 1999 (in CTI since 2007)

Nearly 150 events, involving more than 4000 engineers

For many diverse technologies

2010

EUROCAE#4 (Air Traffic Management)

15

EUROCAE#4 (Air Traffic Management)

Femtocell#1

DECT CAT-IQ 2.0#1,#2

GPON#5 (Gigabit Optical Networks)

Electronic Signature XaDES, CaDES

SIPit#27 (Taiwan)

Planned in 2011

Femtocell#2,#3, IMS#4, Eurocae#5, SIPit#28, IPv6, DSC Marine Radios

sebastian.mueller@etsi.org

16

Thank you

Additional Material

Conformance testing:

• Process for testing that an implementation is compliant with a protocol standard, which is realized by test systems simulating the protocol with test scripts executed against the implementation under test.

Interoperability testing:

Few definitions

Interoperability testing:

• Process for testing that devices can inter-operate, which is realized by connecting devices from different vendors and operating them, either manually or automatically, according to scenarios based on a protocol standard.

The ETSI Guide EG 202 798 is a Testing framework:

• a document providing guidance and examples necessary for the development and implementation of a test specification.

CAM / DENM / GN test specifications

Example of a table in the CAM PICS

Example of a CAM test purpose

Example of a CAM Test Case

The functional TTCN-3 test architecture

The "Test Control" enables the management of the TTCN-3 test execution

The "TTCN-3 test components" are processes providing the test behaviour

The "Codec" is a functional part of the test system to encode and decode messages (ASN.1)

The "Test adapter" (TA) realizes the interface between the TTCN-3 ports using TTCN-3 messages, and the physical interfaces provided by the IUT

The “Upper Tester Transport” (Bus controller) enables sending or receiving service primitives from protocol layers above the IUT or from the management / security entity

23