MAKE YOUR GLOBAL

MARKET EASY

www.eglobalmark.com

A SURVEY ON MODEL-BASED TESTING

TOOLS FOR TEST CASE GENERATION

Wenbin Li, Easy Global Market, France

OUTLINE

TMPA 2017, Wenbin Li (Easy Global Market) Page 2

Easy Global Market (EGM)˃ Activities ˃ Projects

Model-Based Testing (MBT)˃ Concept and Workflow˃ Aspects

Survey Criteria˃Model Specification˃ Test Generation˃ Test Description˃Overall Support

Survey MBT Tools Future Directions

EASY GLOBAL MARKET: ACTIVITIES

TMPA 2017, Wenbin Li (Easy Global Market) Page 3

EU H2020: PHANTOM

TMPA 2017, Wenbin Li (Easy Global Market) Page 4

•Multi-Objective Programming for Parallel Computing System

Model-based Testing for

MODEL BASED TESTING WORKFLOW

TMPA 2017, Wenbin Li (Easy Global Market) Page 5

(1) Creation of MBT Models

(2) Generation of Test Cases

(3) Concretization of Test Cases

(4) Execution of Test Cases

(5) Results Analysis

MODEL BASED TESTING ASPECTS

TMPA 2017, Wenbin Li (Easy Global Market) Page 6

Three Aspects

˃ Test Case Generation

˃ Test Data Generation

˃ Test Script Generation

Survey:

˃ Overall Support

˃ Special Focus on Test Case Gen.

Why Test Cases Generation?

˃ Complicated Selection Criteria

˃ Significant Benefits

SURVEY CRITERIA

TMPA 2017, Wenbin Li (Easy Global Market) Page 7

“A taxonomy of model-based testing approaches” by Utting, M., Pretschner, A., Legeard, B.

Extension

˃ Test Description

˃ Overall Support

SURVEY CRITERIA

TMPA 2017, Wenbin Li (Easy Global Market) Page 8

SURVEY CRITERIA: MODEL SPECIFICATION

TMPA 2017, Wenbin Li (Easy Global Market) Page 9

Model

Specification

Scope

Characteristics

Paradigm

Input(I)/Input&Ouput(IO)

Untimed(U)/Timed(T)

Deterministic(D)/Non-Deterministic(N)/Both(B)

Discrete(D)/Continuous(C)/Hybrid(H)

Pre-Post Domains

Transition Based

History Based

Functional

Operational

Stochastic

Data Flow

SURVEY CRITERIA: TEST GENERATION

TMPA 2017, Wenbin Li (Easy Global Market) Page 10

Test Generation

Test Selection

Criteria

Technology

Structural Coverage

Data Coverage

Requirement Coverage

Test Case Specification

Random&Stochastic

Fault Based

Random Generation

Search Based

Model Checking

Symbolic Execution

Theorem Proving

Constraint Solving

Process Online/Offline/Both

SURVEY CRITERIA: TEST DESCRIPTION

TMPA 2017, Wenbin Li (Easy Global Market) Page 11

Test

Description

Test Traceability

Test Oracle

Format

Yes/No

Yes/No

Testing Languages: TTCN-3, Junit, etc.

Programing Languages: Java, C++, etc.

Mark Up Languages: XML, HTML, etc.

Test Frameworks: Selenium, Robot, etc.

Other Formats

SURVEY CRITERIA: OVERALL SUPPORT

TMPA 2017, Wenbin Li (Easy Global Market) Page 12

Overall

Support

Step

Coverage

Tool Type

Open Source(O)

Academic(A)

Commercial(C)

Step 1, 2, 3, 4, 5

Aspect

Coverage

1. Test Case Generation

2. Test Data Generation

3. Test Script Generation

SURVEY ON MBT TOOLS: MODELS AND TEST

TMPA 2017, Wenbin Li (Easy Global Market) Page 13

ToolsModel Specification Test Description

Scope Characteristics Paradigm Traceability Oracles Formats

DIVERSITY IO U/B/D Transition Based No Yes TTCN-3

FMBT IO U/D/D Pre-Post Domains No No AAL/Python

Modbat IO U/B/D Transition based Yes Yes Other formats

TCG IO U/B/D Transition based No No Other formats

Tcases I U/D/D Pre-Post Domains No No XML, JUnit

MISTA IO U/B/DOperational,

Transition BasedNo No

Python, HTML,

Selenium, etc.

MoMuT IO T/D/H Transition Based Yes No Other formats

HTG IO T/D/H

Transition Based,

Functional No No C/C++

Lurette IO T/B/H Functional Yes Yes Lutin

VERA IO U/D/D Transition based No Yes XML

CompleteTest IO T/D/H Data Flow Yes Yes C

CertifyIT IO U/D/DPre-Post Domains,

Transition BasedYes Yes

JUnit, HTML, Perl,

HP quality center, etc.

PragmaDev IO T/B/HTransition-based,

History BasedYes Yes TTCN-3, C/C++

SURVEY ON MBT TOOLS: TEST AND OVERALL SUPPORT

TMPA 2017, Wenbin Li (Easy Global Market) Page 14

ToolsTest Generation Overall Support

Test Selection Technology Process Step Aspect Type

DIVERSITY

Structural Coverage,

Random&Stochastic,

Test Case Specification

Symbolic Execution Offline 1-2 1,2 O

FMBTStructural Coverage,

Test Case SpecificationSearch Based Both 1-5 1,3 O

Modbat Random&Stochastic,Random Generation,

Search BasedBoth 1-5 1 O

TCGStructural Coverage,

Random&Stochastic

Search Based,

Random GenerationOffline 1-2 1,2 O

Tcases Data Coverage Constraint Solving Offline 1-3 1 O

MISTAStructural Coverage,

Random&Stochastic

Search Based,

Random GenerationBoth 1-4 1,3 O

MoMuT Fault Based Search Based Offline 1-2 1 A

HTG Data CoverageRandom Generation,

Search BasedOffline 1-3 1 A

Lurette Random&Stochastic Random Generation Online 1-3 1,2 A

VERA Structural Coverage Search based Both 1-5 1,3 A

CompleteTest Structural CoverageModel Checking,

Search BasedOffline 1-3 1 A

CertifyITStructural Coverage,

Test Case Specification

Search Based,

Model Checking Offline 1-3 1,3 C

PragmaDevStructural Coverage,

Random&StochasticSymbolic Execution Offline 1-5 1,2 C

DIVERSITY

TMPA 2017, Wenbin Li (Easy Global Market) Page 15

SURVEY ON MBT TOOLS: MODELS AND TEST

TMPA 2017, Wenbin Li (Easy Global Market) Page 16

ToolsModel Specification Test Description

Scope Characteristics Paradigm Traceability Oracles Formats

DIVERSITY IO U/B/D Transition Based No Yes TTCN-3

FMBT IO U/D/D Pre-Post Domains No No AAL/Python

Modbat IO U/B/D Transition based Yes Yes Other formats

TCG IO U/B/D Transition based No No Other formats

Tcases I U/D/D Pre-Post Domains No No XML, JUnit

MISTA IO U/B/DOperational,

Transition BasedNo No

Python, HTML,

Selenium, etc.

MoMuT IO T/D/H Transition Based Yes No Other formats

HTG IO T/D/H

Transition Based,

Functional No No C/C++

Lurette IO T/B/H Functional Yes Yes Lutin

VERA IO U/D/D Transition based No Yes XML

CompleteTest IO T/D/H Data Flow Yes Yes C

CertifyIT IO U/D/DPre-Post Domains,

Transition BasedYes Yes

JUnit, HTML, Perl,

HP quality center, etc.

PragmaDev IO T/B/HTransition-based,

History BasedYes Yes TTCN-3, C/C++

SURVEY ON MBT TOOLS: TEST AND OVERALL SUPPORT

TMPA 2017, Wenbin Li (Easy Global Market) Page 17

ToolsTest Generation Overall Support

Test Selection Technology Process Step Aspect Type

DIVERSITY

Structural Coverage,

Random&Stochastic,

Test Case Specification

Symbolic Execution Offline 1-2 1,2 O

FMBTStructural Coverage,

Test Case SpecificationSearch Based Both 1-5 1,3 O

Modbat Random&Stochastic,Random Generation,

Search BasedBoth 1-5 1 O

TCGStructural Coverage,

Random&Stochastic

Search Based,

Random GenerationOffline 1-2 1,2 O

Tcases Data Coverage Constraint Solving Offline 1-3 1 O

MISTAStructural Coverage,

Random&Stochastic

Search Based,

Random GenerationBoth 1-4 1,3 O

MoMuT Fault Based Search Based Offline 1-2 1 A

HTG Data CoverageRandom Generation,

Search BasedOffline 1-3 1 A

Lurette Random&Stochastic Random Generation Online 1-3 1,2 A

VERA Structural Coverage Search based Both 1-5 1,3 A

CompleteTest Structural CoverageModel Checking,

Search BasedOffline 1-3 1 A

CertifyITStructural Coverage,

Test Case Specification

Search Based,

Model Checking Offline 1-3 1,3 C

PragmaDevStructural Coverage,

Random&StochasticSymbolic Execution Offline 1-5 1,2 C

FUTURE DIRECTIONS

TMPA 2017, Wenbin Li (Easy Global Market) Page 18

Support of online testing for real time systems

Generation of test cases in standard testing language

Improvement of MBT tool interfaces

Record of traceability

Automatic generation of test oracle

THANK YOU

Tel:

E.mail:www.eglobalmark.com

•Dr. Wenbin Li

•Research Engineer

•+33-(0)9 82 33 06 72

•wenbin.li@eglobalmark.com