7/26/2019 Slides SEAA

1/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

A systematic mapping study on test

generation from Input/Output TransitionSystems

Sofia Costa PaivaAdenilso Simao

Instituto de Ciencias Matematicas e de ComputacaoSao Paulo University, Brazil

August 27, 2015

1 / 1 9

http://find/

7/26/2019 Slides SEAA

2/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

1 IntroductionModel-Based Testing

Problem Statement2 Mapping Planning and Conduction

Goal and Research QuestionsSearch Strategy and Extraction Process

3 Analysis and Discussion of ResultsGeneral Information about StudiesIOTS Model CharacteristicsTest Selection CriteriaTechnology

Test ExecutionMap of selected studiesThreats to validity

4 Conclusion

2 / 1 9

http://find/

7/26/2019 Slides SEAA

3/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

Context

Software testing:Demonstrate the actual and expected behaviors of a systemare at least similarManual and laborious process

Model-Based Testing (MBT)Strategy to minimize time and resource limitations

Automate the generation of high-quality test suites frombehavioural models.A model of desired implementation behaviour: the startingpoint of MBT.

3 / 1 9

http://find/

7/26/2019 Slides SEAA

4/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

Techniques used in MBT

Finite State Machines -FSM

Test generation methodsfrom FSM guaranteefault coverage using a

fault model

Input/Output TransitionSystems - IOTS

Existing works merelyindicates that test suitemust be generated in a

nondeterministic way

4 / 1 9

I d i M i Pl i d C d i A l i d Di i f R l C l i

http://find/

7/26/2019 Slides SEAA

5/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

Goal

Several recent studies have addressed issues related to testgeneration from IOTSs

IOTS is a suitable formalism to model complex systems, suchas communication protocols and distributed systems

Systematic mapping study

Important step to identify relevant studies and shortcomingsRigorous process of assessment and interpretation of allavailable studiesReported systematic mapping studies regarding MBT notincluding IOTS formalism.

5 / 1 9

I t d ti M i Pl i d C d ti A l i d Di i f R lt C l i

http://find/

7/26/2019 Slides SEAA

6/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

Goal and Research Questions

GoalIdentify methods for test case generation from IOTSs througha systematic mapping.

Research questions:

RQ1. What are the main characteristics of IOTS modelsapplied in the studies?RQ2. What are the test selection criteria used in the studies?RQ3. What are the technologies used in the test generationfrom IOTSs?

RQ4. What are the features of the test executionimplemented in the studies?

6 / 1 9

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

7/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

Mapping Planning and Conduction

Search strategy: using control studies

Keywords: test generation and IOTS

Engines: IEEE Xplore, Springer Link, Scopus, Web of

Knowledge, ACM Digital Library, ScienceDirect, Compendexand Oxford Journals

Figure : Mapping Process

7 / 1 9

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/http://goback/

7/26/2019 Slides SEAA

8/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

Taxonomy of MBT approaches

8 / 1 9

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

9/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

Taxonomy of MBT approaches

9 / 1 9

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

10/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

Analysis and Discussion of Results

General information about studies

Figure : Distribution of studies overthe years

Figure : Distribution ofstudies into forums

10/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

11/19

pp g g y

Analysis and Discussion of Results

General information about studiesComputational support

TGV (6 studies)

TorX (1 study)UPPAAL (1 study)

Tipe of evidence:

68% provide only examples13% report a case study8% report experimental results10% only discuss the application of the approach

11/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/http://goback/

7/26/2019 Slides SEAA

12/19

pp g g y

Analysis and Discussion of Results

RQ1. IOTS Model Characteristics

Untimed (92%) andTimed (8%)

Deterministic (88%) andnon-deterministic (12%)

Discrete (98%), hybrid(1%) and continuous(1%)

12/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

13/19

Analysis and Discussion of Results

RQ2. Test Selection Criteria

Structural model coverage (14 studies)Requirements coverage: test purpose (26 studies)Requirements coverage: properties (5 studies)Test case specification (7 studies)Test case specification: iocotheory (31 studies)Fault-based (12 studies)

13/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

14/19

Analysis and Discussion of Results

RQ3. Technology

Random/non-deterministic (26 studies)Search-based algorithms (26 studies)Model-checking (8 studies)Simbolic execution (10 studies)

Constraint solving (6 studies)Inference rules (5 studies)Data abstraction (1 study)

14/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

15/19

Analysis and Discussion of Results

RQ4. Test ExecutionOnline (34 studies)Offline (25 studies)Communication:

Synchronous (26 studies)

Asynchronous (10 studies)Distributed (12 studies)

Online and synchronous test generation have been widely used

adopted by the classical method of Tretmans.

Online testing use randomize choice techniques

provided only a probabilistic guarantee of coverage for(unrealistic) a long time execution.

15/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

16/19

Analysis and Discussion of Results

Map of selected studies regarding Test Generation criteria andTechnology

16/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

17/19

Threats to validity

Selection bias was treated by defining our inclusion criteria forgathering the largest possible amount of papers that would fitthe test generation from the IOTSs domain

A specialist validated the results so that inaccuracy in dataextraction and misclassification could be alleviated.

17/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

18/19

Conclusion

Lack of a standard of IOTS model characteristics may restrictthe adoption of each method.Widely used coverage criteria, such as state and transitioncoverage, have been applied to IOTSs in only few recentstudies.Most of test generation technologies are non-deterministic

There exist no standard fault models, leading to most testgeneration algorithms being nondeterministic.

Completeness is guaranteed in theory, by repeating theprocess an unbounded number of times.

Recent studies on IOTS-testing have adopted concepts of

FSM-testingHowever, fault models (which guarantee the completeness oftest suites) has played a relatively small role in IOTS testing.

Future work:Replication of study and include more evidence

Quality evaluation18/19

Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion

http://find/

7/26/2019 Slides SEAA

19/19

A systematic mapping study on test

generation from Input/Output TransitionSystems

Sofia Costa Paivasofialc@icmc.usp.br

Instituto de Ciencias Matematicas e de ComputacaoSao Paulo University, Brazil

August 27, 2015

19/19

http://find/