Download pdf - Quality Assurance of Test Artifacts - UCMantares.sip.ucm.es/tarot09/index_files/Grabowski-Zeiss-TAROT09.pdf · 21 TAROT 09 Constructive Product Quality Assurance

1

Quality Engineering for Test Artifacts

Jens Grabowski, Benjamin Zeiss University of Göttingen, Germany

2 2 TAROT 09

Motivation   Test specification complexity is growing

  Current TTCN-3 test suites:   SIP (ETSI TS 102 027-3 - v4.2.5): 61.990 LOC   IPv6 Core Protocol (ETSI TS 102.516 - v3.1.1): 67.580 LOC   3GPP / LTE (STF 160): 71.687 LOC

  Quality Assurance (QA) of test specifications is necessary!

  Quality assurance classification:   Organizational Quality Assurance (infrastructure and management)

  Constructive Product Quality Assurance (avoiding issues, preventative)   Analytical Product Quality Assurance (eliminating issues, reactive)

  How can we apply QA to test specification development?

3 3 TAROT 09

  Testing and Test Control Notation version 3   Language for specifying and implementing distributed tests.

  Standardized by   European Telecommunications Standards Institute (ETSI),

  International Telecommunication Union (ITU).

  History:   Standardisation bodies publish (e.g. for ISDN, GSM, UMTS):

  Specification of a communication protocol,

  Test suite to check conformance of a protocol implementation to its specification.

  Industry:   Implements specified protocols in their equipment,

  Execute standardised test suites against their implementation.

  Today:   TTCN-3 not only used in telecommunication domain, but for

Internet, Service-Oriented Architectures, Automotive, …

TTCN-3 (1/2)

4 4 TAROT 09

TTCN-3 (2/2)   Example:

module exampleModule { ... type record IpAddressType { charstring ipAddress }; template IpAddressType localhostTemplate := { ipAddress := "127.0.0.1" } testcase exampleTestCase() runs on ExampleComponent { portA.send(localhostTemplate); alt { [] portB.receive(localhostTemplate) { setverdict(pass); } [] portB.receive(IpAddressType:{*}) { setverdict(fail); } } } }

  Look and feel of common programming languages

5 5 TAROT 09

  Organizational Quality Assurance

  Constructive Product Quality Assurance

  Analytical Product Quality Assurance

  Tools

  Summary / Conclusion

Agenda

6 6 TAROT 09




  Tools


Agenda

2

7 7 TAROT 09

Organizational Quality Assurance

  Assurance of infrastructure and management quality

  Infrastructure quality:   Configuration management   Defect management   …

  Management quality:   Test process model   Process improvement models

8 8 TAROT 09

Organizational Quality Assurance: Test process model (1/3)   Describes activities that need to happen in development

  On what subject something needs to happen   When something happens   Why it needs to happen

  Example: ISQTB fundamental test process:

9 9 TAROT 09

Organizational Quality Assurance: Test process model (2/3)

  Example: ISQTB fundamental test process (continued)   Test planning and test control

  Testing approach, resource planning, test strategy, schedule, metrics definition for monitoring and control

  Test plan document: IEEE 829 Test Plan Specification

  Test analysis and test design   Decision what to test, how much effort to spend, what types of

testing, tools, risk analysis   Test specification: IEEE 829 Test Design Specification   Document reviews

  Test implementation and test execution   Definition of high-level and concrete test cases   TTCN-3

10 10 TAROT 09

Organizational Quality Assurance: Test process model (3/3)

  Example: ISQTB fundamental test process (continued)

  Evaluation of exit criteria and reporting   Test summary report (test progress / test conclusion)   Exit criteria from the planning phase satisfied?

  Closure   Wrap-up reports   Archiving of test documents and test data

11 11 TAROT 09

Organizational Quality Assurance

  Assurance of infrastructure and management quality

  Infrastructure quality:   Configuration management   Defect management   …

  Management quality:   Test process model   Process improvement models

12 12 TAROT 09

Organizational Quality Assurance: Process improvement models

  Generic process improvement models   Capability Maturity Model integrated (CMMI)   SPICE (ISO 15504)

  Test process improvement models   Test Maturity Model integrated (TMMi)   Test Process Improvement (TPI)   Critical Testing Processes (CTP)   Systematic Test and Evaluation Process (STEP)

3

13 13 TAROT 09

  TMMi = Test Maturity Model integrated

  Originally developed by Ilene Burnstein at Illinois Institute of Technology since1996

  Complementary to CMMI

  TMMi reference model describes level 1-3 in detail

  Can be mapped to other models like TPI

Organizational Quality Assurance: Process improvement models: TMMi (1/3)

14 14 TAROT 09

  Staged model   Level 1 (Initial):

  Chaos, ad-hoc debugging

  Level 2 (Managed):   Test policy and strategy, test planning, test monitoring and control, test design

and execution, test environment   Main objective: does the product satisfy the specified requirements?

(Functional Testing)

  Level 3 (Defined):   Test organization, test training program, test life-cycle and integration, non-

functional testing, peer reviews   Testing at an early project stage   More rigorous process descriptions


15 15 TAROT 09

  Staged model (continued)

  Level 4 (Management and Measurement):   Test measurement, software quality evaluation, advanced peer reviews   Measurements to support decision making

  Level 5 (Optimization):   Defect prevention, test process optimization, quality control

  Continuous test process improvement based on quantitative understanding


16 16 TAROT 09




  Tools


Agenda

17 17 TAROT 09

Constructive Product Quality Assurance   Training and technology certifications

  High-level testing languages

  Best practices:

  Test development guidelines

  Test patterns

  Tooling

18 18 TAROT 09



  Best practices:


  Test patterns

  Tooling

4

19 19 TAROT 09

Constructive Product Quality Assurance: Training and technology certifications (1/2)   Quality Assurance Institute (QAI)

  http://www.qaiworldwide.org   Certified Software Test Engineer (CSTE)   Certified Software Quality Analyst (CSQA)

  American Society for Quality (ASQ)   http://www.asq.org   Software Quality Engineer (CSQE)   Quality Inspector (CQI)   …

  International Institute for Software Testing (IIST)   http://www.iist.org   Certified Software Test Professional (CSTP)   Certified Test Manager (CTM)

20 20 TAROT 09

Constructive Product Quality Assurance: Training and technology certifications (2/2)   International Software Qualifications Testing Board (ISQTB)

  http://www.istqb.org   Certified Tester Foundation Level (CTFL)   Certified Tester Advanced Level (CTAL)   Certified Tester Expert Level (in preparation)

  German Testing Board (GTB)   http://www.german-testing-board.de   TTCN-3 Certificate

  http://german-testing-board.info/de/ttcn3_certificate.shtm

21 21 TAROT 09



  Best practices:


  Test patterns

  Tooling

22 22 TAROT 09

Constructive Product Quality Assurance: High-level testing languages

  Test purpose specification languages:   Formal enough to parse   Informal enough for non-technicians to understand   Possibly: transformation of test purposes into test cases   Possible choices: ETSI TPLan, UML Testing Profile

  Test specification and implementation languages:   Strongly typed (to make analysis and refactoring easy)   High-level, abstracted from technical details   Allows full test automation   Possible choices: TTCN-3, JUnit, UML Testing Profile

23 23 TAROT 09



  Best practices:


  Test patterns

  Tooling

24 24 TAROT 09

Constructive Product Quality Assurance: Best practices: test development guidelines   Naming conventions

  Example: module names start with an upper-case letter

  Code formatting guidelines   Example: spaces instead of tabs

  Code documentation guidelines   Example: all test cases must have a @desc and @param description

  Code style guidelines   Example: no labels or goto statements

  Usage of test patterns

5

25 25 TAROT 09



  Best practices:


  Test patterns

  Tooling

26 26 TAROT 09

Constructive Product Quality Assurance: Best practices: test patterns   Identify and document general and proven solutions for commonly

occurring problems

  Provide a common vocabulary for these solutions

  Catalog with a fixed notation

  Exemplified for TTCN-3: classification proposal for TTCN-3 specification test patterns (Vouffo-Feudijo and Schieferdecker)   Architectural patterns   Behavioral patterns   Data patterns   Test reuse patterns

27 27 TAROT 09

Constructive Product Quality Assurance: Best practices: test patterns – example (1/3)

Name Timer on transmission

Class Behavior

Testing phases Specification

Testing goals All

Application domain Any

Intend Avoid deadlock situation when transmitting data from test component

Context For the testing of reactive systems tested typically via interfaces

Parameter Timer duration

28 28 TAROT 09


Roles test component, source port, destination port

Detailed description After calling a method from a test component or sending a message on a given port, a timer should be started to avoid deadlock in case the SUT does not reply to the function call or to the transmitted message.

Example function timedSend (template <outMessageType> <outMsg>, <OutPortType> <outPort>, timer <t>, template <inMessageType> <inMsg>, <InPortType> <inPort>) returns verdicttype { <outPort>.send (<outMsg>); <t>.start; alt { []<inPort>.receive (<inMsg>) { <t>.stop; return pass; } []<t>.timeout{return fail;} } }

29 29 TAROT 09


Consequences None

Related patterns Default pattern

Known uses Protocol testing

30 30 TAROT 09



  Best practices:


  Test patterns

  Tooling

6

31 31 TAROT 09

Constructive Product Quality Assurance: Best practices: tooling (1/2)

  Are the tools standards conformant?   Can I switch tool vendors easily?

  Do the tools interoperate with other tools?

  Tool efficiency?

  Integrated Development Environment?

  Documentation generator?

32 32 TAROT 09

Constructive Product Quality Assurance: Best practices: tooling (2/2)

  Reporting?

  Analytical support?

  Support for quality assurance?

  Version control?

  Defect management system?

  Build automation / continuous integration?

33 33 TAROT 09




  Tools


Agenda

34 34 TAROT 09

Analytical Product Quality Assurance

- Code Review ...

Analytical Test Quality Assurance

Static Testing

With Computer

Dynamic Testing

- Guideline Checking - Test Metrics - Smell Detection - …

- Simulation -  Model-Based Analysis -  ...

Without Computer

  Improvement: test refactoring (and debugging/correction)

35 35 TAROT 09


- Code Review ...


Static Testing

With Computer

Dynamic Testing



Without Computer

  Improvement: test refactoring 36 36 TAROT 09

Analytical Product Quality Assurance: Static testing: code reviews (1/2)

7

37 37 TAROT 09

Analytical Product Quality Assurance: Static testing: code reviews (2/2)

  Review types:   Informal review

  Reviewers are asked to deliver remarks.   No meeting, no formal documentation.

  Walkthrough   Author presents artifact to the reviewers in a meeting. Remarks based

on the presentation.

  Technical review   Does the test conform to its specification?   Fitness for its purpose, Standards compliance.   Meeting, unanimous review result.

  Inspection   Formal with focus on possible defect, document quality, product and

process quality.   Strict Agenda.

38 38 TAROT 09


- Code Review ...


Static Testing

With Computer

Dynamic Testing



Without Computer

  Improvement: test refactoring

39 39 TAROT 09

Analytical Product Quality Assurance: Static testing: guideline checking   Semantic analysis / type checking

  Float assignment to an integer variable,   Reading of undefined variable,   Call to undefined function,   Number of arguments mismatch,   Duplicate functions/test case names,   etc.   Should be done by the compiler

  Naming conventions / style guidelines   Test case should start with prefix “TC_”,   Every test case must have a T3Doc comment,   Log messages must have a certain format,   Types should be defined in alphabetic order,   Variables should be declared first in a test case,   etc.

40 40 TAROT 09


- Code Review ...


Static Testing

With Computer

Dynamic Testing



Without Computer


41 41 TAROT 09

Analytical Product Quality Assurance: Static testing: test metrics (1/11) “When you measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science.”

Lord Kelvin, 1891

“You cannot control what you cannot measure.” Tom DeMarco, 1982

42 42 TAROT 09

Analytical Product Quality Assurance: Static testing: test metrics (2/11)

  Metric: measurement scale + measurement   Mapping of a software property into a numerical value or symbol of

the measurement scale

  Vast amount of metrics have been suggested:   Counting (Number of XXX, LOC)   Cyclomatic complexity   Coverage   Coupling   …

8

43 43 TAROT 09


 Metrics classification (Fenton, Pfleeger):

Software metrics

Processes

Size Metrics

Resources Products

Internal Attributes External Attributes

Structural Metrics

44 44 TAROT 09


 The inconvenient truth:   Metrics are not universal!   Metric thresholds are not universal!

45 45 TAROT 09

Analytical Product Quality Assurance: Static testing: test metrics (5/11)   You are often interested in exactly the opposite of what you

measure:   90% tests with verdict “pass”

  Why are 10% of the tests failing or inconclusive? Do the 10% reflect the severity of the negative test results?

  90% specification coverage   Why is 10% of the specification untested? Hard to test?

  Number of test cases   How can I select test cases (maximize coverage, minimize test executions)?

  Metrics tell you what happened, but not why!

46 46 TAROT 09


  Reoccurring problems with metric definitions:   The metric is unbelievable. People don’t understand it.

  The metric is not defined clear or it is ambiguous or non-objective.

  The metric is not repeatable.

  The metric is not comparable to other measurements.

  People can fool the metric to make it look good.

  It is unclear what good and bad values for the metric are (thresholds).

  The metric is “just” interesting and not measured for a reason.

47 47 TAROT 09


  What should be measured then???   A quality model is required!

  A quality model for a product provides:   The main quality characteristics of a product,   How these quality characteristics are subdivided.

  ISO 9126-1:   Software engineering – Product quality – Quality Model

  Quality models for   Internal quality,   External quality,   Quality in use.

Quality is composed of discrete characteristics, which may be structured into further sub-characteristcs.

48 48 TAROT 09


External and Internal Quality

Functionality Reliability Usability Efficiency Portability Maintainability

Suitability

Accuracy

Interoperability

Security

Functionality Compliance

Maturity

Fault-Tolerance

Recoverability

Reliability Compliance

Understand-ability

Learnability

Operability

Attractiveness

Usability Compliance

Time Behaviour

Resource Utilisation

Efficiency Compliance

Analysability

Changeability

Stability

Testability

Maintainability Compliance

Adaptability

Installability

Co-Existence

Replaceability

Portability Compliance

9

49 49 TAROT 09


Test Specification Quality

Suitability

Accuracy

Interoperability

Security

Functionality Compliance

Maturity

Fault-Tolerance

Recoverability

Reliability Compliance

Understand-ability

Learnability

Operability

Attractiveness

Usability Compliance

Time Behaviour

Resource Utilisation

Efficiency Compliance

Analysability

Changeability

Stability

Testability

Maintainability Compliance

Adaptability

Installability

Co-Existence

Replaceability

Portability Compliance

Functionality Reliability Usability Efficiency Portability Maintainability Test

Effectivity

Test Coverage

Test Correctness

Security Fault-

Revealing Capability

Test Effectivity

Compliance

Test Repeatability

Reusability

Coupling

Flexibility

Comprehen-sibility

Reusability Compliance

Test Evaluability

50 50 TAROT 09

Quality Assessment: Static testing: test metrics (10/11)

  How do we define metrics for the quality model?

  Goal, Question, Metric (GQM) approach from Basili and Weiss (1984)

1.  State the goal to be achieved due to the measurement.

2.  Develop questions that breaks the goal into it’s major components. What are the attributes that need to be captured?

3.  Develop metrics that answer the questions, preferably free from side-effects.

51 51 TAROT 09

Quality Assessment: Static testing: test metrics (11/11)

  Example for GQM:   Goal:

  Evaluate the changeability of templates in a TTCN-3 specification

  Questions:   Are inline templates used?   Are there many duplicate inline templates?   What is the degree of test data duplication?

  Metrics:   Number of inline templates   Number of inline templates that are duplicates  

52 52 TAROT 09


- Code Review ...


Static Testing

With Computer

Dynamic Testing



Without Computer


53 53 TAROT 09

Analytical Product Quality Assurance: Static testing: smell detection (1/2)

  Patterns of possibly inappropriate code usage.

  Notion of metrics and code smells not disjoint:   Code smell Metric: count occurrences of code smell.

  Metric Code smell: metric violates boundary.

  > 38 TTCN-3 code smells identified (master’s thesis of Martin Bisanz).

  Automatic detection of test smells using static analysis:   Detection of patterns in the Abstract Syntax Tree (AST)   Violation of metric thresholds.

“certain structures in the code that suggest (sometimes they scream for) the possibility of refactoring”

Fowler: Refactoring – Improving the Design of Existing Code. Addison-Wesley, 1999

54 54 TAROT 09

Analytical Product Quality Assurance: Static testing: smell detection (2/2)

Metric / TTCN-3 Code Smell SIP IPv6

Lines of code 42397 46163

Number of testcases 528 295

Number of components 2 10

Duplicate Alt Branches 938 224

Activation Asymmetries 73 317

Distinct Magic Values 135 222

Unused Definitions 50 156

10

55 55 TAROT 09


- Code Review ...


Static Testing

With Computer

Dynamic Testing



Without Computer

  Improvement: test refactoring 56 56 TAROT 09

Analytical Product Quality Assurance: Dynamic testing: model-based analysis (1/6)

  Testing test specifications?   Unreasonable!   We would need tests for the tests that test the tests etc.

  Our solution:   Simulation of test behavior (branch coverage)   Construction of a model from the simulated test runs   Checking the model for anomalies using temporal logic   Result can essentially be considered an testing of specific

properties

57 57 TAROT 09


Structural Property

Linear Temporal Logic Spin / Promela Processes

Model Checking with Spin

Test Component Models

58 58 TAROT 09



Linear Temporal Logic

Structural Property

Spin / Promela Processes

No verdict set!


59 59 TAROT 09



Structural Property



•  Simple possible LTL formula: •  For each path, no verdict is set until

a verdict becomes either pass, inconclusive or fail

•  More sophisticated and comprehensive formulas possible!

60 60 TAROT 09


Structural Property



•  Result: •  The structural property does not hold in

all possible paths. •  The failing traces.


11

61 61 TAROT 09

Analytical Product Quality Assurance: Dynamic testing: model-based analysis (6/6)   Further possibilities:

  Deadlocks   Idle PTC   Default Asymmetries   …

  10+ properties documented (work in progress)

62 62 TAROT 09


- Code Review ...


Static Testing

With Computer

Dynamic Testing



Without Computer


63 63 TAROT 09

Analytical Product Quality Assurance: Improvement: test refactoring (1/6)

„a change made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behavior“

Fowler, M.: Refactoring – Improving the Design of Existing Code. Addison-Wesley (1999)

  Refactoring is a systematic way to restructure test code – no bugfixing, no new functionality

  Improves quality characteristics like:   Usability

  Maintainability   Reusability

  TTCN-3 refactoring catalogue with more than 50 refactorings:   Test behavior, test data, overall test suite structure

  http://www.trex.informatik.uni-goettingen.de/trac/wiki/TTCN3RefactoringCatalogue

64 64 TAROT 09


  Fixed format:   Name   Summary   Motivation   Mechanics   Example

 unrefactored   refactored

65 65 TAROT 09


Refactoring Example: Inline Template Parameter

  Summary:   Inline a template parameter which is always given the same actual value.

  Motivation:   Unneeded parameters create

  code clutter,

  more coupling than needed.

  Mechanics:   Copy template.

  Remove parameter from formal parameter list,

  Replace each reference to formal parameter inside the template with the common actual parameter value.

  Remove actual parameter from each template reference.

  Remove original template.

66 66 TAROT 09

Analytical Product Quality Assurance: Improvement: test refactoring (4/6)   TTCN-3 example (unrefactored):

module ExampleModule {

template ExampleType exampleTemplate(charstring addressParameter):={

ipv6:=false,

ipAddress:=addressParameter

}

testcase exampleTestCase() runs on ExampleComponent {

portA.send(exampleTemplate("127.0.0.1"));

portB.receive(exampleTemplate("127.0.0.1"));

}

}

12

67 67 TAROT 09

Analytical Product Quality Assurance: Improvement: test refactoring (5/6)   TTCN-3 example (refactored):

module ExampleModule {

template ExampleType exampleTemplate:={

ipv6:=false,

ipAddress:="127.0.0.1"

}

testcase exampleTestCase() runs on ExampleComponent {

portA.send(exampleTemplate);

portB.receive(exampleTemplate);

}

}

68 68 TAROT 09


68 68

•  Effects on the number of templates (experiments):

11,5%

53,4%

36,8%

69 69 TAROT 09




  Tools


Agenda

70 70 TAROT 09

Tools for general-purpose languages

  Guideline checking and smell detection:   Java: Findbugs, PMD, Checkstyle, Sonar (free)   C/C++: Coverity Prevent, PCLint (commercial)   C#: FxCop, StyleCop, devAdvantage

  Software metrics:   Java: Eclipse Metrics Plugin (free), CodePro AnalytiX (commercial)   C#: DevMetrics (free)

71 71 TAROT 09

Tools for general-purpose languages

  Refactoring:   Java: Eclipse, Netbeans   C/C++: Eclipse, SlickEdit   C#: Visual Studio+Resharper

  Dynamic Analysis:   Java: Java Pathfinder (free)   C/C++: Valgrind (free)

72 72 TAROT 09

Tools developed in Göttingen   TRex

  Metrics   Smell Detection   Refactoring

  ETSICheck   TTCN-3 Guideline Checker

  ETSIDOC   TTCN-3 Documentation Generator (HTML/PDF)

  Tool for Model-Based Analysis   Reverse-Engineering Models from TTCN-3 Specifications   Analyzing Properties using Model-Checking

13

73 73 TAROT 09




  Tools


Agenda

74 74 TAROT 09

Summary   Today's test specifications are complex

  Quality Assurance (QA) is necessary

  QA aspects to be considered:   Organizational QA

  Test process model, test process improvement model

  Constructive Product QA   Training, code / style guidelines, test patterns, …

  Analytical Product QA   Static testing, dynamic testing, refactoring, debugging/correction

75 75 TAROT 09

Conclusion   QA in testing involves more than coding guidelines!

  QA in testing needs to be considered early on   Example: consideration of tool support in decision making

  This talk covered only selected aspects of test QA   Organizational aspects were only indicated   Test process metrics not covered   Test debugging methodologies not covered

  Raise the awareness for QA in testing!

76 76 TAROT 09

Thank you for your attention!

77 77 TAROT 09

Literature (1/6)   Organizational Quality Assurance:

  Carnegie Mellon Software Engineering Institute: CMMI for Development Version 1.2. 2006.

  TMMi Foundation: TMMi Reference Model Version 2.0. 2009.

  Koomen, T., Pol, M.: Test process improvement: a practical step-by-step guide to structured testing. Addison-Wesley, 1999.

  TTCN-3 Tools:   http://www.trex.informatik.uni-goettingen.de   http://www.ttcn-3.org/CommercialTools.htm

78 78 TAROT 09

Literature (2/6)   General-Purpose Tools:

  http://www.eclipse.org   http://www.netbeans.org   http://metrics.sourceforge.net   http://www.instantiations.com/codepro   http://www.slickedit.com   http://javapathfinder.sourceforge.net   http://valgrind.org   http://findbugs.sourceforge.net   http://sonar.codehaus.org   http://www.coverity.com   http://www.gimpel.com   http://code.msdn.microsoft.com/sourceanalysis   http://www.anticipatingminds.com

14

79 79 TAROT 09

Literature (3/6)   Constructive Quality Assurance:

  Vouffo-Feudijo, A. and Schieferdecker, I.: Test Patterns with TTCN-3. FATES 2004, Volume 3395 of Lecture Notes in Computer Science, 2004.

  Din, G.: A Performance Test Design Method and its Implementation Patterns for Multi-Services Systems. Fraunhofer IRB Verlag, 2009.

  Spillner, A., Linz, T, Schaefer, H.: Software Testing Foundations: A Study Guide for the Certified Tester Exam. Rocky-Nook, 2007.

  Meszaros, G.: xUnit Test Patterns. Addison-Wesley, 2007.

80 80 TAROT 09

Literature (4/6)   Analytical Quality Assurance:

  Static Testing of Test Specifications:   Neukirchen, H., Zeiss, B., Grabowski, J.: An Approach to Quality

Engineering of TTCN-3Test Specifications. International Journal on Software Tools for Technology Transfer (STTT) 10(4), 2008.

  Neukirchen, H., Zeiss, B., Grabowski, J., Baker, P., Evans, D.: Quality Assurance for TTCN-3 Test Specifications. Software Testing, Verification and Reliability (STVR) 18(2), 2008.

  Zeiß, B., Vega, D., Schieferdecker, I., Neukirchen H., Grabowski, J.. Applying the ISO 9126 Quality Model to Test Specifications — Exemplified for TTCN-3 Test Specifications. Software Engineering 2007 (SE 2007), 2007.

  Din, G., Vega, D., Schieferdecker, I.: Automated Maintainability of TTCN-3 Test Suites Based on Guideline Checking. Software Technologies for Embedded and Ubiquitous Systems. Volume 5287 of Lecture Notes in Computer Science, 2008.

81 81 TAROT 09


  Static Testing of Test Specifications:   Vega, D., Din, G., Schieferdecker, I.: TTCN-3 Test Data Analyser Using

Constraint Programming. Proceedings of the 2008 19th International Conference on Systems Engineering, 2008.

  Bisanz, M.: Pattern-based Smell Detection in TTCN-3 Test Suites. Master’s Thesis, ZFI-BM-2006-44, ISSN 1612-6793, Zentrum für Informatik, Georg-August-Universität Göttingen , 2006.

  Zeiss, B.: A Refactoring Tool for TTCN-3. Master’s Thesis, ZFI-BM-2006-05, ISSN 1612-6793, Zentrum für Informatik, Georg-August-Universität Göttingen, 2006.

  Nödler, J.: An XML-based Approach for Software Analysis -- Applied to Detect Bad Smells in TTCN-3 Test Suites. Master’s Thesis, ZFI-BM-2007-36, ISSN 1612-6793, Zentrum für Informatik, Georg-August-Universität Göttingen, 2007.

82 82 TAROT 09


  Dynamic Testing of Test Specifications:   Zeiss, B., Grabowski, J.: Reverse-Engineering Test Behavior Models for

the Analysis of Structural Anomalies. TESTCOM/FATES 2008 Short Papers, 2008.

  Makedonski, P.: Equivalence Checking of TTCN-3 Test Case Behavior. Master’s Thesis, ZFI-MSC-2008-16, ISSN 1612-6793, Zentrum für Informatik, Georg-August-Universität Göttingen, 2008.

83 83 TAROT 09

Contact

 Web:   http://www.swe.informatik.uni-goettingen.de

 E-Mail:   [email protected]   [email protected]

84 84 TAROT 09

Special Thanks  Andreas Ulrich, Siemens AG

  Philip Makedonski, University of Göttingen