Systematic Black-Box Testing of Computer-Based Systems through Formal Abstraction Techniques

8/11/2019 Systematic Black-Box Testing of Computer-Based Systems through Formal Abstraction Techniques

1/151

Systematic Black-Box Testing of

Computer-Based Systems through

Formal Abstraction Techniques

Bernhard Klaus Aichernig

Dissertation

vorgelegt zur Erreichung des akademischen Grades

Doktor der Technischen Wissenschaften

an der

Technischen Universitat Graz

Graz, im Janner 2001


2/151


3/151

Abstract

In the past of computer science, sometimes, a gap between the testing and the formalmethods community could be observed. Testers did not believe in the applicability offormal verification techniques to real world problems, and formal methods advocatescould not accept testing-techniques as an adequate verification method for producing

correct software. However, today the gap is closing and this thesis contributes tobuilding the bridge by offering new results on testing based on formal developmenttechniques.

Others have worked on testing based on formal specifications before, but only afew considered the related development techniques such as program synthesis. Mostof the previous work focused on generating test-cases by partitioning the formalspecification into equivalence classes of input-output behavior. This related workon testing is reviewed.

In this thesis three formal development techniques have been examined, regardingtheir support in the testing of computer-based systems: Light-weight approachesto formal methods, the refinement calculus, and data-reification techniques. As a

result, three novel testing techniques have been developed.First, a light-weight approach to testing, based on an executable formal specifi-

cation, has been applied in two industrial experiments in the domain of voice com-munication for air-traffic control. Abstract VDM prototypes have been interpretedin order to assess and extend existing system-level test-cases. The encouraging re-sults demonstrate that formality is essential for testing high-quality systems. Exactfigures of the detected errors in the test documents and of the efforts are presented.

Second, a main contribution of this thesis is a novel approach to calculate test-cases. It is based on the observation that test-cases are abstractions of systemspecifications. The refinement calculus of Back and von Wright is used to formulate

abstraction rules for deriving correct test-case scenarios from formal specifications.The advantage of this abstraction approach is that simple input-output test-cases aswell as testing scenarios can be handled. This is demonstrated by giving abstractionrules for partition analysis, mutation testing, and scenarios.

Third, it is demonstrated that abstract formal specifications can be used as auto-matic testing oracles for concrete implementations. Here, classical data-refinementtechniques are combined with features of modern tools in order to evaluate testresults with respect to a formal specification.

iii


4/151

iv


5/151

Zusammenfassung

In der Vergangenheit der Informatik konnte manchmal eine Kluft zwischen denAnhangern des Testens und der formalen Methoden festgestellt werden. Testerglaubten nicht an die Anwendbarkeit von formalen Verifikationsmethoden in derrealen Welt. Befurworter von formalen Methoden wiederum, konnten die Testtech-

niken nicht als adaquate Verifikationsmethode akzeptieren. Heute aber, schliet sichdie Kluft, und diese Dissertation tragt dazu bei, diese zuuberbrucken.

Andere haben bereits vorher im Bereich Testen basierend auf formalen Spezifi-kationen gearbeitet, aber nur wenige haben die verwandten Entwicklungsmethoden,wie die Programmsynthese, mit in Betracht gezogen. Es wird ein Uberblick dieservorangegangenen Arbeiten gegeben.

In Rahmen dieser Dissertation wurden drei formale Entwicklungsmethoden be-zuglich ihrer Unterstutzung des Testens von computerbasierten Systemen unter-sucht: Leichte Ansatze zu formalen Methoden, der Refinement-Kalkul und Daten-verfeinerungstechniken. Das Resultat sind drei neue Testtechniken.

Erstens: Ein leichter Ansatz des Testens, basierend auf ausfuhrbaren formalen

Spezifikationen, wurde in zwei industriellen Experimenten im Bereich der Sprachver-mittlung zur Flugkontrolle angewandt. Abstrakte VDM-Prototypen wurden inter-pretiert, um existierende Testfalle zu bewerten und zu erweitern. Die ermutigendenErgebnisse zeigen, da Formalitat essentiell fur das Testen von hoch-qualitativenSystemen ist. Exakte Zahlen der in den Testdokumenten gefundenen Fehler unddes Aufwands werden angegeben.

Zweitens: Ein Hauptbeitrag dieser Arbeit ist ein neuer Ansatz, um Testfalle zuberechnen. Er basiert auf der Beobachtung, da Testfalle Abstraktionen von System-spezifikationen sind. Der Refinement-Kalkul von Back und vonWright wird verwen-det, um formale Abstraktionsregeln zu formulieren. Mit diesen Regeln konnen kor-

rekte Testfallszenarien aus formalen Spezifikationen abgeleitet werden. Der Vorteildieses Abstraktionsansatzes ist es, da sowohl einfache Eingabe-Ausgabe Testfalleals auch Testszenarien behandelt werden konnen. Es werden Regeln fur Partitions-testen, Mutationstesten und Szenarien angegeben.

Drittens: Es wird gezeigt, da abstrakte formale Spezifikationen als automatischeTestorakel fur konkrete Implementierungen verwendet werden konnen. Hier werdenklassische Datenverfeinerungstechniken mit den Moglichkeiten moderner Werkzeugekombiniert, um Testergebnisse bezuglich einer formalen Spezifikation auszuwerten.

v


6/151

vi


7/151

Acknowledgments

I would like to express my gratitude to all those people who have supported methroughout my research.

First of all, I wish to thank Prof. Peter Lucas, my supervisor, for his help andpatience during the course of this work. It was him, who introduced me to formal

development methods and has given me the opportunity to work as his assistant.Then, I would also like to thank my colleague Andreas Kerschbaumer, who shared

in many fruitful discussions his opinions and points of view with me.Thanks belong also to my students in software technology. It has been their

constructive feedback on the subject, that convinced me that the formal techniquespresented in this thesis might be accepted by well-educated engineers.

Especially, I want to thank my students Johann Horl and Andreas Gerstingerwho took part in the two industrial experiments at Frequentis. It has been theirability that made the experiments a success. Both experiments have been fundedby Frequentis.

Finally, I am indebted to my wife, Kathrin, for her support and for encouraging

me all the way. Thanks,

Bernhard K. Aichernig

vii


8/151

viii


9/151

Contents

1 Introduction 11.1 Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1.1 The Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1.2 Black-Box and White-Box Testing . . . . . . . . . . . . . . . . 31.1.3 Test Automation . . . . . . . . . . . . . . . . . . . . . . . . . 41.2 Formal Development Methods . . . . . . . . . . . . . . . . . . . . . . 5

1.2.1 Formal Specification Techniques . . . . . . . . . . . . . . . . . 61.2.2 Design Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 81.2.3 Formal Methods and Testing . . . . . . . . . . . . . . . . . . . 9

1.3 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.4 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.6 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.7 Structure of this Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2 Review of Related Work on Testing 152.1 Partition Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.1.1 Domain Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 152.1.2 DNF-Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . 162.1.3 Separation of the Input . . . . . . . . . . . . . . . . . . . . . . 192.1.4 Structural Partitioning . . . . . . . . . . . . . . . . . . . . . . 20

2.2 Mutation Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.3 Structuring Test-Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2.3.1 Classification Trees . . . . . . . . . . . . . . . . . . . . . . . . 222.3.2 Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2.4 Test Sequencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.4.1 Finite State Machine (FSM) Calculation . . . . . . . . . . . . 232.4.2 Refinement of a FSM . . . . . . . . . . . . . . . . . . . . . . . 23

2.5 Test Oracles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.6 Algebraic Specifications . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.6.1 Testing a Data Type Implementation . . . . . . . . . . . . . . 252.6.2 Selection Strategies . . . . . . . . . . . . . . . . . . . . . . . . 26

2.7 Behavioral Specifications . . . . . . . . . . . . . . . . . . . . . . . . . 28

ix


10/151

x CONTENTS

2.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282.8.1 A New Approach to Test Strategies . . . . . . . . . . . . . . . 282.8.2 Algebraic vs. Model-Based Specifications . . . . . . . . . . . . 29

2.8.3 Taking the Dynamic Behavior into Account . . . . . . . . . . 302.8.4 How Much Does It Cost? . . . . . . . . . . . . . . . . . . . . . 302.8.5 New Tools New Oracle Approaches . . . . . . . . . . . . . 30

3 A Light-Weight Approach to Testing 333.1 Light-Weight Formal Methods . . . . . . . . . . . . . . . . . . . . . . 333.2 VDM and Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.2.1 The Vienna Development Method . . . . . . . . . . . . . . . . 353.2.2 IFAD VDMTools . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.3 Test Design with Abstract Prototypes . . . . . . . . . . . . . . . . . . 38

3.3.1 Abstract Prototypes . . . . . . . . . . . . . . . . . . . . . . . 383.3.2 The Light-Weight Test Approach . . . . . . . . . . . . . . . . 393.4 Formal Specifications as a Catalyst in Validation . . . . . . . . . . . . 41

3.4.1 Issues in the Requirements and Test Documentation . . . . . . 413.4.2 A Catalyst in Validation . . . . . . . . . . . . . . . . . . . . . 42

3.5 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 42

4 Industrial Experiences 434.1 The Application Domain . . . . . . . . . . . . . . . . . . . . . . . . . 43

4.1.1 Air Traffic Control . . . . . . . . . . . . . . . . . . . . . . . . 434.1.2 Voice Communication in Air Traffic Control . . . . . . . . . . 44

4.1.3 A Network of Voice Communication Systems . . . . . . . . . . 454.2 Experiment I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.1 Requirements Validation . . . . . . . . . . . . . . . . . . . . . 464.2.2 Test Case Validation . . . . . . . . . . . . . . . . . . . . . . . 504.2.3 Remarks on Experiment I . . . . . . . . . . . . . . . . . . . . 53

4.3 Experiment II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554.3.1 Requirements Validation . . . . . . . . . . . . . . . . . . . . . 554.3.2 Examples of Requirement Issues . . . . . . . . . . . . . . . . . 564.3.3 Test-Case Validation . . . . . . . . . . . . . . . . . . . . . . . 594.3.4 Remarks on Experiment II . . . . . . . . . . . . . . . . . . . . 60


5 Test-Cases are Abstractions! 635.1 The New Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5.1.1 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.1.2 Test-Design as a Formal Synthesis Problem . . . . . . . . . . . 63

5.2 The Refinement Calculus . . . . . . . . . . . . . . . . . . . . . . . . . 645.2.1 Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645.2.2 Example Contracts . . . . . . . . . . . . . . . . . . . . . . . . 65


11/151

CONTENTS xi

5.2.3 Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.2.4 Refinement and Abstraction . . . . . . . . . . . . . . . . . . . 68

5.3 Test-Cases as Abstractions . . . . . . . . . . . . . . . . . . . . . . . . 69

5.3.1 Input-Output Tests . . . . . . . . . . . . . . . . . . . . . . . . 695.3.2 Non-Deterministic Test-Cases . . . . . . . . . . . . . . . . . . 715.3.3 Partition Tests . . . . . . . . . . . . . . . . . . . . . . . . . . 71

5.4 Testing Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . 725.4.1 User Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . 735.4.2 Iterative Choice . . . . . . . . . . . . . . . . . . . . . . . . . . 735.4.3 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74


6 Testing Strategies 77

6.1 Partition Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776.2 Structural Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . 806.3 Mutation Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.4 Calculating Scenarios for Testing . . . . . . . . . . . . . . . . . . . . 83

6.4.1 Critical Remarks on FSM approaches . . . . . . . . . . . . . . 836.4.2 Compositions . . . . . . . . . . . . . . . . . . . . . . . . . . . 846.4.3 Scenario Synthesis . . . . . . . . . . . . . . . . . . . . . . . . 846.4.4 Scenario Based Testing Strategies . . . . . . . . . . . . . . . . 84

6.5 Example: Process Scheduler . . . . . . . . . . . . . . . . . . . . . . . 856.5.1 Interactive Process Scheduler . . . . . . . . . . . . . . . . . . 856.5.2 Interaction Partitioning . . . . . . . . . . . . . . . . . . . . . 87

6.5.3 Compositions . . . . . . . . . . . . . . . . . . . . . . . . . . . 886.5.4 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89


7 Automated Test Result Verification 917.1 From Formal Specifications to Oracles . . . . . . . . . . . . . . . . . 91

7.1.1 VDM-SL Specifications as Oracles . . . . . . . . . . . . . . . . 917.1.2 The Homomorphism of Abstraction . . . . . . . . . . . . . . . 93

7.2 Automated Test Evaluation . . . . . . . . . . . . . . . . . . . . . . . 947.2.1 Code-Generation of Oracles . . . . . . . . . . . . . . . . . . . 947.2.2 Interpretation of Oracles with the Dynamic Link Facility . . . 967.2.3 CORBA Oracles . . . . . . . . . . . . . . . . . . . . . . . . . 96


8 Case Study 998.1 Roles in Voice Communication . . . . . . . . . . . . . . . . . . . . . . 99

8.1.1 An Abstraction of the Systems State . . . . . . . . . . . . . . 1008.1.2 Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

8.2 Tester-System Interactions . . . . . . . . . . . . . . . . . . . . . . . . 102


12/151

xii CONTENTS

8.3 DNF-Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1058.4 Scenario Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

8.4.1 Interaction Compositions . . . . . . . . . . . . . . . . . . . . . 106

8.4.2 Selecting Test Scenarios . . . . . . . . . . . . . . . . . . . . . 1118.5 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 116

9 Concluding Remarks 1199.1 Research Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

9.1.1 Light-Weight Approach . . . . . . . . . . . . . . . . . . . . . . 1199.1.2 Abstraction-Based Test Synthesis . . . . . . . . . . . . . . . . 1209.1.3 Formal Specifications as Oracles . . . . . . . . . . . . . . . . . 121

9.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1219.3 Epilogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

Bibliography 122


13/151

Chapter 1

Introduction

This dissertation presents novel approaches to software testing. The testing tech-niques are based on formal development methods for software. These formal meth-ods usually start with a formal description of the software requirements and refinethis specification into an executable implementation. The advantage is the possibil-ity of proving the correctness of the implementation with respect to its specification.In this work, formal requirements specifications as well as formal development meth-ods are applied in the design, validation and execution of software tests. It turnedout that abstraction techniques are the key to a more systematic testing process.This chapter gives a general introduction into the subject, motivations, aims andscientific contributions of our work.

1.1 Software Testing

1.1.1 The Challenge

Software testing is a challenging task. It should evaluate the software to demonstratethat it meets the requirements. This is difficult, both in theory and in practice.However, software is increasingly being used in our daily life as well as in safety-critical systems where failure could be life threatening. This enforces the need foran improved systematic discipline of testing which must be scientifically justifiable.

Testing and Correctness

From theory it is known that testing cannot guarantee the absence of failures incomplex computer-based systems. The reason is the discrete nature of software.For example, if two test-points in the input-space of a piece of software have beentested successfully, nothing is known about the behavior of the software for otherinputs situated between these test-points. This is fundamentally different to otherengineering disciplines, mainly dealing with continuous physical domains.

1


14/151

2 CHAPTER 1. INTRODUCTION

In order to certify a software for being correct with respect to its requirements, itshould be stimulated with all possible inputs. However, the input-space of complexsoftware is in general too large. Consequently, a representative subset of inputs, the

test-cases, has to be selected. What is considered to be representative is based on ahypothesis. This means that testing does only show that a system behaves correctlywith respect to its requirements for the actual test-cases, the rest is an assumption.Therefore, the purpose of software testing is rather the detection of failures than thedemonstration of their absence. Edsgar Dijksta summarized this in his well-knownstatement: Program testing can be used to show the presence of bugs, but never toshow their absence [DDH72]. In practice, further difficulties with respect to testingarise.

Testing and Requirements

The testing process is strongly dependent on the quality of the requirements doc-umentation available to a tester. User requirements that have not been explicitlydocumented cannot be systematically tested. It is known from practice that ap-proximately 60 percent of all defects in the requirements phase are due to missingrequirements [Per95]. Therefore, the systematic capturing of user requirements is aprerequisite to successful software testing. However, experience shows that require-ment documents written in a natural language tend to be ambiguous and unsound.Consequently, formal specification languages have been designed for capturing re-quirements in an unambiguous and sound software specification. In Chapter 3 ourown experience from two industrial experiments is presented where such formalspecification techniques have been successfully applied for increasing the quality offunctional requirements in a safety-critical domain.

Testing in the Development Process

Testing has to be considered with respect to the development process, too. In theclassical waterfall-model of the software development process, testing is presented asthe last phase [Roy70]. This causes a tendency to cut the testing efforts, if projectsexceed their schedules. The situation for testing is even more dramatic, if it isrealized that most of todays big software projects do not meet their schedules, asreported in [Gib94] and more recently in [GSPC99]. Consequently, the tests haveto be designed as soon as possible in the development process, after the softwarerequirements are available.

Execution of the test can take place as soon as software units are ready for com-pilation or interpretation. If components are ready for integration, these tests areperformed until the final system-level tests are carried out. The V-model [fSSaCB91]stresses the different levels of testing and shows the relation to the developmentphases explicitly.


15/151

1.1. SOFTWARE TESTING 3

In Boehms spiral process model testing takes a central role [Boe86]. This ap-proach proposes rapid prototyping. A prototype of software is an incomplete imple-mentation of software that mimics the behavior we think the user need [Sta89]. In

rapid prototyping, software is implemented by the incremental development of a pro-totype that is tested and analyzed before new functionality is added. This processmodel is motivated by the fact that testing should be performed as soon as possible.The earlier in the process a fault is discovered and corrected, the cheaper the correc-tion early detected faults are cheap faults. Thus, with respect to testing Boehmsays: Do it early! [Boe81]. In Chapter 3, executable formal specifications are pro-posed as a prototype for an early validation of requirements as well as system-leveltest-cases.

Becks recent Extreme Programming approach is even more radical [Bec99]. Herejects the classical process models. However, testing plays a central role in Extreme

Programming. One of its rules is that for each class, sufficient test-routines haveto be implemented as a part of the class prior to the methods design. In addition,more frequent regression tests are proposed after each working day of a program-ming team, when the software has been extended or small changes took place. Theadvantage is twofold: First, the scheduling problem is solved, since testing is anintegral part of the design and implementation process. Second, the testing routinesact as a formal specification of the class methods. This will become more obvious inChapter 5, when it is demonstrated that test-cases are in fact formal specifications.

1.1.2 Black-Box and White-Box Testing

This thesis is about test techniques. A test technique or test strategy is a systematicmethod used to select and/or generate tests to be included in a test-suite. Twofundamentally different categories of test techniques exist, black-box and white-boxtechniques. Black-box test techniques are based on requirements. They are thereforealso called behavioral or functional techniques. Testing done under a black-boxstrategy is called black-box testing. Black-box test strategies do ignore the internalstructure of the object under test. Thus, a black-box strategy does not consider howthe test-object have been implemented, but what its requirements are. The subjectof this thesis is black-box testing. For a more detailed introduction to black-boxtechniques we refer to [Bei95].

In white-box testing, the tests are derived from the structure of the tested object.Consequently, this testing method is also called structural testing. Well-knownexample test strategies of this kind are to execute every statement at least once(statement testing), to execute every branch at least once (branch testing), and totest the use of all data object (data-flow testing). For a detailed presentation andcomparison of these techniques we refer to [Bei90] and [Mye79].

In this thesis it is shown that white-box test strategies can be applied to require-ments, if these requirements are captured in a formal specification. This means thatstructural techniques can be applied for black-box testing, if the structure of the


16/151


specification is analyzed. One approach is to specify an abstract and executableprototype solely based on the requirements. Then, for example, branch testingis possible. This method and its encouraging results are presented in Chapter 3.

A more advanced method derives test-cases directly from a more abstract non-executable specification by applying synthesis rules to the specification. Here, thestructural techniques are incorporated into the synthesis rules. This novel approachis shown in Chapter 6.

1.1.3 Test Automation

Testing is expensive and its efforts are often underestimated. The costs of testingare dependent on the number and subtlety of detected faults during testing. Brookspoints out that software engineers are usually too optimistic with respect to failure

rates, Therefore, testing is usually the most miss-scheduled part of programming[Bro75]. He proposes that thehalfof the schedule should be devoted to testing: 1/3planning, 1/6 coding, 1/4 component test and early system test, 1/4 system testwith all components in hand. Another much-quoted heuristic rule distributes theefforts as follows: 40 % design, 20 % coding, 40% testing [Per95]. The figures givenby Boehm [Boe75] distinguish between different types of software systems. He foundthat for testing of command and control systems 34 %, space-born systems 46 %,operating systems 50 %, scientific systems 30 %, and business systems 28 % of thetotal development costs are spent. Boehms figures from 1975 are still in use as theirpresentation in the standard literature for software engineering [Som92] indicates.Although, more elaborate regression models for estimating the efforts of testing exist

(see e.g. [Lyu95, MIO90]), these heuristic data demonstrate how expensive testingcan be.

One way to decrease these huge efforts is the automation of testing. Otherarguments for test automation summarized in [Bei95] are that

manual tests are error-prone: a typical test-case contains 250 characters.Statistics from professional keypunchers indicate that manually entered testshave at least one keystroke-error. Furthermore visual test-verification is likelyto fail;

manual testing leads to false confidence: manual test-execution is difficult and

boring. Often effort and accomplishment are equated, giving the impressionthat hard work means well tested;

dependability requirements demand automated testing: all statistically validmethods of estimating software dependability are based on fully automatictest methods for executing enough test-cases;

a test has to be run several times: in practice, tests are executed more thanonce. The reasons are that a test-design may have been wrong, a failure


17/151


18/151


Formal development methods are mathematical methods for producing software(and today digital hardware). Due to the discrete nature of software, the mathemat-ics needed is fundamentally different to that used in other engineering disciplines

mainly dealing with continuous domains. In software engineering, formal logic andalgebraic structures are the most important mathematical tools. The termformalmeans the use of methods of reasoning that are sound by virtue of their form andindependent of their content.

The UK Military of Defense standard on the procurement of safety-critical soft-ware [oD89] defines a formal method as:

A software specification and production method, based on a mathe-matical system, that comprises: a collection of mathematical notationsaddressing the specification, design and development phases of software

production; a well-founded logical system in which formal verificationand proofs of other properties can be formulated; and methodologicalframework within which software may be verified from the specificationin a formally verifiable manner.

From the definition above it can be concluded that formal specifications of require-ments and formal verification of software are the corner-stone of a formal method.However, recently light-weight approaches to formal methods have gained popularityfor transferring these techniques into industry [ELC+98, JW96, Jon96]. In this con-textlight-weightindicates that the methods focus is rather on specification than onformal proofs. In Chapter 3 these approaches are examined with respect to testing.

In the following, an overview on the different formal techniques should give anintroduction to the advantages mathematics provides for software development and for testing. A more detailed introduction to formal development methods canbe found, for example, in the formal methods guidebooks of NASA [NAS95, KK97].

1.2.1 Formal Specification Techniques

A formal specification is the documentation of the functional requirements of acomputer-based system in a mathematical language, the specification language.Since, the semantics of a specification language is formally defined, formal spec-ifications can capture software requirements in an unambiguous and yet abstractstyle.

Dependent on the mathematical apparatus used, several different specificationtechniques have been established. In this work, the following three fundamentallydifferent approaches to specification are distinguished: model-based, algebraic andbehavioral specifications. The reason for this classification are the completely dif-ferent consequences of these techniques for testing (see the survey of related workin Chapter 2).


19/151

1.2. FORMAL DEVELOPMENT METHODS 7

Model-based Specification

Model-based specification techniques use mathematical objects for modeling a sys-

tems state explicitly. Numbers, sets, sequences, finite mappings, Cartesian prod-ucts and composite types are typical types of objects that are applied to model asystem on an adequate level of abstraction. In contrast to conventional program-ming languages, logical constraints on the types, here called data-invariants, maybe added to specify additional properties of a model. Functionality is specified bystate-changes expressed by functions or relations on the state model. Consequently,these specifications are also called state-based.

Prominent formal methods of this kind are VDM [Jon90, FL98], Z [Spi88, Spi92,WD96], and B [Abr96]. Furthermore, the formalism of Back and von Wrightsrefinement calculus is model-oriented [BvW98]. The testing techniques presentedin this work are solely based on model-based specification techniques. However, for

the discussion of the related work the next chapter, a basic knowledge of the othertwo specification methods is necessary.

Algebraic Specification

In an algebraic specification technique an object class or type is specified in termsof the relationships between the operations defined on that type. Unlike in model-based specifications, nothing is said about the internal representation of the definedobjects. Hence, in general, algebraic specifications are more abstract than model-based specifications.

Algebraic specification was brought to prominence by Guttag [Gut77] in thespecification of abstract data types. Various notations for algebraic specificationhave been developed, including OBJ [FGJM95] , Larch [GHW85, GH93] and CASL[Mos99]. In RAISE [Gro92, Gro95], algebraic specification and the model-basedtechniques from VDM have been combined. A notation independent presentationof the subject can be found in [HL93].

Behavioral Specification

Abstract specifications of the behavior of a system are called behavioral specifica-tions. This type of formalisms describe ratherhowa system behaves over time thanwhatthe possible computations can be. Control and concurrent systems are typicalapplication domains for this specification style.

If the description of the behavior should mainly focus on the set of possiblereachable states, finite state machines are used to capture the dynamic state-changes.

If the specification is focused rather on the possible events that can occur in asystem than on its reachable states, then labeled transition systems are in use. Theprocess algebras of CSP [Hoa85] and the -Calculus [Mil99] belong to this class ofspecification techniques.


20/151


Further well-known behavioral specification techniques are Petri-nets [Pet77,Pet81], and Estelle, LOTOS, SDL, [Tur93], well-suited for defining communicationprotocols.

However, some development methods, like UML, use diagrammatic behavioralspecifications that lack a formal semantics. These methods are not considered to beformal by the author.

Behavioral specification techniques can be combined with non-behavioral meth-ods. VDM++ (see Chapter 3) is an example of combining model-based with be-havioral specifications, LOTOS an example of the combination with the algebraicspecification style.

1.2.2 Design Methods

A toolbox of a formal development method should include both, a formal specifi-cation language as well as a formal design technique. Usually, a formal design isstarted from an abstract formal specification and then successively refined into anexecutable implementation. This process is called refinement. In model-based tech-niques, both, the data model (data refinement, also called data reification) as wellas the operations (operational refinement) are refined.

Two different formal approaches to the refinement problem exist, known asinventand verifyandprogram synthesis. In this thesis it is demonstrated how both methodscan be adapted for the application in testing.

Invent and Verify

In this classical design paradigm, a refined version of the specification is invented andafterwards verified that the refinement is correct with respect to its specification.Each method provides the proof obligations necessary for proving the refinementrelation. For example, the M of VDM (Vienna Development Method) appliesinvent and verify[Jon90].

It is demonstrated in Chapter 7 that this approach can be adapted for testingsuch that an abstract formal VDM specification can be used as a test oracle.

Program SynthesisProgram Synthesis is the systematic transformation of a specification into an im-plementation. Formal transformation rules guarantee that the implementation iscorrect by construction if these synthesis rules are respected.

Dijkstra has been the first who proposed this approach for the development ofstate-based programs [Dij76]. Additional heuristics for calculating programs havebeen presented by Gries [Gri81]. Further progress in program synthesis has beenmade by Back and von Wright [BvW98], and Morgan [Mor90]. They extended the


21/151

1.2. FORMAL DEVELOPMENT METHODS 9

approach to refinement calculi with their own language for defining contracts andlarge sets of rules for calculating programs.

Program synthesis have been applied to functional programs, too: Martin-Loef

presented an intuitionistic type theory for deriving functional programs from typedeclarations [ML85]. Manna and Waldinger [MW92] developed a tabular prooftechnique, such that functional programs are developed as side effects during theproof.

A formal refinement calculus enables the development of tools that supportthe interactive synthesis of programs. In [vW94] an automation of Back and vonWrights refinement calculus have been described. Coen adapted Martin-Loefs workand developed a constructive type theory for the theorem prover Isabelle [Coe92].

In this thesis it is the first time that program synthesis is applied to test-casesynthesis (Chapter 5 & 6). Here, Back and von Wrights refinement calculus is used.

1.2.3 Formal Methods and Testing

In the past of computer science, sometimes a gap between the testing and the formalmethods community could be observed. Testers did not believe in the applicability offormal verification techniques to real world problems, and formal methods advocatescould not accept testing-techniques as an adequate verification method for producingcorrect software. However, recently the gap is closing.

Today, light-weight approaches to formal methods invite engineers to take theadvantages of formal specification techniques without focusing alone on correctnessproofs. Having precise and unambiguous formal specifications available, is the pre-

requisite in order to automate black-box testing. The importance of black-boxtesting has increased, due to the growing awareness that a combination of black-and the more traditional white-box testing uncovers more defects than applying onetechnique solely. Especially, the object-oriented paradigm and the increasing use ofComponents Of The Shelf (COTS), shifted the focus of interest towards black-boxapproaches [Bei95].

A further reason for an easier integration of testing methods and formal methodshas been the success of model-checking. Model-checking is in fact exhaustive testingof a formal model. For model-checking to be applicable to software, abstract modelsof the software with finite data domains have to be build. These abstractions have tobe proven to be correct. As a consequence, integrated tools with model-checkers andtheorem provers are under development, as presented recently in [Rus00, BGL+00].

In this work the application of formal development methods to system-level test-ing is considered. This part of the testing process will always be necessary. Forproving a computer-based system to be correct, it would be necessary to model thewhole environment the system interacts with which is impossible.

The Philosophical Point of View. The relation between testing and formalspecification is reflected in the philosophical points of view of famous Austrians:


22/151


Ludwig Wittgenstein, the Wiener Kreis, and Sir Karl Popper. Wittgenstein and theWiener Kreis proposed logic for describing and solving scientific problems [Wit21,Kra50]. Popper realized that only through falsification truth can be obtained. He

consequently proposed his demarcation criterion according to which any empiricallyrefutable consistent proposition should be regarded as scientific. For theories to beobservationally falsifiable, they must be stated precisely and clearly [Pop35, Pop63].Their results can be applied to software development as well, with the followingconsequences:

A formal specification of requirements helps to solve systematically the prob-lem of software development by offering the apparatus of logic (or more generalmathematics).

The validation of the formal specification as well as the implemented solutiononly is feasible through falsification which is testing.

For being falsifiable (or testable), the requirements description has to be un-ambiguous and sound ensured by formal specification and verification tech-niques.

1.3 Objective

The overall objective of this thesis is to contribute to the combination of formaldevelopment techniques and testing. Despite the fact that a lot of previous workrelated to formal methods and testing exists, this thesis should demonstrate thatmore can be done. The aim of this thesis is to examine the formal developmenttechniques with respect to their usefulness in the testing process. Three formaltechniques are considered from a testers perspective:

Light-weight formal methodsrecently gained increased popularity, due to theireasier integration into an industrial environment. This thesis should demon-strate the advantages, costs and feasibility of applying this kind of formaltechniques to testing.

Invent and verifydevelopment techniques are traditionally applied in formalprogram verification with proofs. Where proofs are not feasible, but formalspecification have been successfully implemented, it is our aim to show thatthese more abstract specifications can be used as test oracles.

Program synthesis techniques ensure the development of correct programs.Our aim is to develop system-level test-cases with the same level of rigor.This thesis should prove that program synthesis techniques can be applied toproduce correct test-cases.


23/151

1.4. MOTIVATION 11

It has not been our objective to produce a ready to use tool-set for testing software,but to examine and develop the techniques necessary for developing new testingtools tools more advanced than available today.

1.4 Motivation

During the last few years, the interest in formal software development has been grow-ing rapidly. However, despite their long existence since the 60s, formal methods arenot yet widely accepted and applied in the software industry. Our overall motivationfor this work is to facilitate this technological transfer and thus to contribute to theefforts in turning software development into a mature engineering discipline.

From our experience in industry, we are convinced that test-engineers are apromising target group for introducing formal methods. Especially, in the domain

of safety-critical systems, testers are really concerned about the effectiveness andmaturity of their methods. In this thesis both, practical results that can be applieddirectly, and new theoretical results that need further adaption and tool support tobe transfered into industrial practice are presented.

The central role of testing in software engineering demands scientifically justi-fiable testing methods. Today, test-plans are rather complex procedures, includingtest-cases with several sequences of interaction with a system. In a systematic soft-ware development process, these test-cases have to be documented correctly andunambiguously. If a tester cannot rely on his test-plans, they are rather useless.This was the motivation for our work on the application of program synthesis tech-

niques to test-case synthesis.A further motivation of our work is the fact that test-cases can be used forvalidating a formal specification by users not familiar with formal notations. Thepremise for such a test-case based validation is that the test-cases have been correctlyderived from the specification.

1.5 Contributions

In this thesis three formal development techniques are examined, if and how theysupport the testing of computer-based systems: Light-weight approaches to formal

methods, the refinement calculus, and data-reification techniques. As a result, noveltesting techniques are presented.The scientific contributions of this work can be summarized as follows:

A light-weight approach to specification based testing has been developed.The approach is based on executable VDM specifications and the commercialIFAD VDMTools. Two industrial experiments demonstrate its successful ap-plicability. Figures of the efforts and efficiency of the method are presented.The experiments have been designed, supervised and controlled by the author.


24/151


The actual specification work has been done by two master thesis students ofthe author.

A main contribution of this work is the realization and formal proof that test-cases are in fact abstract formal specifications. Our abstraction theorems holdfor simple input-output test-cases as well as for test scenarios consisting ofseveral steps of interaction.

The observed property that test-cases are abstractions of specifications leadto a new test synthesis approach based on program synthesis. In this thesis itis the first time that the refinement calculus is applied to the problem of testsynthesis. The result is an abstraction calculus for testing in order to calculatetest-cases according to a testing strategy.

New formal synthesis rules for the test strategies of partition testing, mutationtesting and test-sequencing are presented and formally proved.

Finally, it is shown how formal specifications can be used as test-oracles. Aframework using VDM and new features of the IFAD VDMTools is discussed.

1.6 Publications

The research for this thesis lead to 10 publications of the intermediate scientificresults.

Our light-weight approach to testing has been described in [HA99, HA00a,HA00b, AGA00]. It has been shown that this validation approach can beextended to hybrid systems as well in [AK00]. The article in the IEEE Software

journal [HA00b] was first submitted to the Fourth International Conferenceon Requirements Engineering (ICRE2000). The program committee selectedthis article as best paper and forwarded it to the IEEE Software journal.

The work on a calculus for test-synthesis lead to the most recent publication[Aic01].

The ideas and techniques to use formal specifications as test-oracles can befound in [Aic98, Aic99a, Aic99b, Aic99c].

1.7 Structure of this Thesis

Chapter 1 introduces to the general subject of this thesis: software testing andformal development methods. Furthermore, the objectives, the motivations, asummary of contributions, and the structure of this thesis are presented.


25/151

1.7. STRUCTURE OF THIS THESIS 13

Chapter 2 summarizes the related work on testing based on formal developmentmethods. In this chapter, the existing test-case generation techniques basedon formal specifications are explained. The summary distinguishes between

testing techniques based on model-based, algebraic, and behavioral formalspecifications. Based on this technical review of the related work, the newcontributions of this thesis are discussed.

Chapter 3 presents a light-weight approach to testing based on executable formalspecifications. The technique uses an executable VDM specification of thesystem requirements and IFAD VDMTools for validating the design of thesystem-level test-cases.

Chapter 4 Here, two industrial experiments in the domain of air-traffic control arepresented in which the light-weight approach have been applied. Its applicabil-

ity in an industrial environment is demonstrated by the encouraging results ofthese experiments. Detailed empirical data support our claims. Furthermore,example serve to illustrate the proposed method.

Chapter 5 develops the foundation for a new test-synthesis calculus. It is provedthat test-cases are in fact abstract formal specifications. This insight holds forsimple input-output test-cases as well as for test scenarios consisting of severalsteps of user interaction. The refinement calculus is used to formulate andprove this abstraction property.

Chapter 6 presents formal rules of the new abstraction calculus for test synthesis.This is the first time that program synthesis techniques are applied to thetest-case synthesis problem. Different test strategies are formulated in thisnew formal framework. Rules for partition testing, mutation testing and thecalculation of test scenarios of interactive systems are defined.

Chapter 7 shows how abstract model-based specifications can be used as test or-acles for an automatic evaluation of the test results. The technique is basedon VDM but can be applied to other model-based formal methods. It is ex-plained how the invent and verifyproof technique can be applied to testing.Furthermore, it is shown that the new features of IFAD VDMTools facilitatethe implementation of this oracle framework for testing.

Chapter 8 demonstrates the new test-calculus by means of a case-study. A voicecommunication system for air-traffic serves as an example for showing theapplication of the different testing techniques.

Chapter 9 gives a brief summary of this work. Furthermore, the final conclusionsare drawn by the author. The thesis closes with an outlook and motivationof the possible future work that may be initiated by the contributions of thisthesis.


26/151



27/151

Chapter 2

Review of Related Work onTesting

In this chapter a review of related work on testing based on model-based formalmethods is presented. Since our approaches to testing is based on the Vienna De-velopment Method (VDM) and the refinement calculus, the previous work on testingwith model-based specifications is most closely related to our own work. However,at the end of this chapter, testing methods based on algebraic and behavioral spec-ifications are presented. The discussion of the testing techniques of these methodsshows the different challenges with respect to testing, depending on the choice ofthe specification style.

2.1 Partition Testing

The most common strategy for generating tests from model-based specifications ispartition testing. Partition testing applies strategies based on dividing the set of allpossible inputs into equivalence classes under an appropriately defined equivalencerelation [HT88, OB88, RC81]. In the late 80s the formal methods communityrealized that partition testing techniques may be applied to formal specifications.Especially, for state-based specifications the adaption of well-known techniques fromsource-code based testing is straight forward.

2.1.1 Domain Testing

Scullard reports in [Scu88] on the industrial use of VDM for selecting test cases inorder to validate a micro-processor. More precisely, domain testing is applied toVDM specifications.

Domain testing analyzes the input-domain of a systems functions, i.e. its signa-tures. The strategy is based on a simple processing model in which the system undertest accepts an input vector, which then goes through a classification process, e.g.

15


28/151

16 CHAPTER 2. REVIEW OF RELATED WORK ON TESTING

a case statement. Each of these cases is processed separately. The main concern indomain testing is the correctness of the classification process as contrasted with thecorrectness of the processing done for each case. Usually, in domain testing, domains

are defined by a set of boundary inequalities. A simple example of a domain testingtechnique is the choice of extreme values of input parameters [Coh78, JW94].

Since all input values of a model-based specification have a concrete represen-tation, domain analysis can be performed on it. Furthermore, preconditions, data-invariants, and post-conditions can be used to synthesize subdomains. Scullardstest-case extraction process for VDM can be summarized as follows:

1. Domain reduction: Reduction of the signature of the function to basic typesonly. Compound types are replaced by the Cartesian product of their compo-nents. Furthermore preconditions and data-invariants are taken into accountto reduce the input-domain further.

2. External analysis: Selecting a set of typical values for each basic type of thereduced signature. For ordered types extreme values and one or more centralvalues are chosen, for unordered types all values are selected.

3. Internal analysis: Identification of possible cases by skolemizing the postcon-dition of the function.

Scullard gives an example of his technique and discusses that an automated testgeneration system would be useful. However, he did not formulize the underlyingpartitioning method. This has been done in the subsequent work on VDM andtesting by Dick & Faivre.

2.1.2 DNF-Partitioning

Dick & Faivre. One of the most cited works on test-case generation from model-based specifications is [DF93]. Dick and Faivre realized that transforming a formalVDM specification into a disjunctive normal form (DNF) constitutes an adequatepartitioning method. A Prolog tool has been developed by them that rewrites data-invariants, preconditions and postconditions according to the following rules for aDNF construction:

A B(A B) (A B) (A B)

A B A B

For model-based specifications this partitioning process is extended to type asser-tions of the form x: Tdenoting the values of variable xto be of type T. A specifi-cation containingx: Tcan be further partitioned into special values v: Tand othervalues:

x:T (x=v x=v) x:T


29/151


30/151


Helke et al. [HNS97] demonstrate that a theorem prover like Isabelle [Pau94]can be used to generate such test-classes based on DNF rewriting. It is shown howconstructive proofs may be used to calculate the partitions of a Z representation in

the HOL instance of this generic theorem prover. HOL stands for higher order logicand is based on Alonzo Churchs work on a simple theory of types [Chu40].

Van Aertryck. In [Aer98] the tool-supported method CASTING for automat-ically generating test-cases is presented. CASTING is generic in the sense thatit is applicable to different model-based specification languages as well as to pro-gramming languages. This is realized by an internal representation using attributegrammars in which the test-case selection strategies can be encoded. An instan-tiation of CASTING for the B-method is presented, where DNF partitions can begenerated automatically from Bs abstract machine specifications. The tool uses a

constraint solver for finding actual test-cases.

Behnia & Waeselynck. A problem concerning DNF partitioning and classicallogics is discussed in [BW99]: DNF rewriting of B specifications may lead to unde-fined partition specifications.

Consider a specification expression, where fis a partial function.:

(xdom(f) f(x) =y)

Applying the DNF approach would result in three disjoint partitions:

(x /dom(f) f(x)=y) (x /dom(f) f(x) =y) (xdom(f) f(x)=y)

It can be easily seen that the first two cases are undefined, since f(x) is undefined.The authors propose an extended deduction system, where definedness is treatedexplicitly by an operator denoting well-definedness. Such problems have beenforeseen by Jones and are solved by extending VDMs logic to a three-valued logicof partial functions, where the third logic value is undefinedness [Jon90, BFL+94].Consequently, for VDM this problem does not exist the partitions above arewell-defined in VDM-SL. This shows that Jones is right, when he says in [Jon98]:

The lessons which I believe it is important to draw from my series ofattempts to find a satisfactory way to reason about partial functionsinclude the fact that a real difficulty should be faced rather than ignored( I can accept any approach to this problem much more readily thanpretending it does not exist).

Behnia & Waeselynck solve this problem for the B method by designing an alterna-tive set of partitioning heuristics that do not generate undefined partition predicates.Their approach is described in Section 2.1.4.


31/151

2.1. PARTITION TESTING 19

2.1.3 Separation of the Input

Hieron

Hieron proposes another heuristic of partitioning model-based specifications that iscoarser than pure DNF-partitioning [Hie97]. His partitioning technique rewrites Zoperation predicates as (X1 Y1) (Xn Yn) where

predicates Xirefer only to the inputs;

predicates Yirefer to the outputs.

This form of the predicates separates the description of the input Xi from the op-erational part Yispecified by a relation. Since Xi and Yimay contain disjunction,the rewritten operation predicate is not necessarily in DNF. The motivation is not

to separate constraints that apply to the same category of variables.For example, let us reconsider an operation for calculating the maximum of twonumbers with the formal input parameters x, y and the output parameter max.Then the predicate that specifies the computation of the maximum ofx and y

((x>y x=y) (max=x)) ((x


32/151


33/151


34/151


2.3 Structuring Test-Cases

In general, an automated DNF-partitioning produces many unstructured test-cases.

Therefore, some kind of user interaction is necessary for selecting and structuringthe relevant test-cases in a test suite. Classification trees and abstraction hierarchiesare suitable tools for handling this problem, as explained in the following sections.

2.3.1 Classification Trees

In order to enhance DNF partitioning, the classification tree method is applied forselecting only interesting partitions in [SCS97]. A classification tree is a graphicalrepresentation of the stepwise partitioning of an input domain into relevant parti-tions [GG93, GWG95]. The root of the tree represents the complete input domain

of the system under test. Its children nodes are predicates representing the parti-tions. The tree grows as the partitions are refined. The leaves represent the finalpartitioning of the input domain and serves as the head of a selection table fromwhich the actual test-cases are chosen. The presented tool provides a graphical userinterface that represents the partitions of a Z specification in such a classificationtree. The advantage of this approach is that the partitions are well structured andthat only relevant ones can be refined or chosen to be final interactively. This solvesthe test-case explosion problem due to DNF partitioning.

2.3.2 Abstraction

Stepney realized the abstraction relation between Z specifications of test-cases andobject-oriented Z specifications of operations [Ste95]. Test-case specifications areabstract in the sense that only relevant input and before-states are tested and thatonly interesting consequences of an operation should be observed. Formally, thismeans that for designing a test-case specification of an operation, the preconditionis strengthened and the postcondition is weakened. This is the inverse process ofrefinement abstraction. In other words, her method of test suit design works byintroducing non-determinism into the original specification.

Her group developed a tool for interactively calculating such partition abstrac-tions and to structure them for reuse. With this tool, the test-cases of a test suiteare structured in a hierarchy of abstractions visualized in a tree. The root of the treeis the original specification. Following the tree towards its leaves means abstraction.The children are either a set of partitions, or more non-deterministic versions of theparent test-case specifications.

Our work can be seen as an extension of this view on testing. Unlike Stepney, weuse the refinement calculus for defining the abstraction rules for designing test-cases.Furthermore, we cover the sequencing of test-cases that is only mentioned by her.Chapter 5 presents our approach to this topic.


35/151


36/151


37/151

2.6. ALGEBRAIC SPECIFICATIONS 25

If this relation is evaluated it can serve as an oracle, and can decide if a testsucceeds. This is in contrast to using a prototype for calculating the actual square-root which is far more complicated then using the specification above.

In [HM96] such a test environment is described that generates a predicate eval-uator as an oracle from a Z specification.

In Chapter 7 an alternative approach is shown, where implicit VDM specifica-tions are directly used as test-oracles.

2.6 Algebraic Specifications

In the following, an overview of the algebraic testing techniques proposed by Marie-Claude Gaudel et al. [BGM91, BGM92, Gau95, GJ98] is presented. Although,algebraic specifications are not the subject of this thesis, the review of this work

should explain the differences to model-based testing techniques. It has to be men-tioned that Dick & Faivres early testing techniques on VDM have been initiallyinspired by this work [BGM91].

Figure 2.1 shows an algebraic specification of sorted lists in the notation ofLOTOS [ISO89]. Such a specification has two parts: a signature = (S,F), whereSis a finite set ofsortsandFis a finite name ofoperation namesover the sorts inS,and a finite set ofaxioms. The sorts in our examples are List, Elem and Boolean;the new operation sortedis defined by three axioms.

2.6.1 Testing a Data Type Implementation

Testing code against an algebraic specification consists of showing that the finalsystem satisfies the axioms in the specification. To create tests for a given axiom,the variables of the axiom are instantiated with values. To run such a test, theresulting expressions are evaluated. If the results satisfy the axiom then the test ispassed, otherwise it is failed.

An advantage of algebraic specifications in testing is the ease with which testdata are constructed. The constructive nature of the specifications shows how tobuild various instances of the defined data types, which also implicitly include testsequencing information. There is also great potential for automatically generatingtest cases from specifications by steadily building up data from the base cases using

constructor operations.In the SortedList example, a possible test, inspired by the third equation, is:

sorted(cons(A, cons(B, cons(C, nil)))) = (A B) sorted(cons(B, cons(C, nil)))

The corresponding test procedure for testing an implementation Pconsists of twocomputations and one comparison, namely

computing the representation in P of cons(A, cons(B, cons(C,nil))), callingthesorted function in Pon it, and storing the result;


38/151


39/151


40/151


Antoy & Hamlet have presented a framework, where implementations are auto-matically checked against their algebraic specifications [AH00]. In contrast to thealgebraic test approach described above, they evaluate the specification by rewriting

it following the axioms. Here, the test-results of the implementation are mappedand compared to the resulting terms of the specification. Only, simple examples oftesting a set implementation are shown.

2.7 Behavioral Specifications

A specification is calledbehavioral, if the dynamic behavior of a system is describedrather than its data and data transformations. Finite state machines and labeledtransition systems are examples of this kind of specification style. Behavioral spec-ifications are applied in domains where systems have a rather complex state butsimple state transitions, like in communication protocols and control applications.

Behavioral specifications are well suited for automatically generating sequencesof test-cases, since feasible applications of state operations are made explicit. There-fore, Dick & Faivre have shown how to produce a behavioralfinite state machinefrom a computationalVDM specification as has been explained in Section 2.4.

A class of test generation approaches starts directly from such behavioral speci-fications, like finite state machines [Cho78, FvBKG91, LY94] or finite labeled tran-sition systems [dNH84, Hen88, Bri89, PF90, Tre92, Pel96, PS96] .

Another class of approaches combines computational and behavioral specifica-tions. As we do, MacColl and Carrington are planing to develop a testing framework

for interactive systems. They use a combination of Z and behavioral specifications(the process algebras CSP and CCS) for specifying such systems [MC99]. In [GJ98]such behavioral specifications are combined with algebraic testing techniques. Fur-thermore, in [SS98] testing from combined Z and statechart specifications is dis-cussed.

2.8 Discussion

As shown above, a lot of work has been done on generating test-cases from formalspecifications. However, in this thesis it is demonstrated that there is still room for

innovation.

2.8.1 A New Approach to Test Strategies

This thesis seeks to extend the body of work for model-based specifications summa-rized above. A new framework under which all testing strategies, like partitioningand mutation heuristics, can be formally specified, analyzed and used to automatethe generation of test case specifications. In contrast to previous works with the


41/151

2.8. DISCUSSION 29

same goal, e.g. Burtons recent work on Z [Bur00, BCGM00], our mathematicaltreatment of test-cases can handle test-sequencing as well. Unlike the work above,our approach is based on program synthesis techniques. Since test-case generation

is a synthesis process, we find it more natural to use a development method likethe refinement calculus in order to calculate test-cases. The basis to this uniformframework is the view that all kinds of test-cases are mathematical abstractionsof model-based specifications. This new approach to testing will be presented inChapter 5.

Having realized the abstraction relation of test-cases, the refinement calculuscan be applied for formalizing synthesis rules for testing. This is the first time thatBack and von Wrights refinement calculus [BvW98] is applied for the developmentof test-cases. In Chapter 6 test strategies for partitioning, mutation testing aswell as test sequencing are formalized as proof rules in the refinement calculus. It

will be shown that the formalization itself revealed interesting properties of theheuristics that have to be considered, when automation is envisaged. The uniformframework based on the abstraction relation provides an environment under whichthe relationship between various testing heuristics can be investigated in order todevelop more efficient heuristics.

2.8.2 Algebraic vs. Model-Based Specifications

Algebraic specifications are not the topic of this thesis. As explained in the introduc-

tion, one of our motivations for this work is the technology transfer of formal spec-ification techniques into industry. The author shares the opinion of Jones [Jon90]that model-based specifications are more adequate for the practicing engineer andthat algebraic definitions should be left to the formal method experts. Although ingeneral, algebraic specifications are more abstract than model-based specifications,their validation is more difficult for non-mathematicians: It is not easy to determine,if the axioms in a complex algebraic specification are sufficient and sound.

With respect to testing Hayes [Hay86] argues that algebraic techniques are bestsuited to testing primitive data types and that, for more complex abstract datatypes, a model-based specification is simpler. As noted by Gaudel, predicate logicspecifications are more general than algebraic specifications. However, the price of

this generality is the restriction that, in general, only specifications of classes oftest-cases can be generated automatically. Algebraic techniques can generate testdata as well.

It is our opinion, that a higher degree of automation of test-case generationshould not be gained through a specification technique that is more error-pronefor non-experts in formal methods. The experiments presented in Chapter 3 willdemonstrate how error-prone informal specifications are. The situation should notbe made worse by introducing another source of errors.


42/151


2.8.3 Taking the Dynamic Behavior into Account

The following chapter will show that todays safety-critical systems may be highly

interactive as well as involve complex data manipulations. Typical test-cases of suchcomplex systems are scenarios of usage. Thus, a testing strategy must treat both,the input-output relation as well as the possible sequences of interaction.

As has been described above, recent work on model-based specifications hasfocused on the combination of behavioral and computational methods for coveringthe different aspects of functionality. In contrast, this work uses Back and vonWrights contract language for describing the behavior of interactive systems. Ithas been shown that Back and von Wrights refinement calculus covers such type ofsystems in [BMvW99].

The innovation in the work presented here is that the synthesis of test-sequences(scenarios) from a specification is considered as an abstraction problem. Our se-quencing technique does not calculate the complete finite state machine prior totest-case selection and thus differs from the approaches above. In an environment,where change requests are common, calculating the whole finite state machine isnot very efficient. Our method focuses on the impact on the possible scenarios bysystematically analyzing interaction compositions.

2.8.4 How Much Does It Cost?

All the methods presented above rely on formal specifications and thus will not beapplied in industry if formal specifications fail to be accepted. A common questionof the practicing software engineer is: What are the efforts and the payoffs?, orsimply How much does it cost?.

Too few of the previous works answer this question. One of the reasons is thatoften only simple examples serve to demonstrate new research in the area of testing.In this work, we try to answer the question partly by presenting the results of twoindustrial experiments described in the next chapter. In these experiments a light-weight approach to testing has been used that does not represent fundamentallynew techniques. The test method is based on executable specifications. However,the exact figures of the efforts and detected issues are presented. Furthermore, theempirical results have motivated our more theoretical research of Chapter 5 and 6.

2.8.5 New Tools New Oracle Approaches

It has been shown that formal specifications have been used as test oracles before.However, in this area of automating the test process, the availability of new tools isessential. Previous work on oracles has mainly focused on code generation or directlyexecutable specifications. In Chapter 7 we will present a novel oracle technique,where recently available VDM tools are combined with well-known proof techniques


43/151

2.8. DISCUSSION 31

of VDM. This enables the direct use of VDMs pre-post condition specifications astest oracles.


44/151



45/151

Chapter 3

A Light-Weight Approach toTesting

In this chapter a test design method is presented that has been applied in twoindustrial projects. It is a method for designing system-level test cases for complexcomputer-based systems. This light-weight approach is based on executable formalspecifications that capture the user requirements of a system under test.

The details of the two projects and their encouraging results are described inthe next chapter. Here, the method and its foundation are introduced. After adefinition and discussion on light-weight formal methods, the Vienna DevelopmentMethod is described in more detail. Next, it is shown how VDM and VDMTools canbe used in test design. Finally, the relation between our approach and the important

validation aspect in testing are discussed.

3.1 Light-Weight Formal Methods

During the last few years, the interest in formal software development has beengrowing rapidly. One indicator for this growing popularity is the fact that everymain conference on software engineering features sessions about formal methods.

Several reasons can be identified; among others the following three:

the combination of formal and more traditional approaches,

the availability of commercial tools to support the developer,

the beginning awareness that the application of formal methods does not nec-essarily imply the use of formal correctness proofs.

The last item lead to a new class of formal methods that is often called light-weightformal methodsor formal methods light.

33


46/151


47/151

3.2. VDM AND TOOLS 35

Some formal methods facilitate the specification in an explicit style such thata tool can interpret or generate code from this kind of formal specifications. Theadvantage of these executable specificationsis

+ that testing can be used for raising the confidence in the specification.

The disadvantage of executable formal specifications is that

they are in general less abstract than more implicitly defined specifications, likenon-deterministic pre- and postconditions.

Our light-weight approach to testing is based on executable specifications of asystems requirements. The method uses these abstract system prototypes not onlyfor checking the specification, but for:

the validation of the requirements,

the validation of existing system-level test cases,

the design of system-level test-cases, and

the validation of the formal specification.

The formal method that provides such a light-weight application is the ViennaDevelopment Method supported by IFADs VDMTools. The following section willgive a short introduction to VDM and the used set of tools.

3.2 VDM and Tools

3.2.1 The Vienna Development Method

In the following, the Vienna Development Method(VDM) and its specification lan-guages are described. The contents are partly based on [PT89, PvKT91, LHB+96,Lar95].

VDM is a formal method for the description and development of computer basedsystems. It is a model-based method, i.e. its formal descriptions (VDM specifica-tions) consist of a model of the system being constructed.

A system design is generated through a series of specifications, where each spec-ification is more concrete and closer to the implementation than the previous one[Bj89]. Each of these development steps introduces proof obligations which, whenappropriately discharged, ensure the (relative) correctness of the implemented sys-tem.

VDM is one of the most widely used formal methods and has been applied tothe construction of a large variety of software systems [Bee86, Be80, Bod88, BFL96,Cot84, Han94, JS93, Lam91, MS93, OH95, Sad88, TaA88].


48/151


49/151

3.2. VDM AND TOOLS 37

In the light-weight use of VDM, the author follows the tradition of the Danishschool of VDM. In this thesis the main focus concerning VDM is rather on modelingcomplex computer-based systems than on proving their correct implementations.

However, the proof obligations for a correct refinement will be used for testingpurposes in Chapter 7.

It might be due to the long tradition of light-weight applications of VDM inDenmark that the tools that provide the best support of this approach are Danish the IFAD VDMTools.

3.2.2 IFAD VDMTools

Tool support of formal methods is crucial for transferring these techniques intoindustry. Our light-weight approach is based on the features currently offered by

the commercial IFAD VDMTools. Two versions of these tools exist, one for thetraditional VDM-SL, one for the object-oriented VDM++. Both have been used inour industrial experiments.

The following features are provided by the last versions of IFAD VDMTools:

Syntax checking of VDM specifications. To provide a literate programming style[Knuth84] for specifications, the VDM source code may include LATEX typesettext under the UNIX family of operating systems. Under Windows, VDMspecification can be included into MS Word documents.

Static and dynamic type checking features the testing of pre- and postcondi-tions, and data-invariants.

Pretty printing. The VDM source code in ASCII format is converted into themathematical notation for VDM-SL by generating a LATEX document underUNIX.

Execution. Specification interpretation is provided for all executable language con-structs.

Test coverage analysis. Test coverage information can be automatically recordedduring the evaluation of a test-suite. The specifier can at any point check whichparts of the specification are most frequently evaluated and which parts have

not been covered at all.

Source-level debugging. The debug mode is used to interpret the specification.This mode provides an overview of break points, function trace, and definitionsavailable in the specification.

Specification management maintains a project by keeping track of the status ofmodules and remembering this status from session to session. Furthermore,all the functionalities are accessible through a graphical user interface.


50/151

38 CHAPTER 3. A LIGHT-WEIGHT APPROACH TO TESTING

Code generation An explicit form of a specification that may be compiled intoC++ or Java source code.

Dynamic Link Facility With this feature external C++ libraries may be linkedto the VDM-SL tool. This enables the combined interpretation of VDM-SLspecifications and C++ source code [FL96].

CORBA API For VDM++ an API has been developed through which the tool canbe accessed as a CORBA object. This enables the call of the specification frominside a programming language that is supported by the CORBA standard[OMG96].

The specification interpreter together with the test coverage analysis tool ismainly used for our light-weight approach to system-level testing. The possibili-

ties of combining programs and executable specifications establishes the test oracleframework that is introduced in Chapter 2.5.

3.3 Test Design with Abstract Prototypes

3.3.1 Abstract Prototypes

The subject of this thesis are abstraction techniques in testing. In the following,our light-weight approach that is based on an abstract but executable model of thesystems requirements is introduced. Such an abstract prototype that is defined ina formal specification language, like VDM-SL, differs from conventional prototypesas used in rapid prototyping processes in many ways:

abstract model of state: the prototype is abstract in the sense that implemen-tation details are not included in the model of the state. Here, an adequateabstraction level mainly depends on the abstraction level of the requirements.

data invariants: the state model can be further constrained by adding logicalproperties to a data model. These data invariants must hold before and afterthe execution of an operation and can be dynamically checked during inter-

pretation of the prototype.

preconditions: further abstraction of the formal prototype may be gained by re-ducing the definition of a systems functionality to common cases explicitlystated in the requirements. Special cases, like exceptions may be subject for alater, more detailed design. Preconditions are used for defining the propertythat must hold for executing an operation succesfully. Here, successful execu-tion means an execution that terminates. Preconditions may be dynamicallychecked during execution of the prototype.


51/151

3.3. TEST DESIGN WITH ABSTRACT PROTOTYPES 39

advanced language features: A specification language usually offers more ad-vanced language constructs than traditional programming languages. This isdue to their mathematical nature. Examples of such features are

logical quantors that are executable over finite domains;

comprehension expressions, like set comprehensions;

pattern matching, like in functional programming languages.

These language features enable an executable formal specification to serve as anabstract prototype for test-case design and validation. In the following, the approachis explained in more detail.

3.3.2 The Light-Weight Test Approach

Our method proposes the use of a model-based specification language with an exe-cutable language subset supported by a specification interpreter. In our case VDMand IFAD VDMTools have been chosen for building a formal and executable modelrepresenting the requirements of the system under test. Although the model is exe-cutable, it is still abstract in the sense that implementation details are not included.Only abstract mathematical objects like sets, sequences, finite mappings are usedto capture the relevant requirements.

Once an executable formal model has been established, the interpreter tool canbe used to run test-cases on this formal prototype. With this test-case animationalready designed test-cases can be validated, or new test cases can be designed using

structural techniques. IFAD VDMTools support this process with their test coverageanalysis tool that highlights uncovered parts in the VDM specification.

Figure 3.1 presents an overview of the processes of our approach that have beendeveloped for the two experiments in industry. The phases are described as follows:

1. The starting point is the informal requirements documentation containing theuser and software requirements of the system that should be tested.

2. Usually, in an industrial environment a traditional test design process is al-ready established. This traditional test design method takes the informalrequirement documents and produces one test-case for each requirement.

3. In parallel to the conventional test design, a fo

Documents

Systematic Black-Box Testing of Computer-Based Systems through Formal Abstraction Techniques