Object-Oriented Testing - Mälardalen University Sweden · Institution for Computer Science (IDt) Report - Object Oriented Testing ... Models for OOP 9 Side effects 9 ... Polymorphism

Institution for Computer Science (IDt) Report - Object Oriented Testing Box 883 Object Oriented Development, S- 721 23 VÄSTERÅS Advanced Course, CD5130 SWEDEN May, 2000

Object-Oriented Testing

John Lindholm, [email protected] Examinator: Mikael Sandberg

Object Oriented Testing John Lindholm, [email protected]

CONTENTS

Introduction 1

Scope 1

Advantages 1

Disadvantages 1

Structure of the report 1

Testing Software 3

The Testing Process 3

Test planning 5

Testing Object-Oriented systems 6

Types of test in OO 6 Unique bug hazards 7 Object-oriented test automation 7 Test-effective process 7

Test design 7

Using templates 8

Models for OOP 9

Side effects 9

Test models 9

Combinational Models 10 Decision tables 10

State Machines 11

UML 13 Architectural design workflow 13

Patterns 14

Test Design Pattern 14

Test Design Pattern Template 14

Using Test Patterns 14

Tools for running and evaluating test scenarios 16

Test hooks in the scenarios 16

Test Automation 16

Two approaches to evaluating your test 16

Oracles 17

Oracle Patterns 17 Judging 17 Prespecification 17 Gold Standard 18 Organic 18

Measure the testing 19

Testing the code 19


Conclusion 20


Introduction Modern software is often developed in an object-oriented methodology. This implicates that the design, analysis and programming are dealing with classes, frameworks, subsystems and application systems. Testing is often a problem in software engineering, especially in object-oriented programming because of the inheritance, late binding, and encapsulation. Many of the techniques for testing systems apply to all types of systems, including object-oriented ones. However, one should take special care because of the object-oriented characteristics.

There are differences between object-oriented and traditional testing in two ways. First there are aspects of oo that make testing easier and others that make it harder. See Figure 1.

Scope This is a report in the advanced course for object-oriented development, CD5130, taking place at the second spring semester 2000, at Mälardalens högskola, Västerås Sweden. The purpose is to inform about testing in general, oo-testing, and some methods to use.

Advantages The primary role of software testing is to reveal bugs that would be too costly or impossible to find with other verification and validation technique. A secondary purpose is to show that the system under test complies with its stated requirements for a given test suite. If there is effective testing it usually implies that the system will be reliable, safe and successful.

Disadvantages Large costs for finding bugs, takes a lot of time and effort, and the result can't be a perfect system. (Cause there's no perfect system).

Structure of the report This report is made by reading a lot of books and by studying several pages on the Internet.

Modularity

Quicker development

Small methods

More integration

Complex interfaces

Dynamic binding

Polymorphism

Inheritance

Encapsulation

Reuse

Interfaces identified

early

Easier Harder

Figure 1: Eaiser and harder parts of testing object-oriented systems [Graham 1996s]


The result is divided into the following chapters: • Testing Software • Testing Object-oriented Systems • Models for OOP • Patterns • Tools for running and evaluating test scenarios • Measure the testing • Conclusion The first chapter, Testing Software, is about how testing is performed in general. The second and the third chapters deal with how to test OO systems. The following chapters concern methods how to do it and finally there is a conclusion.


Testing Software Software testing is the least understood part of the development process. The following question is often asked: - How did those bugs escape testing? The answer could be any of these: • The user executed untested code. • The order in which statements were executed in actual used differed from that during

testing. This order can determine whether software works or fails. • The user applied a combination of untested input values. The possible input

combinations that thousands of users can make across a given software interface are simply too numerous for testers to apply them all. Testers must make decisions about which inputs to test, and sometimes testers make the wrong decisions.

• The user's operating environment was never tested. We might have known about the environment but had no time to test it. Perhaps we did not (or could not) replicate the user's combination of hardware, peripherals, operating system, and applications in our testing lab. For example, although companies that write networking software are unlikely to create a thousand-node network in their testing lab, users can - and do - create such networks.

"Every software development organization tests its products, yet delivered software always contains residual defects of varying severity. Sometimes it's hard to imagine how a tester missed a particularly glaring fault. In too many organizations, testers are ill equipped for the difficult task of testing ever-more-complex software products. Informal surveys of seminar audiences suggest that few of those who perform testing (either as a profession or as an adjunct to development or other roles) have been adequately trained in testing or have software testing books on their desks". James Whittaker1 sheds some light on why testing today's software products is so challenging, and he identifies several solid approaches that all testers should be able to thoughtfully apply. The effective tester has a rich toolkit of fundamental testing techniques, understands how the product will be used in its operating environment, has a nose for where subtle bugs might lurk in the product, and employs a bag of tricks to flush them out.

The Testing Process To plan and execute tests, software testers must consider the software and the function it computes, the inputs and how they can be combined, and the environment in which the software will eventually operate. This difficult, time-consuming process requires technical sophistication and proper planning. Testers must not only have good development skills - testing often requires a great deal of coding - but also be knowledgeable in formal languages, graph theory and algorithms.

1 James E. Whittaker is an associate professor of computer science at the Florida Institute of Technology, Melbourne, and chair of the software engineering program. He is the founder and codirector of the Center for Software Engineering Research, an industry-sponsored university research lab dedicated to advancing software engineering theory and practice. His research interests are in software engineering, particularly testing and coding.


Software testing is the process whereby we prove that software is incorrect. It is not the process of making perfect software (called Quality assurance), or fixing bugs and bad software, but deals with finding bugs, and establishing that the error has been corrected. A failure occurs when the output of the system is not in accordance with the intended result. The service that the system is meant to provide is not delivered. An error is present when a part of the system is in a wrong state. It could be in the actual system, or in the specification. A fault is the cause of an error. It could be a failure in another system or a human that does the wrong thing. Software testing is when you provoke the system to get a failure, so that the underlying error can be corrected. A large percentage of development costs are currently being allocated to testing. Any system in a finance, telecom, air transportation or military setting must perform on a higher level regarding software reability. Software systems are written after specifications, and any errors in the requirement, specification or design phases that get into the programs are classified as specification bugs. This usually refers to some function that is working, but that doesn't work in the way the end-user wanted it to. Mistyped or wrong code is called programmer errors. Example "=" instead of "==". This type of error is easily detectable by standard methods (debugging or false output). Gold plating is when developers add functionality that is not in the specifications, thereby adding instability and making the system harder to support. The result is complex dependencies and also impossible to track failures or bugs. This happens when developers are working in small subsystems and cannot see themselves as significant contributors to the whole system. Certain tests are very good at finding errors in the code, but cannot find errors of missing functionality. This is when the system doesn't cover all the areas originally intended. The opposite of non-existing code is dead code, code that will never be executed. Often faulty logical assumptions on the programmers part leads to sections of the code that cannot be reached. Example: a if-statement that always valuate to true, but with code to handle both true and false situations. This can indicate a fundamental error in the system design. A system that does not work as the paper manual or on-line documentation says it should is suffering from a documentation bug. Software testing methodology is to test all cases that's possible in a program. This is impossible so you choose a method that test all states and lines of code at least once until you find a bugg. Then you test again…2 Writing testable software is the design and implementation of a special kind of software system: one that exercises another software system with the intent of finding bugs. Tests are designed by analysing the system under test and desiding how it's likely to be "buggy". Manual testing is important but testing is about the development of an automated system to implement an application specific test design.

2 Erik Bohman - Report UAB/I/G-97:007 Uen - Ericsson AB


Test planning Careful test planning helps to design and organize the tests, so that we are testing appropriately and thoroughly. Each step of the testing process must be planned. The test process has a life of its own within the development cycle, and it can proceed in parallel with many of the other development activities. We must plan each of these test steps: 1. Establiching test objectives 2. Designing test cases 3. Writing test cases 4. Testing test cases 5. Execution tests 6. Evaluating test results The test objective tells us what kinds of test cases to generate. The test case design is key to successful testing. If test cases are not representative and do not thoroughly exercise the functions that demonstrate the correctness and validity of the system, then the remainder of the testing process is useless. Therefore, running a test begins with reviewing the test cases to verify that they are correct, feasible, provide the desired degree of coverage, and demonstrate the desired functionality. Once these checks have been made, one can actually execute the tests.


Fig 2.2 Iterative Development [Software Engineering, Pfleeger, 1998]

Testing Object-Oriented systems In the development process there are several phases of testing depending on the evolution model. Using the incremental/iterative development model (see Figure 2.1 and 2.2), as often in OOD, you are forced to deal with testing in several stages.

Types of test in OO Different types of testing can help to eliminate different kinds of defect: • Usability testing tests that the system will be easy to use effectively. This may include

testing the whole system, some part of it, or its documentation. • Module or unit testing tests the individualmodules of the system.In an OO development, as

we shall see, this normally means the classes. • Integration testing tests that the parts of the system at some level work together correctly.

For example,that the objects involved in a collaboration work together as planned, or that components and frameworks really do plug together as they should.

• System testing tests that the system meets its requirements, both functional and nonfunctional.

• Acceptance testing is usually performed by the customer, or some independent group representing the customer.It validates that the system is really fit for purpose.

• Performance testing may happen at any level(module, integration, or system testing) to check that the system's performance is or will be satisfactory.

• Stress testing is a kind of performance testing that puts the system under greater loads thanwill normaly be expected, to check that it degrades gracefully rather than failing catastrophically.

• Regression testing tests that those features of a system, which worked before a modification still work after it. The tests carried out are some of those carries out previously.Unit tests for any affected units, integration tests for any subsystem which includes a modified unit, and some or all of the whole system tests(usability, system, acceptance, performance and stress tests

There are significant differences between testing procedural and object-oriented software. The differences implies the three basic guidelines in testing: 1. Unique bug hazards. 2. Object-oriented test automation

Fig 2.1 Incremental Development


3. Test-effective process

Unique bug hazards Test design must be based on the bug hazards that are unique to the OOP paradigm. These techniques must be augmented with applicable established test design practices.

Object-oriented test automation Application-specific test tools must be object-oriented and must offset obstacles to testability intrinsic to the OOP paradigm.

Test-effective process The testing process must adapt to iterative and incremental development and mosaic modularity. The intrinsic structure of the OO paradigm requires that test design consider method, class, and cluster scope simultaneously. Unit testing is easier in OOA/D but integration testing must be much more extensive. Inheritance introduces the need for more testing. An inherited functon needs additional testing if: • it is redefined. • it has a special behaviour in a derived class. • other functions in that class are supposed to be consistent.

Test design Test design requires solving problems similar to those encountered in the analysis, design, and programming of an application system. Test models are developed to represent responsibilities. The test can either be performed manually or by a test automation system. Test design involves several steps: 1. Identify, model, and analyze the responsibility of the system under test. 2. Design test cases based on this external perspective 3. Add test cases based on code analysis, suspicions, and heuristics. 4. Develop expected results for each test case or choose an approach to evaluate the pass/no

pass status of each test case. Figure 3 describe the test design process from a system engineer view.


Using templates A template for the test execution: 1. Establish that the implementation under test is minimally operational by exercising the

interfaces between its parts. 2. Execute the test suite; the result of each test is evaluated as pass or no pass. 3. Use a coverage tool to instrument the implementation under test. Rerun the test suite and

evaluate the reported coverage. 4. If necessary, develop additional tests to exercise uncovered code. 5. Stop testing when the coverage goal is met and all tests pass.

Application Modeling

Test Design

Test Automation

System

System Under Test

Post-release Bugs

Application Requirements, Deisgn, Code Application-specific

Test Models General Test Models

Test Suites

Pass/No Pass

Test Inputs

Actual Results

Figure 3. A systems engineering view of testing


Models for OOP The object-oriented paradigm presents many test problems. This is a result of the encapsulation of operations and variables into a class, the variety of ways a system of objects can be composed, and the compression of complex runtime behavior into a few simple statements. The hierachy level creates different behaviors at different levels; an ancestor class can behave in a different way than the child class and vice versa. The message passing can be complex and result in some behavior that one did not expect etc. Multipel inheritance is also a complex aspect. Polymorphism and dynamic binding increases the number of execution paths. All in all - there are many parameters to concern.

Side effects There has been several studies using metrics for object-oriented software engineering. The following results has been found: • Classes that send relatively more messages to instance variable and message parameter objects

are more likely to be "buggy". • Classes with greater depth and higher specialization are more likely to be "buggy". • No significant correlation was found between classes that lack cohesion and the relative

frequency of bugs3. Other experiences • The OO learning curve is very steep and causes many first-use errors. • OO analysis and design seem to have higher defect potentials than older design methods. • OO progamming languages seem to have lower defect potentials than procedural

programming languages.

Test models A model captures the relationships in the system. A model has four main elements: subject, point of view/theory, representation and technique. Models role in testing are for validation and verification of the relationships among the artifacts. There are several techniques for this: • Informal or semiformal requirements using UML4diagrams • A representation meta-model5 using OCL6 syntax • A formal representation using OCL expressions • An implementation using Java7 package Figure 4 suggests the relationships between the models that exist.

3 Basili and Briand - studies on small C++ systems 4 Unified Modeling Language 5 A meta-model is the definition of a modeling technique: symbols used in its notation, rules for using these symbols, concepts associated with symbols, and compositions of symbols. An instance-model is a representation constructed using the technique defined by the meta-model. 6 Object Constraint Language 7 Java is a trademark from Sun


When using a test model there are several requirements that have to be fulfilled. First of all the model should support both manual and automated test generation. The test model must also be complete, represent all actions, and be accurate. It must both abstract and preserve the involving details. Finally it must represent all states in the model.

Combinational Models Many applications must select output actions by evaluating combinations of input conditions. Such requirements can appear at method, class, cluster, subsystem, and user case scope. Combinational logic provides an effective test model for these kinds of condition-action relationships. A straightforward model of decision tables is an ideal representation for test models. They support several test design patterns. • Combinational Function Test shows how they apply at method scope. • Extended Use Case Test can produce a test suite at any scope for which a use case can be

defined.

Decision tables The basic steps with decision tables are:

Meta-Model

Component Representation

Required Behavior

Component Implementation

Responsibility-based Testing

Responsibility-based Test Model

Implementation-based Test Model

Implementation-based Testing

Validation

Completeness Checking

Consistency Checking

Observed Behavior

Figure 4. The relationship of models, testing and validation


1. Model the implementaion or capability under test with a decision table 2. Validate the decision table model with the help of boolean algebra. 3. Derive the logic function 4. Choose a test suite generation strategy 5. Generate test cases In OO development a decision table can model the implementation control flow relationships and provide an implementation-independent model of class responsibilities. Decision tables are also necessary to produce testable use cases, which can then be used to develop tests at subsystems and system scope. The UML Activity Diagram decision notation can be refered as a decision table.

State Machines Behavior bugs are likely because of the complex and implicit structure of OOP. That's why state machines8 are an effective, fundamental technique for design and test of the complex behaviors of OOP. Although the number of message sequences and instance variable value combinations are infinite for practical purpose, can a state machine provide a compact and predictable model of behavior. Nearly all OOA/D use state models and offer state modeling guidance. The simplest strategy is to hard-code the table structure using switch and ifelse expressions. This is error-prone and difficult to maintain, especially for large machines. Another way is to store states in datastructures that corresponds to a state table. In the event-to-state format type of table, each event in the model has a row; each state occupies one column. It can be hard finding names to all different states in a state table and is hard to scale when more states join the tables. Statecharts have been adapted by most OOA/D methodologies and the UML.Statecharts use a graphic shorthand to avoid enumerations of all states. Statecharts extend the basic model in several ways to overcome the basic machine's scalability and concurrency problems.

Figure 5: State Machine in UML

8 See course AUBER AutomataTheory at Mälardalens högskola, Västerås - http://www.idt.mdh.se/kurser/ for further information about state machines


Harel9 has developed an interpretation of statecharts for object-oriented implementations, which includes all of the original statechart semantics. Table 1 lists some key differences between UML statecharts and Harel's model. UML Statechart10 Harel Statechart11 Basic semantics Hybrid Mealy Event content Message parameters,<<signal>>

mapping Any analog or digital signal of interest

Composite events Not allowed, unless represented as parameters of a single message

Arbitrary number of external signals

Bulit-in operations After,when,entry, exit, do, history history, entered(s),exited(s),true(condition),tr!(c) (make true), fs!(c)

Interpretation of concurrency Global visibility of variables defined in the "context" of the statechart and public within the scope of the modeled component

Processes that execute independently; does not model a collection of variables( objects) in the same memory space as concurrent processes.

Broadcast event Not defined,but can be modelled by messages on actions

Built-in broadcasting across boundaries of all processes

Implicit mechanism Class(as server) with unspecified client

Nonspecified

Process Not defined, the scope of a state or superstate is a class or a method.

An "XOR" group or superstate

Transition components Limited to message semantics Any combination of states, transitions, guards, and labels

History Shallow and deep semantics Built-in function Scope/ownership of guards Guards are visible to implicit

client unless specified otherwice. As implied of the object-model diagram

Ordering of actions Default is single-thread synchronous message calling sequence. Multi-thread semantics may be modeled.

Default is asynchronous;mesage passing is assumed to be queued.

Transition interval Indefinite nonzero interval Instantaneous atomic

Table 1: UML Statechart versus Harel Statechart [Test Object-Oriented Systems by R. Binder, 2000]

In the state pattern, each state subclass is flyweigth singleton12 object. The state machine is represented with a root abstract base superclass, which defines a method interface for each event in the machine. Each state is represented with a subclass that inherits this interface and implements the necessary actions for the event/state pair. The problem adressed by state is not implementation of a state machine, but the organisation of related behaviours A usable object-oriented state model must answer several questions • How are states defined and what do they represent for class specifications? For class

implementations? For objects? • How are events and responses defined? What do they represent? • How is control represented? 9 David Harel. Statecharts: a visual formalism for complex systems. Science of Computer Progr., 1987 10 OMG 98 11 Harel 97 12 Software Design Patterns by the GoF, 1995


A testable model has sufficient information to allow automatic generation of test cases. This requires a model that is correct, complete, and updated. • It's a complete and accurate reflection of the kind of implementations to be tested. The

model must represent all features that should be tested • It allows for abstraction of detail, which, if modeled explicitly, would make the cost of testing

prohibitive • It preserves detail that is essential for revealing faults and demonstrating conformance. • It presents all events to which the implementation under test (IUT) must respond. A

message, exception, or interrupt can be produced to test each event. • It represents all actions such that we can generate a program to decide whether some action

has been produced. • It defines state so that the checking of resultant can be automated.

UML UML models are common in object-oriented testing. It is possible to develop a test suite for an application modeled with UML diagrams. The only requirement on the UML model is that it has the sufficient information to produce the test cases. This implicates that there are some considerations to be done when developing the UML diagram if it should be available for testing. There are two approaches for the UML diagrams. First, it is possible to interpret generic test requirements for the applciation and then develop the indicted tests. Second, one may apply a test design pattern13. There is an intersection between UML and the test design patterns14 that is described in the book by Binder [1].

Architectural design workflow The UML architectural design workflow can be described as follows: • Select scenarios • Identify main classes and their responsibility • Distribute behavior on classes • Structure in subsystems, layers, define interfaces • Implement architectural prototype • Derive test from use cases • Evaluate architecture • Iterate (Goto1)

13 will be explained later in this report - Chapter - Patterns 14 That will be described in the following Chapter - Patterns


Patterns

Test Design Pattern A pattern is a generalized solution to a specific, recurring design problem. Patterns are also known as software design patterns, which captures certain stereotypical relationships of classes and objects15.

Test Design Pattern Template A pattern template is a list of subjects that comprise a pattern. Several templates for software design patterns have been developed. Those patterns have become suitable for testing. Test design patterns are based upon the general patterns and have a template "look-a-like" those of GoF. They consist of the following items: • Name • Intent • Context • Fault Model • Strategy • Entry Criteria • Exit Criteria • Consequences • Known Uses • Related Patterns

Using Test Patterns A pattern is a general guidance for testing. Patterns should not be interpreted dogmatically. It is important to read through the whole pattern description16 before applying it. There are tables for consideration to reflect on etc.

Test Design Patterns Method Scope Category-Partition

Combinational Function Recursive Function Polymorphic Message

Design a test suite based on input/output analysis Design a test suite for behaviors selected by combinational logic. Design a test suite for a recursive function. Design a test suite for a client of a polymorphic server.

Class Scope Invariant Boundaries Nonmodal Class Modal Class Quasi-modal Class Transitive Operations

Identify test vectors for complex domains. Design a test suite for a class without sequential constraints. Design a test suite for a class with sequential constraints. Design a test suite for a class with content-determined sequential constraints. Design a test suite for methods that are transitive.

Class Scope Integration

Small Pop Alpha-Omega Cycle

Order of code/test at method/class scope. Order of code/test at method/class scope.

Flattened Class Scope

Polymorphic Server Modal Hierarchy

Design a test suite for LSP compliance of polymorphic server hierarchy. Design a test suite for hierarchy of modal classes.

15 Example: Model-Control-View in the book "Software Design Patterns" by the GoF 16 Testing Object-Oriented Systems - Chapter 9 - Patterns, pp. 338 - 797


Reusable Components

Abstract Class Generic Class New Framework Popular Framework

Develop and test an interface implementation. Develop and test an implementation of a parameterized class. Develop and test a demo application of a new framework. Test changes to a widely used framework

Subsystem Class Associations Round-trip Scenario Controlled Exception Mode Machine

Design a test suite for implementation of class associations. Design and test suite for aggregate state-based behavior. Design a test suite to verify exception handling Design a test suite for implementation of stimulus-response scenarios.

Integration Big Bang Bottom-up Top-down Collaborations Backbone Layers Client/Server Distributed Services High-frequency

Try everything at the same time. Integration by dependencies. Integration by control hierarchy. Integration by cluster scenarios. Hybrid integration of subsystems. Integration for layered architecture Integration for client/server architecture Integration for distributed architecture Build and test at frequent regular intervals.

Application Scope

Extended Use Cases Covered in CRUD Allocate by Frequency

Develop testable use cases, design a test suite to cover application input/output relationships. Exercise all basic operations. Allocate system test effort to maximize operational reliability.

Regression Test Retest All Retest Risky Use Cases Retest Profile Retest Changed Code Retest Within Firewalls

Rerun all tests. Rerun tests of risky code. Rerun tests by frequency of use Rerun tests for code that depends on changes. Rerun test for code affected by changes.

Table 2: Test Design Patterns [Testing Object-Oriented Systems p. 317]


Tools for running and evaluating test scenarios It's tedious and unreliable to do much testing by hand; proper testing involves lots of tests, lots of inputs, and lots of comparisons of outputs. Testing should therefore be done by programs, which don't get tired or careless. It's worth taking the time to write a script or trivial program that encapsulates all the tests, so a complete test suite can be run by pushing a single button. The easier a test suite is to run, the more often you'll run it and the less likely you will skip it when time is short.

Test hooks in the scenarios Complete automation requires simulations of each input source and output destination of the entire operational environment. Testers often include data-gathering code in the simulated environment as test hooks or asserts. This code provides information about internal variables, object properties, and so forth. These hooks are removed when the software is released, but during test scenario execution they provide valuable information that helps testers identify failures and isolate faults17. Scenario evaluation is easily stated but difficult to do. There are actually performed by a human oracle: a tester who visually monitors screen output and analyzes output data. Because manually applying test scenarios is labor-intensive and error-prone, testers try to automate the test scenarios as much as possible. This automation can be performed by test automation or by automated oracles.

Test Automation To be effective and repeatable, testing must be automated. The appropriate extent of automated testing depends on your testing goals, budget, software process, kind of application under development, and of the development and target environment. Test automation system is different for the following systems: • A control vehicle system for an onboard microprocessor communicating over a small

LAN and cross-compiled from a desktop workstation. • A multimedia decision support application with daily changes to requirements. • A high-volume transaction-processing system for a relational DBMS running on servers

with many interfaces to legacy systems. Repeatability is the primary cost justification for automated testing. Test cases, test suites, and test harnesses must be maintained if they are to be repeatable. This work is no less difficult or expensive than maintaining application software.

Two approaches to evaluating your test In dealing with the problems of test evaluation, researchers are pursuing two approaches: formalism, and embedded test code.

17 B.W. Kernighan The Practice of Programming, Addison Wesley, 1999


Formalism chiefly involves the hard work of formalizing the way specifications are written and the way that designs and code are derived from them18. Both object-oriented and structured development contains mechanisms for formally expressing specifications to simplify the task of comparing expected and actual behavior. Industry has typically shied away from formal methods; nonetheless, a good specification, even an informal one, is still extremely helpful. Without a specification, testers are likely to find only the most obvious bugs. Furthermore, the absence of a specification wastes significant time when testers report unspecified features as bugs. There are essentially two types of embedded test code. The simplest type is test code that exposes certain internal data objects or states that make it easier for an external oracle to judge correctness. As implemented, such functionality is invisible to users. Testers can access test code results through, for example, a test API or a debugger. A more complex type of embedded code features self-testing programs19. Sometimes this involves coding multiple solutions to the problem and having one solution check the other, or writing inverse routines that undo each operation. If an operation is performed and then undone, the resulting software state should be equivalent to its preoperational state. In this situation, the oracle is not perfect; there could be a bug in both operations where each bug masks the other20.

Oracles A test case pattern is when the Implementation Under Test, IUT, produces an acceptable result. Pass/no pass evaluation is made by comparing the actual result with an expected result established by a trusted source. In testing jargong, an oracle is a trusted source of expected results. The oracle can be a program specification, a table of examples, or simply the programmer's knowledge of how a program should operate.

Oracle Patterns There is 15 oracle patterns discussed in Binders book [1]. They are presented as micropatterns. There are four groups of patterns: Judging Prespecification Gold Standard Organic

Judging The tester evaluates pass/no pass by looking at the output on a screen, a listing, using a debugger, or another suitable human interface.

Prespecification The expected results are determined before the test is run. These results are made available to a comparator.

18 D.K: Peters and D.L: Parnas, "Using Test Oracles Generated from Program Documentation," IEE Trans. Software Eng., Vol. 24, No. 3, Mar 1998, pp.161-173 19 D. Knuth, "Literate Programming", The Computer J., Vol. 27, No. 2. May 1984, pp. 97-111 20 J.A.Whittaker "What is Software Testing? And Why is it so hard?", IEEE Software Eng., Jan./Feb. 2000, pp. 77


In this category there are four kind of oracles: • Solved Example Oracle • Simulation Oracle • Approximation Oracle • Parametric Oracle

Gold Standard Gold Standard is an existing implementation that is trusted. There are four kind of Gold Standard oracles: • Trusted System Oracle • Parallel Test Oracle • Regression Test Oracle • Voting Oracle

Organic The Organic oracle exploits an informational redundancy present in some systems. There are seven categories of the Organic oracle: • Smoke Test • Reversing Oracle • Built-in Check Oracle • Built-in Test Oracle • Executable Specification Oracle • Generated Implementation Oracle • Different But Equivalent Oracle


Measure the testing Several researchers have proposed testers to begin testing object-oriented systems by asking several questions. These questions are helpful to determining when to stop testing. But for determining when a product is ready to release, is more complex. Testers want quantitative measures of the number of bugs left in the software and of the probability that any of these bugs will be discovered in the field. If testers can achieve such a measure, they know to stop testing.

Testing the code Before starting the test the tester can address these questions: • When your code expects a unique value, is there a path that generates a unique result? • When there are many possible values, is there a way to select a unique result? • Are there useful cases that are not handled? The next step is to check the objects and classes themselves; missing objects, unnecessary classes, missing or unnecessary associations, or incorrect placement of associations or attributes21. There are some guidelines to help identifying these conditions during the testing. They note that objects might be missing if: • you find asymmetric associations or generalizations • you find disparate attributes and operations on a class • one class is playing two or more orles • an operation has no good target class • you find two associations wiht the same name and purpose A class might be unnecessary if it has no attributes, operations, or associations. Similarly, an association might be unnecessary if it has redundant information r if no operations use a path. If role names are too broad or narrow for their placement, an association may be in the wrong place etc. There are also other suggestions on how to perform the oo-testing: Address the test to several levels 1. functions 2. classes 3. clusters22 4. and finally the system as a whole

21 Rumbaugh et. all Object-Oriented Modeling and Design, Englewood Cliffs, NJ Prentice Hall 22 interacting groups of collaborating objects


Conclusion The book, Testing Object-Oriented systems, Models Patterns and Tools, has nearly 1200 pages and is a complete reference to the subject. It contains deep information about testing and is very comprehensive. It's difficult to write something about a 1200 p. summary of the huge area of testing. If someone asked me to be a tester I wouldn't accept the offer because lack of experience (code analysis, suspicions, and heuristics). You need a feeling for what to test and it comes with experience, and reading the book! Testing traditional systems apply well to functions, but many approaches do not take into account the object states needed to test classes. This is the reason for statechart testing beeing one of the best methods for OOA/D. This can be performed by UML and be integrated from the beginning of the software development process. By using the book you can develop your own test analyze,test desig, test software and maintain your test software in a proper way. It's a timeconsuming aproach, but it was probably a lot worse when you had to look for information in articles and papers. Testing object-oriented software is a lot of planning and evaluating. Hard work for those who need a sequre working system. If they don't use the book it will cost more and take more time to reach their goal, if they succed at all. All programmers who expect to be payed should have knowledge about testing and testing software. I'm going to continue learning about testing.


REFERENCES [1] Namn, Titel, Förlag, Årtal, ISBN Robert V.Binder.Testing Object-Oriented Systems.Models, Patterns, and Tools Addison Wesley, 2000, 0-201-80938-9 Perdita Stevens, Using UML Software Engineering with Objects and Components Addison Wesley, 2000, 0-201-64860-1 Gamma Helm Johnson Vlissides, Design Patterns Elementsof Reusable Object Oriented Software, Addison Wesley, 1997, 0-201-63361-2 Shari Lawrence Pleeger, Software Engineering Theory and Practice, Prentice Hall, 0-13-081272-2 Brian W Kernighan Rob Pike, The Practice of Programming, Addison Wesley, 0-201-61586-x James A Whittaker, What is Software Testing? And why Is It So Hard? IEEE Software, 2000


Questions 1.What can testers (you) use decision tables and state machines for? What is their purpose? To create… 2.What is the name of the new form of state machines from UML and Harel? 3.What makes OO testing more complicated than procedural testing? One example.

Documents

Object-Oriented Testing - Mälardalen University Sweden · Institution for Computer Science (IDt) Report - Object Oriented Testing ... Models for OOP 9 Side effects 9 ... Polymorphism