Implementation-Based Testing

Implementation-Based Testing

• Review Class 9

• Implementation-based Testing

CEN 5076 Class 10 – 11/07

CEN 5076 Class 10 - 11/07 2

Test Adequacy Criteria

Definitions from Zhu et al. ‘97:

1. Test Data Adequacy as Stopping Rules: A test data adequacy criterion C is a function C: P X S X T → {true, false}. C(p, s, t) = true means that t is adequate for testing program p against specification s according to criterion C, otherwise t is inadequate.

CEN 5076 Class 10 - 11/07 3


2. Test Data Adequacy Criteria as Measurements: A test data adequacy criterion C is a function C: P X S X T → [0,1]. C(p, s, t) = r means that the adequate of testing the program p by the test set t w.r.t. specification s is of degree r according to criterion C. The greater the value of r, the more adequate the testing.

CEN 5076 Class 10 - 11/07 4


3. Test Data Adequacy Criteria as Generators: A test data adequacy criterion C is a function C: P X S → 2T. A test set t C(p, s) means that t satisfies C w.r.t. p and s, and it is said that t is adequate for (p, s) according to C.

CEN 5076 Class 10 - 11/07 5

Types of TestingSpecification-based (functional)

• Specifies the required testing in terms of identified features of the specification or the requirements of the software, so that a test set is adequate if all the identified features have been fully exercised. [Zhu ’97]

Implementation-based (program-based)

• Specifies testing requirements in terms of the program under test and decides if a test set is adequate according to whether the program has been thoroughly exercised.

CEN 5076 Class 10 - 11/07 6

Specification-based Testing cont

• A specification has two facets: a syntactic structure and a semantic structure.

• There are two role the specifications plays during software testing:

1. To provide the necessary information to check whether the output of the program is correct, the oracle problem.

2. Provide information to select test cases and to measure test adequacy

CEN 5076 Class 10 - 11/07 7


Techniques used to represent the specification:

1. Logic-based

2. State machines

3. Algebraic specs.

Model-based

Property-oriented

CEN 5076 Class 10 - 11/07 8


Test generation for Logic-based models:

• All-Explicit variants

• All-Variants, All-True, All-False, All-Primes

• Each-Condition/All-Conditions

• BDD-Determinants

• Variable Negation

• Nonbinary Variable Domain Analysis

CEN 5076 Class 10 - 11/07 9


Test generation for state-based models:

• Piecewise coverage - all states, all events, all actions.

• All transitions

• All n-Transition Sequences

• All Round-trip Paths

• M-length Signature

CEN 5076 Class 10 - 11/07 10

Exhaustive

Strategy

M-lengthSignature

All Round-trip Paths

All Transition k-tuples

All Transitions

All States All Events All Actions

Faults Revealed

All possible invalid statesAll event and action faults

All invalid statesUp to M extra statesAll event and action faults

All invalid/missing statesSome extra statesAll event and action faults

All event and action faultsSome missing/invalid states

Some event and action faults

Fig. State-based test power hierarchy

CEN 5076 Class 10 - 11/07 11


3. Algebraic Specifications

• Algebraic spec consists of a set of operations that the operations of the software must satisfy.

• An algebraic spec has two parts:

- Syntactic - consists of functions name and their signatures (the input and output types).

- Semantic part – consists of a list of axioms describing relation among functions

CEN 5076 Class 10 - 11/07 12

Specification-based Testing cont• Syntax part:

- Functions that return values of type T (type of algebraic spec) are called constructors and transformers.

- Functions that return values other than T are called observers.

• Semantic part:

- Allows for a list of preconditions describing the domains of functions.

- Allows functions to return an error if element is outside of the domain.

CEN 5076 Class 10 - 11/07 13

Specification-based Testing cont• An equation in an algebraic spec. consists of

two terms as two sides of the eqn.

• A term is constructed from three types of symbols: variables, constants, and operators.

• Each term has two interpretations in the context of testing.

1. As a sequence of calls to the operations that implement the operators specified in the spec.

2. As a value that is the result of the sequence of operations.

CEN 5076 Class 10 - 11/07 14

Specification-based Testing cont• For a algebraic spec to be useful it must be

consistent and sufficiently complete.

- Consistent must not contain any contradictory axioms i.e., no contradiction should be derivable.

- Sufficiently complete if for every sequence w in W the result of applying each observer is defined. W is the set of all operation sequences consisting of constructors and transformers.

• See handout for example of an algebraic spec for a priority queue ASTOOT Doong and Frankl ‘94

CEN 5076 Class 10 - 11/07 15


• The regularity hypothesis [Bouge et al. 1986] assumes that for some complexity degree k, if a program satisfies an equation t(x) = t’(x) on all data of complexity not higher than k, then the program satisfies the eqn on all data.

• No way to determine the complexity k s.t. only the test cases of complexity less than k need to be tested.

CEN 5076 Class 10 - 11/07 16


• Bouge et al. 1986 Bernot et al.1991 suggest that the selection of test cases should be based on partitioning the set of ground terms according to their complexity so that the regularity and uniformity hypotheses in the subsets in the partitioned are assumed.

• The complexity of a test case is the depth of nesting of the operators in the ground term. Ground terms are terms without variables.

CEN 5076 Class 10 - 11/07 17


• The selection of test cases should first consider constants specified by the spec, then all the values generated by one application of operations on constants, the values generated by two applications on constants etc..

CEN 5076 Class 10 - 11/07 18

Specification-based Testing contConstructing Test Cases for Classes:

• Test cases are usually identified from the class specification. The spec can be stated in OCL, natural language, and/or state transition diagrams.

• Test case construction from pre- and postconditions

- Create test cases that covers all possible combinations of situations in which a precondition can hold and a postcondition can be achieved. P. 169 – 176 in text.

CEN 5076 Class 10 - 11/07 19


• It is possible to express the adequacy for the spec in terms of the number of pairs of pre- and postconditions that have been covered.

CEN 5076 Class 10 - 11/07 20

Implementation-based Testing• Specifies testing requirements in terms of

the program under test and decides if a test set is adequate according to whether the program has been thoroughly exercised.

• There are two main groups of implementation-based structural test adequacy criteria.

1. Control-flow

2. Data-flow

CEN 5076 Class 10 - 11/07 21

Control Flow Adequacy Criteria

• Flow Graph (FG) [Binder 00]- A segment is represented by a node in the FG (a

sphere). A segment is one or more contiguous statements with no conditionally executed statements.

- A conditional transfer of control is a branch. A branch is represented by an outgoing edge in the FG.

- The entry point of a method is represented by the entry node, which is a node no incoming edges. The exit point of a method is represented by the exit node, which has no outbound edges.

CEN 5076 Class 10 - 11/07 22

Control Flow Adequacy Criteria cont

Example: Source code1.public int Fun(int x){2. k = 0;3. while (x <= 10 && k < 3){4. if (x%2 != 0)5. k = k + 1;6. x = x + 1;7. }8. if (x < 0){9. x = 10;10. k = 0;11. }12. return k;}

K=0

x<=10 && k<3

x%2 != 0

x=x+1

k=k+1

return k

B

C

D

E

F

I

EntryFlow graph

A

X<0

X = 10K = 0

T

F

TT

FF

G

H

CEN 5076 Class 10 - 11/07 23


Types of code coverage based on FG model:

1. Statement coverage – A set P of execution paths satisfies the statement coverage criterion iff for all nodes n in the FG, there is at least one path p in P s.t. n is on the path p.

• Generate test data to execute every stmt in the program at least once.

• Give a value(s) of x in the example that satisfies statement coverage.

CEN 5076 Class 10 - 11/07 24


2. Branch coverage – A set P of execution paths satisfies the branch coverage criterion iff for all edges e in the FG, there is at least one path p in P s.t. p contains edge e.

• Generate test data to exercise the true and false outcomes of every decision.

• Give a value(s) of x in the example that satisfies branch coverage.

CEN 5076 Class 10 - 11/07 25


3. Path coverage – A set P of execution paths satisfies the path coverage criterion iff P contains all execution paths from the begin node to the end node in the FG.

• Generate test data to exercise all paths in the program. Note every path through a loop is considered to be different and distinct.

• Give a value(s) of x in the example that satisfies path coverage.

CEN 5076 Class 10 - 11/07 26


4. Cyclomatic-Number Criterion – A set P of execution paths satisfies the cyclomatic number criterion iff P contains at least one set of v independent paths, where v = e – n + p is the cyclomatic number of the FG.

• n - number of vertices, e - is the number of edges, p - is the number of strongly connected components.

• A graph is strongly connected if, for any two nodes a and b there exists a path from a to b and a path from b to a.

CEN 5076 Class 10 - 11/07 27


• Basis path testing – is a way of easily identifying the upper bound for the number of paths necessary in order to achieve branch coverage.

v = (e + 1) - n + 1 McCabe Cycolmatic Complexity Number

• Note this is for a strongly connected FG, add edge from exit node to entry node, hence e + 1.

• Note some formulae include the extra edge in the “e”.

• Identify the basis set for the FG example.

CEN 5076 Class 10 - 11/07 28


Types of code coverage based on program text:

5. Multiple Condition Coverage – A test set T is said to be adequate according to the multiple-condition-coverage criterion if, for every condition C, which consists of atomic predicates (p1, p2, …,pn), and all the possible combinations (b1, b2, …,bn) of their truth values, there is al least one test case in T s.t. the value of pi equals bi, i= 1, 2, …,n.

• Generate test data to exercise all possible combinations of true and false outcomes of conditions within a decision.

• Create test set for example that satisfies the MCC

CEN 5076 Class 10 - 11/07 29


Points to note:

• Path coverage not always practical due to loops i.e., infinite number of paths.

• Statement and branch coverage are not finitely applicable due to dead code and infeasible branches.

• Can define a finitely applicable version of the criteria.

CEN 5076 Class 10 - 11/07 30


• However we may not be able to decide whether a test set satisfies a given feasible adequacy criterion e.g. whether a statement in a program is feasible is undecidable.

• Research into flow graph criteria has focused on the most important subsets of paths e.g., length-n path coverage, level-i path coverage, cyclomatic-number coverage,

CEN 5076 Class 10 - 11/07 31

Data-Flow Adequacy Criteria

• Data-flow analysis of test adequacy is concerned with the coverage of flow-graph paths that are significant for the data flow in the program.

• Each variable occurrence is classified as either a definition occurrence or a use occurrence.

• A definitional occurrence (def) of a variable is where a value is bound to the variable.

• A use occurrence (use) of a variable is where the value of the variable is referred.

CEN 5076 Class 10 - 11/07 32

Data-Flow Adequacy Criteria cont

• Two types of use occurrences:

Predicate use (p-use) – if the value of a variable is used to decide whether a predicate is true or false.

Computation use (c-use) – variable used in the evaluation of an expression, call-by-value argument or an output statement.

• Data-flow test adequacy analysis is concerned with subpaths from nodes with a def to nodes with a use.

• A def-clear path w.r.t a variable x is a path where for all nodes in the path there is no def occurrence of x.

CEN 5076 Class 10 - 11/07 33

Data-Flow Adequacy Criteria cont• A global c-use of a variable x is where no def of x

precedes the c-use within the node in which it occurs.

• A def of a variable x at node u reaches a c-use at node v iff there is a path from u to v s.t. p =(u, w1, w2, …,wn, v) and (w1, w2, …,wn) is a def-clear w.r.t x and the occurrence of x at v is a global c-use.

• We say that the def of x at u reaches the c-use of x at v through path p.

CEN 5076 Class 10 - 11/07 34

Data-Flow Adequacy Criteria cont• A def of a variable x at node u reaches a p-use at

node v iff there is a path from u to v s.t. p =(u, w1, w2, …,wn, v) and (w1, w2, …,wn) is a def-clear w.r.t x and there is a predicate occurrence of x associated with the edge from wn to v.

• We say that u reaches the predicate use of x on the edge (wn, v) through path p.

• If there is at least one input datum that can actually cause the execution of the path, then a def feasibly reaches a use of the def.

CEN 5076 Class 10 - 11/07 35

Data-Flow Adequacy Criteria cont• All Definitions Criterion – A set P of execution

paths satisfies the all-defs criterion iff for all defs of x s.t. there is a use of x which is feasibly reached from the def, there is at least one path p in P s.t. p includes a subpath through which the def of x reaches some use of x.

• All Uses Criterion – A set P of execution paths satisfies the all-uses criterion iff for all defs of x and all uses of x that the def feasibly reaches, there is at least one path p in P s.t. p includes a subpath through which that def reaches the use.

CEN 5076 Class 10 - 11/07 36

Data-Flow Adequacy Criteria cont• All Definition-Use-Paths Criterion (All DU-

Paths) – A set P of execution paths satisfies the all-du-paths criterion iff for all defs of x and all paths q through which that def reaches a use of x, there is at least one path p in P s.t. q is a subpath of p, and q is cycle-free or contains only simple cycles.

• Note such a path may be infeasible.

CEN 5076 Class 10 - 11/07 37

Data-Flow Adequacy Criteria contPoints to note:

• Data flow testing methods make no distinction between atomic data and structured or aggregate data.

• Modification and references to an element of a structured datum are regarded as modifications and references to the whole datum. Can lead to errors [Hamlet ’93].

• Dynamic data not considered in the data flow testing i.e., a pointer variable may refer to a number of data and vice versa.

CEN 5076 Class 10 - 11/07 38

Data-Flow Adequacy Criteria cont• For classes there is intra-method, inter-method,

intra-class, and inter-class.

• For each of the data-flow coverage criteria try to identify test cases for the example that provides adequate coverage for feasible paths.

Common errors:

• Using an undefined or uninitialized variable

• Assigning a value to a variable more than once without an intermediate reference.

• Sending a modifier message to an object more than once without an intermediate accessor method.

CEN 5076 Class 10 - 11/07 39

Data-Flow Adequacy Criteria cont• Sending a modifier message to an object more

than once without an intermediate accessor method.

• Deallocating or reinitializing a variable before it is used.

• Using incorrect values in the signature of a polymorphic message, resulting in incorrect binding.

• The container problem: Deleting a collection of object, thereby leaving its members orphaned and inaccessible. More likely in C++ than Java.

CEN 5076 Class 10 - 11/07 40

All-paths

All-du-paths

All-uses

All-p-uses/Some-c-uses

All-c-uses/Some-p-uses

All-defs

All-p-uses

All-edges (branch coverage)

All-nodes (statement coverage)

Partial Ordering of Coverage criteria Rapps and Weyuker ‘85

Criteria A subsumes B

A B

CEN 5076 Class 10 - 11/07 41

Data-Flow Adequacy Criteria cont• The subsume relation is actually a comparison of

adequacy criteria according to the severity of the testing method. Does not say anything about the ability to detect faults.

CEN 5076 Class 10 - 11/07 42

Review Exam 2• Definitions: software testing, testability, test

data selection criteria, test data adequacy criteria, specification-based testing, implementation-based testing.

• Know the following axioms: antiextensionality, general multiple change, antidecomposition, anticomposition.

• Examples of tests that are valid but not reliable and vice versa. The formulas will be provided.

CEN 5076 Class 10 - 11/07 43

Review Exam 2 cont• Software metrics

– Be able to describe how software metrics are used in testing

– Know at least (3) pre-OO metrics and (3) OO metrics

• Specification-based testing: – Types of coverage criteria for logic-based

systems, state machines, algebraic systems. – Be able to draw a state machine for a given

problem and generate test cases to provide specified coverage e.g., all transition k-tuples, all transitions, all states, all events, all actions.

CEN 5076 Class 10 - 11/07 44

Review Exam 2 cont

• Implementation-based testing:

– Control flow: statement coverage, branch coverage, path coverage, basis path coverage, multiple condition coverage (MCC).

– Data flow: all defs, all def-use-paths, all uses.

– Be able to draw a flow graph for a method; identify def-use pairs for the variables in a method.

– Understand the concept of infeasible coverage criteria.

CEN 5076 Class 10 - 11/07 45

Review Exam 2 cont

• Implementation-based testing:

– Given a specific control flow or data flow criteria be able to generate a test set to provide adequate coverage. E.g., be able to identify the set of basis paths in a flow graph for a method and generate test cases to cover all the basis paths.

Documents

Implementation-Based Testing