Software Testing and Reliability

Software Testing

and Reliability

Southern Methodist UniversityCSE 7314

Functional and structural approaches to testing

A few words on combinatorics

• Based on the Cartesian product of sets, we can count the number of possible inputs that a program has, i.e. | I |

Example • Assume a program has a single

input, Customer ID (CID)• May be any value in the domain

{00000-99999}• What is | I | ?

Example• Now assume we add a second

input, the Order ID (OID)• This may be any value in the

domain {00000-99999} as well• Now what is | I |?

Example• Finally, add a credit card number to

the input• This is a 12 digit number• | I | has now reached 10**22

• If we can execute 1 million tests per second, it will take 1016 seconds, or about 300 million years!!

Example• Since we cannot know what data

may exercise a given statement/path in general, we may attempt to resort to exhaustive testing• This attempt is doomed to fail

due to the combinatorial explosion

Engineering the testing process

• Any engineered product (and most other things) can be tested in one of two ways–Knowing the specified function that a

product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function

Engineering the testing process

–Knowing the internal workings of a product, tests can be conducted to ensure that “all gears mesh” (internal operations performed according to specifications)

Structural testing• Uses knowledge of the internal

workings• Also known as Clear box/glass box• Code based• Can be useful for finding interesting

inputs• Misses an entire class of faults,

missing code

Behavioral • Uses knowledge of the specific

function that is to be performed• Based solely on the specification

without regard for the internals• Also known as Black box• More user oriented• Misses an entire class of faults, extra

code (surprises) except by accident

Passing criteria• How do we know when• 1. a single test has passed• 2. when we are done testing

Passing criteria• A single test passes when its

output is correct–This requires a specific definition

of correct and ties into the automated oracle problem

When are we done?• Conway Criteria:• No syntactic errors (it compiles)• No compile errors or immediate

execution failures• There exists Some set of data for which

the program gives the correct output• A typical set of data produces the

correct output

When are we done?• Difficult sets of data produce the

correct output.• All possible data sets in the problem

specification produce the correct output

• All possible data sets and likely erroneous input succeeds.

• All inputs produce the correct output

Nature of software defects

• Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed

Nature of software defects

• We often believe that a logical path is not likely to be executed when, in fact, it may be executed on a regular basis

• Typographical errors are random

More of a case for WHITE box testing……

Overview of Coverage testing

Logical coverage• What does statement coverage

imply ?

Example • 1 IF (( X > 1) AND ( Y == 1))• 2 Z = Z / X• 3 END• 4 IF ((X == 2) OR ( Z >1))• 5 Z = Z + 1• 6 END

What is required for statement coverage ?

Example • To achieve statement coverage, one

can choose x = __, y = __

• Many possible errors are missed:

• 1 IF ((X > 1) OR (Y ==1))• 1 IF ((X >=1) AND (Y ==1))• 1 IF ((X > 1) AND (Y >=1))

Blindness • Statement coverage is blind to

several classes of error when choosing values to achieve coverage

Assignment blindness• Due to assignment of a particular

value to a variable, the error does not propagate

Equality blindness• Due to an Equality check of a

particular variable, the error does not propagate

Self blindness• Conditional itself covers up the

error

Assignment Blindness example

• 1 IF (<conditional>)• 2 X = 1• 3 END• 4 IF (X + Y > 1) //Should have

been (Y > 0)• 5 ....• 6 END


• In this example, values are chosen to make the conditional true, the statement X=1 is executed and the error in line 4 is not seen


• If every path forces the same assignment, then the 'error' doesn't really matter, (does it exist??) –For instance, if the conditional in

statement 1 always evaluated to true

Equality Blindness Example

• 1 IF (A == 2)• 2 .....• 3 IF (B > 1) // Should have been

(A + B > 3)

Equality Blindness Example

• In this example, the value of 2 is chosen for A to force execution of the body

• The error in statement 3 is missed

Self Blindness• 1 IF (X < 1) // Should have been

(2X < 1)

• In this example, the value of 0 is chosen for X

Observation • Statement coverage is the

weakest form of coverage• Many classes of errors can

escape this testing• Typical projects have less than

80% statement coverage!!

Branch coverage• Each branch of a decision must be

executed at least once• This is stronger that statement

coverage, but it is still weak

• Note: A decision is a logical combination of one or more conditions

Branch coverage example • 1 IF (( X > 1) AND ( Y == 1))• 2 Z = Z / X• 3 END• 4 IF ((X == 2) OR ( Z >1))• 5 Z = Z + 1• 6 END

Branch coverage example• To achieve branch coverage, one

option is to choose–x = __, y = __–x = __, y = __

Branch coverage example• Many possible errors are still

missed:

• 1 IF ((X > 1) OR ( Y ==1)• 1 IF ((X >=1) AND (Y ==1))• 1 IF ((X > 1) AND (Y <=1))

Blindness in branch coverage

• Branch coverage is blind to several classes of error when choosing values to achieve coverage–Compound decisions (more than

one condition) are weakly tested–Boundaries of conditions (within a

decision) are not explicitly tested

Condition coverage• Each condition within a decision

must assume all possible values• This is 'potentially' stronger than

branch coverage, but not always• It may, in fact, be weaker

Example of condition coverage

• 1 IF (( X > 1) AND ( Y == 1))• 2 Z = Z / X• 3 END• 4 IF ((X == 2) OR ( Z >1))• 5 Z = Z + 1• 6 END


• To achieve condition coverage for statement 1, one must choose values such that:

• X > 1 ~(Y == 1)• ~(X > 1) Y == 1


• Both of these vectors miss the execution of statement 2

• A better choice may be:

• ~(X > 1) ~(Y == 1)• X > 1 Y == 1

Multi-condition coverage• Every combination of possible

condition values within a decision must be chosen

• This is strictly stronger than branch or condition coverage–still has weaknesses!

Multi-condition coverage example

• 1 IF (( X > 1) AND ( Y == 1))• 2 Z = Z / X• 3 END• 4 IF ((X == 2) OR ( Z >1))• 5 Z = Z + 1• 6 END


• To achieve multi-condition coverage for statement 1, one must choose values such that:

• 1. X > 1 Y == 1• 2. X > 1 ~(Y == 1)• 3. ~(X > 1) Y == 1• 4. ~(X > 1) ~(Y == 1)


• To achieve multi-condition coverage for statement 2, one must choose values such that:

• 5. X == 2 Z > 1• 6. X == 2 ~(Z > 1)• 7. ~(X == 2) Z > 1• 8. ~(X == 2) ~(Z > 1)

Coverage X Y Z Covers

1 0 1 1,5

2 1 4 4,8

1 1 3 2,6

2 0 1 3,7


• There are 9 paths through this code and this test set only covers 4 of those


• Consider this error• Line 2 is incorrectly written as Z

= Z –X• Only the second vector will

execute this line but for the values chosen, Z-X = Z/X


• Also consider that no vector will execute both line 2 and line 5

• In a more complex example, it is easy to imagine variables being set in line 2 that are then incorrectly used in line 5


• For instance:

• 2. Z = Z/X; A = 0• ....• 5. Z = Z + 1; B = 1/A;

Path coverage• Every possible path must be

chosen

• Finding vectors to accomplish this is, in general, undecidable, and usually difficult

Path coverage• This still misses boundary

conditions• If statement 1 was mis-typed as

X >= 1, it is conceivable that the chosen test set does not contain a vector with X=1, missing this error

Basis path testing

Basis path testing• White box technique• Test case designer can derive a

logical complexity measure of a procedural design

• Control flow represented by a control graph

Flow graph notation

sequence

if

Flow graph notationwhile

while

Flow graph notation

case

Flowchart 1

2

3

4

5

6

7 8

910

11

Flow graph1

2,3

6

7 8

9

10

11

4,5

R3

R2

R1

R4region

nodeedge

Compound logicPredicate nodes a

b

y x

x

IF a OR bthen procedure xelse procedure y

ENDIF

Cyclomatic complexity• Software metric that provides a

quantitative measure of the logical complexity of a program

• Value computed defined the number of independent paths in the basis set of a program

• Upper bound for the number of tests that must be conducted to ensure all statements have been executed at least once

Independent path• Any path through the program

that introduces at least one new set of processing statements or a new condition

• Must move along at least one edge that has not been traversed before the path is defined

Example • Path 1; 1-11• Path 2; 1-2-3-4-5-10-1-11• Path 3; 1-2-3-6-8-9-10-1-11• Path 4; 1-2-3-6-7-9-10-1-11• 1-2-3-4-5-10-1-2-3-6-8-9-10-1-11 is

not independent because no new edges traversed

Basis set• Paths 1,2,3, and 4 constitute the

basis set• If tests can be designed to force

execution of these paths, every statement in the program will have been guaranteed to be executed at least one time and every condition will have been executed on its true and false sides

Basis set• Basis set is not unique; a

number of different basis sets can be derived for a given procedural design

How do we know how many paths to look for?

• Compute the cyclomatic complexity

• A graph theoretic computation which yields a good software metric

Three ways of computing• 1. The number of regions of the flow

graph correspond to the cyclomatic complexity

• 2. V(G) = E – N + 2– E = number of flow graph edges– N = number of flow graph nodes

• 3. V(G) = P + 1– P = number of predicate nodes in G

V(G) for Flow graph1

2,3

6

7 8

9

10

11

4,5

R3

R2

R1

R4region

nodeedge

Example • 1. Flow graph has 4 regions• 2. V(G) = 11 edges – 9 nodes + 2 = 4• 3. V(G) = 3 predicate nodes + 1 = 4• V(G) provides upper bound on

number of tests that must be designed and executed to guarantee coverage of all program statements

Deriving test cases• Basis path testing can be applied

to a procedural design or to source code

PDL for test case designPROCEDURE average;* This procedure computes the average of 100 or fewer numbers that lie between bounding values; it also computes the sum and the total number valid.INTERFACE RETURNS average, total.input, total.valid;INTERFACE ACCEPTS value, minimum, maximum;

TYPE value[1:100] IS SCALAR ARRAY;TYPE average, total.input, total.valid; minimum, maximum, sum IS SCALAR;TYPE i IS INTEGER

i = 1;total.input = total.valid = 0;sum =0;DO WHILE value[i] <> -999 AND total.input < 100; increment total.input by 1; IF value[i] >= minimum AND value[i] <= maximum THEN increment total.valid by 1; sum = sum + value[i]; ELSE skip ENDIF increment i by 1;ENDDOIF total.valid > 0 THEN average = sum / total.valid; ELSE average = -999ENDIFEND average

Deriving test cases for “average”

• 1. Using the design or code as a foundation, draw a corresponding flow graph

i = 1;total.input = total.valid = 0;sum =0;DO WHILE value[i] <> -999 AND total.input < 100; increment total.input by 1; IF value[i] >= minimum AND value[i] <= maximum THEN increment total.valid by 1; sum = sum + value[i]; ELSE skip ENDIF increment i by 1;ENDDOIF total.valid > 0 THEN average = sum / total.valid; ELSE average = -999ENDIFEND average

Identify the nodes in this PDL!!!

Draw the Flow graph


• 2. Determine the cyclomatic complexity of the resultant flow graph–V(G) can be determined without

developing a flow graph by counting all conditional statements in the PDL (compound conditions count as two) and then adding 1

Computer V(G) using the three methods

• 2. (continued)–V(G) = –V(G) =–V(G) =


• 3. Determine a basis set of linearly independent paths– Value of V(G) provides the number of

linearly independent paths• Path 1; • Path 2; • Path 3; • Path 4; • Path 5; • Path 6;

List the basis set paths!!


• 4. Prepare test cases that will force execution of each path in the basis set–Data should be chosen to that

conditions at the predicate nodes are appropriately set as each path is tested. Test cases that satisfy the basis set just described are;

Prepare a set of test cases

Summary • Each test case is executed and

compared to expected results• Ensures all statements executed

at least once• Some independent paths cannot

be tested in stand-alone fashion–Tested as part of another path test

Graph matrices

Graph matrices• Testing basis paths can be

mechanized by using graph matrices

• Size based on number of nodes in the graph

• Tabular representation of a flow graph

Graph matrix

1

3

4

2

5

a

b

cd

e

fg

Graph matrix

a

d b

c f

g e

1 2 3 4 5

1

2

3

4

5

node

Connected to node

Link weight• Provides additional information

about control flow• Connection either exists (1) or does

not exist (0)• Can also represent– Probabilities of execution– Processing time of a link–Memory or resources required

Connection matrix

1

1 1

1 1

1 1

1 2 3 4 5

1

2

3

4

5

node

Connected to node

connections

1 – 1 = 0

2 – 1 = 1

2 – 1 = 1

2 – 1 = 1

3 + 1 = 4Cyclomatic complexity

Data flow testing

Data flow testing• This approach selects test paths

according to the locations of definitions and uses of variables in the program

• Definition Use (DU) testing–Does not guarantee coverage of all

branches of a program but does pretty good

Example proc x B1; do while C1 if C2 then

if C4 then B4; else B5;endif;

else if C3

then B2; else B3;endif;

endif; enddo; B6;end proc;

Now apply the DU testing strategy

• We must first know the definitions and uses of variables in each condition or block in the PDL

• Assume variable x is defined in the last statement of blocks B1, B2, B3, B4, and B5 and used in the first statement of blocks B2, B3, B4, B5, and B6


• The DU testing strategy requires an execution of the shortest path from each of Bi, 0<i<5, to each of Bj, 1<j≤6

• This testing also covers any use of variable x in conditions C1, C2, C3, and C4


• There are 25 DU chains of variable x, but we need only five paths to cover these DU chains

• 5 paths are needed to cover the DU chain of x from Bj, 1<j≤5, and other BU chains can be covered by making these five paths contain iterations of the loop

Loop testing

Loop testing• Loops are the cornerstone of many

algorithms in software• White box testing technique• Four different classes of loops– Simple loops– Concatenated loops– Nested loops– Unstructured loops

Simple loops

Testing Simple loops• Skip the loop entirely• Only one pass through the loop• Two passes through the loop• m passes through the loop where m

< n (n is maximum allowable passes through the loop)

• n-1, n, n+1 passes through the loop

Nested loops

Nested loops• Cannot simply extend the

approach from simple loops–Can grow geometrically

• 1. Start at the innermost loop. Set all other loops to minimum values

Nested loops• 2. Conduct simple loop tests for

the innermost loop while holding the outer loops at their minimum iteration parameter (e.g. loop counter) values. Add other tests for out of range or excluded values

Nested loops• 3. work outward, conducting

tests for the next loop, but keeping all other outer loops at minimum values and other nested loops to “typical” values

• 4. continue until all loops have been tested

Concatenated loops

Concatenated loops• These loops can be tested using the

approach defined for simple loops, if each of the loops is independent of the other

• However, if two loops are concatenated and the loop counter for loop 1 us used as the initial value for loop 2, then the loops are not independent. When this happens, use the nesting loops approach

Unstructured loops

Unstructured loop• Redesign !!!• Use structured programming

constructs• http://acweb.colum.edu/users/

rcourington/Progclass/struprog.html

End of lecture

Documents

Software Testing and Reliability