36
Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

Embed Size (px)

Citation preview

Page 1: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

Test Coverage

CS-300 Fall 2005

Supreeth Venkataraman

Page 2: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 2

Test Coverage

Coverage is a measure of how much testing can be done given a set of test cases.

Given a set of test cases, how much of the code in the program being tested is covered?

Or.. Given a set of test cases how many functionalities as listed in the requirements can be tested?

Coverage in most cases is a term associated with white-box testing or structural testing.

Page 3: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 3

Black Box Testing

Black-box testing involves deriving test cases based on the functional requirements of the program being tested.

“Does the software exhibit the intended behavior for each of its functionalities?”

Black-box testing can be used to detect improper functionality, errors of interface, errors in data structures, and initialization and termination errors with little regard for the logical structure of the program. [Pressman 97]

Page 4: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 4

Black Box Testing

We have already seen many types of blackbox testing

Fault-Based testing, boundary-value testing, etc.

Functional testing is thought of as being superior to structural testing in most cases

However, it is acknowledged that code based testing does uncover a lot of defects.

Page 5: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 5

Equivalence Partitioning

A good test case is one that has a good likelihood of uncovering a fault.

Exhaustive testing is impossible due to the infinite domain of inputs.

We must choose a suitable subset of inputs so that “it covers a large part of other possible test cases” – Myers, p.45

Page 6: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 6

Equivalence Partitioning

Partition the input domain into equivalence classes

The goal here is to partition the inputs in such a way that if one test case in an equivalence class is capable of detecting some fault, then all test cases in that class are capable of detecting the same fault.

Page 7: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 7

Equivalence Partitioning

An example of an equivalence class for E-LIB. A set of strings that obey the rules of structure

of the SSN. While checking for the structure of the SSN, if

one test case from our class does not uncover a fault, it is reasonable to assume that none of the strings in the class will reveal a fault.

Page 8: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 8

White Box Testing

White-box testing is also known as structural testing or code-based testing.

The test cases to be used to test the program are derived from the program structure, and not from functional specifications.

Many many variations of white-box testing exist.

Page 9: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 9

White Box Testing – and criticisms

A lot of research has been undertaken to discover the best kind of code coverage.

“All testing that does not derive from functions is fundamentally misguided”

“People do not use software in order to execute its statements; rather they invoke its functionality”

Page 10: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 10

White box or Black box? Achieving code coverage does not imply that

good functional coverage is achieved as well. A good mix of functional and structural testing

is desirable as both functional as well as code coverage are important.

Structural test cases must wait to be derived until the code is written, but functional tests can be derived from requirements.

Weyuker proposed that structural testing be used as a way of evaluating functional test quality.

Page 11: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 11

White box or Black Box?

In many cases, structural coverage is done after adequate functional coverage is achieved.

Structural testing is used to back up functional testing

If structural testing is inadequate, create additional functional test cases.

Page 12: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 12

Test Adequacy

The term “adequacy” can be thought of as asking oneself the question “How many test cases do I need in order to completely cover the entire program based on the technique that I have chosen for testing?

Or….When do I stop testing?

Page 13: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 13

Control Flow Testing

Control-flow testing is based on the flow of control in a program.

There are many variations of control-flow testing. Statement coverage, branch coverage, path coverage etc.

We will examine statement coverage and branch coverage in some detail.

Page 14: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 14

The Program Flowgraph

A program flowgraph graphically represents the potential control flow of a program.

Program flowgraphs are made up of nodes and directed edges

Each node represents a basic block, which can be defined as a collection of statements that are always executed together.

Each directed edge between two nodes A and B in a control flow graph represents a potential control path from A to B

Page 15: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 15

Example Programvoid myflow(int x,int y) { int a; if (y == 0) { a = 1; } else { a = y; } y = y – 1; if (a > 0) { a = x; } else { a = x + 1; } printf(“%d\n”, a);}

Page 16: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 16

2

3

4 6

7

8

911

12

y == 0 y != 0

a > 0 a ≤ 0

1. void myflow(int x, int y) {2. int a;3. if (y == 0) {4. a = 1;5. } else {6. a = y; }7. y = y – 1;8. if (a > 0) {9. a = x;10. } else {11. a = x + 1; }12. printf(“%d\n”, a); }

Page 17: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 17

Statement Coverage

Achieving statement coverage means that all statements in the program have to be executed at least once by a set of test cases

Having complete statement coverage does not necessarily mean that the program cannot fail

Think about the input space.... Statement coverage is also called all-nodes

testing.

Page 18: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 18

Statement Coverage Test Cases

Test case1: input = {x = 7, y = 0}

expected output = {a = 7}

Test case2: input = {x = 7, y = -1}

expected output = {a = 8}

Page 19: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 19

Branch Coverage

Achieving branch coverage means that there must be enough test cases so that every path for a conditional transfer of control is exercised

Or... Every true and false branch have to be exercised by some test case

Complete coverage means that the test cases should be sufficient to traverse all the edges in the program graph.

This form of testing is also known as all-edges testing .

Page 20: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 20

Branch Coverage

The fact that there is complete branch coverage does not mean that all errors will be found. Branch testing does not necessarily check all combinations of control transfers.

Page 21: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 21

Branch Coverage Test Cases

Test case1: input = {x = 7, y = 0}

expected output = {a = 7}

Test case2: input = {x = 7, y = -1}

expected output = {a = 8}

Page 22: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 22

Path Coverage

Path coverage means that all possible execution paths in the program must be executed.

This is a very strong coverage criterion, but impractical.

Programs with loops have an infinite number of execution paths, and thus would need an infinite number of test cases

Page 23: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 23

Data Flow Testing

Data flow testing is based on the definition and use of variables in a program.

Deals with flow of data rather than flow of control.

There are many data-flow testing methods All-definitions coverage, all-uses coverage, all-

du-paths coverage etc. We will look at all-uses testing in some detail

Page 24: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 24

Terminology (Beizer)

Definition – a variable is said to be defined when it is initialized with a value as part of a declaration, or when it appears on the left hand side of an assignment statement .

eg: x = 4;

int y = 0;

Page 25: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 25

Terminology

Use – The value of a variable is said to be used either when the variable appears on the right hand side of an assignment statement or when it appears in a predicate statement like if (x > y) (Beizer)

Eg: y = x;

Page 26: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 26

Terminology

Definition-clear path - This is a set of connected nodes in the flowgraph where some variable x is defined in the first node, and is not subsequently redefined in any other node in the path.

For example, the path 2-3-4-7-8-9-12 is a definition-clear path with respect to variable “x” in figure 2.2.

Page 27: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 27

2

3

4 6

7

8

911

12

y == 0 y != 0

a > 0 a ≤ 0

1. void myflow(int x, int y) {2. int a;3. if (y == 0) {4. a = 1;5. } else {6. a = y; }7. y = y – 1;8. if (a > 0) {9. a = x;10. } else {11. a = x + 1; }12. printf(“%d\n”, a); }

Page 28: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 28

All-Uses Testing

The all-uses treatment states that there must be at least one definition-clear path exercised by some test case from every definition of a variable x to all of its uses.

All-uses testing is a powerful test coverage criterion.

Page 29: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 29

All-Uses test cases Test case1: input = {x = 7, y = 0} expected output = {a = 7} Path = {2-3-4-7-8-9} Test case2: input = {x = 7, y = -1} expected output = {a = 8} Path = {2-3-6-7-8-9} Test case3: input = {x = 7, y = 4} expected output = {a = 7} Path = {2-3-6-7-8-11}

Page 30: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 30

The Subsumes Relationship

The subsumption relationship means that satisfying one test coverage criterion may implicitly force another test coverage criterion to be satisfied as well

Eg: Branch coverage forces statement coverage to be attained as well (This is strict subsumption)

“Subsumes” does not necessarily mean “better”

Page 31: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 31

Subsumption Hierarchy (Rapps and Weyuker)

Page 32: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 32

Mutation Testing

Mutation testing involves changing (mutating) the program source statement(s)

If the actual output from a test case is different from the expected output (hopefully it will be), the test case is said to kill the mutant.

Mutation coverage requires that all mutants be killed

Page 33: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 33

Some Mutation Examples

Operator mutation – change the operator in the mutated version.• Eg: change x + y to x – y or x * y …

• Change x < y to x <=y or x >=y …

If all the mutants are killed we are satisfied that no wrong operator has been mistakenly used.

Page 34: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 34

Choosing a Testing Method

The process• If the development process is not well defined

without a proper test plan one does not really know what type of testing to apply.

• If the project starts getting delayed, then testing time is extremely limited.

Resources available• testing time, testing experience, budget ...

Page 35: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 35

Choosing a Testing Method

Feasibility Issues• Structural testing may not achieve complete

code coverage because of dead code

• Dead code is code that never gets executed.

• There could be dead statements, dead paths etc.

• The tester must devote time to identifying infeasible code.

Page 36: Test Coverage CS-300 Fall 2005 Supreeth Venkataraman

CS-300 Fall 2005 Supreeth Venkataraman 36

Conclusions

In practice, there is no agreement as to what the payoffs are for each form of structural testing

Functional testing has advantages over structural testing because functional tests capture the expected actions of users.