Software Testing part II (white box)santucci/SW_Engineering/Material/05_A_Test_PartII.pdf · White-box testing: general characteristics cont.d • Allows to test parts of the program

Software Testing part II(white box)

Lecturer: Giuseppe Santucci

4. White box testing

• White-box (or Glass-box) testing: general characteristics

• Statement coverage• Decision coverage• Condition coverage• Decision and condition coverage• Coverage of a linearly independent set of paths

2

White-box testing: general characteristics

• Also called– Glass-box testing– Structural testing (compared to functional)

• Exploits the control structure, that means the control or flow graph of programs in order to design test cases

3

White-box testing: general characteristics cont.d

• Allows to test parts of the program– With black-box is not possible

• Allows to cover systematically every part of the program– Often errors are found in special cases

4

Types of white-box tests

Statement coverage

• It is the easiest criterion for a reasonable white-box test

• Necessary but not sufficient• Goal

– To identify a set of test cases sufficient to exercise all statements at least once

6

Statement coverage cont.d

The test case{x != 0; y = any}

Covers the statements

Do you see any problem ?7

Java example:

/*A*/ float x = InOut.readFloat(),y = InOut.readFloat();

/*B*/ if (x != 0)/*C*/ y = y+10;/*D*/ y = y/x;/*E*/ System.out.println(x + ' ' + y);

A

B

C

D

E

x == 0x != 0

Control flow graph for the example

Statement coverage cont.d

• Possible problem– Even in a case of a program without iterations, the

execution of every statement does not guarantee that all possible paths are exercised

• Possible negative consequence:– In previous java example program the error of division

by 0 is not identified (statement D)

8

Decision coverage

• Properly includes statement coverage• Vice versa does not hold• Focuses on all possible decisions in a program • Decisions are in statements

– if, switch, while, for, do• Goal

– To identify a set of test cases sufficient for guaranteeing that each decision will have value “true” at least once and value “false” at least once

9

Decision coverage cont.d

10

Java example:

/*A*/ float x = InOut.readFloat(),y = InOut.readFloat();

/*B*/ if (x != 0)/*C*/ y += 10;/*D*/ y = y/x;/*E*/ System.out.println(x + ' ' + y);

x == 0Test cases1. {x=20; y=30}2. {x=0; y=30}

Cover decisions hence statementsThe second case identifies the error of division by zero in statement D

A

B

C

D

E

x != 0


• Possible problem• If decisions are composed of several conditions

(AND, OR), decision coverage can be not sufficient

11

Java example:/*A*/ float z = InOut.readFloat(),

w = InOut.readFloat();t = InOut.readFloat();

/*B*/ if (z == 0 || w > 0)/*C*/ w = w/z;/*D*/ else w = w + 2/t;/*E*/ System.out.println(z+' '+w+' '+t);

What is the problem here?


• Test cases:– {t = 0; z= 5 ; w= 5} decision TRUE– {t = 0; z= 5 ; w= -5} decision FALSE

• Cover decisions• The risk of division by zero in D is identified

(thanks to t = 0 that is not due to decision coverage strategy !!!! )• The risk of division by zero in C is not identified (z!=0)• We need a criterion that considers the decision’s

conditions (i.e., the atomic Boolean expression components)

(z == 0 || w > 0)

12

Condition coverage

• Does not properly include decision coverage• It is not properly included by decision coverage• Focus on all possible conditions in a program• Conditions are the components of Boolean

expressions • Goal:

– To identify a set of test cases sufficient for guaranteeing that every condition (atomic Boolean expression) included in the program’s decisions have value “true” at least once and value “false” at least once

13

Condition coverage cont.d

14

Test cases{t=0;z=0;w=‐5} 1st condition=T, 2nd condition=F{t=0;z=5;w= 5} 1st condition=F, 2nd condition=TCover conditions

What problem do such case tests make emerge?

Java example:/*A*/ float z = InOut.readFloat(),

w = InOut.readFloat();t = InOut.readFloat();


Condition coverage

• Test cases– {t=0;z=0;w=-5} 1st condition=T, 2nd condition=F– {t=0;z=5;w= 5} 1st condition=F, 2nd condition=T

• Covers conditions• The risk of division by zero in C is identified • The risk of division by zero in D is not identified as

D is never exercised• Decision is always “true”

15

Decision and condition coverage

• Properly includes– Decision coverage

– And, as a consequence, statements coverage

– Condition coverage• Goal

– To identify a set of test cases sufficient for guaranteeing that

• Each decision is “true” at least once and “false” at least once• All conditions composing decisions is “true” at least once and

“false” at least once

16

Decision and condition coverage cont.d

17

Test cases{t=0;z=0;w=5} 1st condition=T, 2nd condition=T, decision=T{t=0;z=5;w=‐5} 1st condition=F, 2nd condition=F, decision=FCover decisions and conditions and identify both division by zero in

statements C and D(thanks to t = 0 that is not due to decision and condition coverage

strategy !!!! )

/*A*/ float z = InOut.readFloat(),w = InOut.readFloat();t = InOut.readFloat();


Summary of test cases so far

18

Test case

t z w cond.1

cond.2

decis. Finds error in C?

Finds error in D?

C1 0 0 5 T T T yes no

C2 0 5 5 F T T no no

C3 0 0 -5 T F T yes no

C4 0 5 -5 F F F no yes

Decision and condition coverage

19

Coverage criterion

Associated test cases

Finds error in C?

Finds error in D?

Decision C2 + C4 no yes

Condition C2 + C3 yes no

Decision and condition

C1 + C4, orC2 + C3 + C4

yes yes

Which one is the best?

Methodological issue

• There are different combination of test cases that guarantee decision and condition coverage but it is not always trivial to find one

• We need a simple and effective method for identifying test cases that cover both decisions and conditions

20

Control flow graph’s paths coverage

• Based on– Criteria of decision and condition coverage– Program’s control flow graph

• Goal – To provide a simple method for guaranteeing decision

and condition coverage

21

Control flow graph

22

while (A) {if (B) { C }else { D }E

}F;

Example:V(G) = 7 ‐ 6 + 2 = 3V(G) = 3

node

edges

region

Predicate nodes

A

B

CD

E

F

Cyclomatic number•V(G) = E ‐ N + 2 (E = number of edges, N = number of nodes)•V(G) = R (R = number of regions)

Control flow graph

23

Sequence ofstatements

Simple if

until

whilecase

What about the for statement? It must be simulated through a while…

Control flow graph without conditions

24

A

B

CD

E

False True

The control flow graph should be more detailed



Control flow graph with complex decisions

25

if (a || b) x;

else y;

b==F

...

a

b

xy

...

a==Ta==F

b==T

if (a && b) x;

else y;

...

a

b

yx

...

a==Fa==T

b==Fb==T

Control flow graph for the program

26

A

B

CD

E

False True

A

z

w

CD

E

== 0!= 0

<= 0 > 0

Control flow graph Without conditions

Control flow graph with conditions



Control flow graph, conditions and decisions

• Boolean values (“true” and “false”) associated with conditions and decisions have corresponding edges

• To design a set of test cases such that all edges of the control flow graph are traversed implies condition and decision coverage

27

Paths of the control flow graph

• The set of test cases corresponds to a set of paths such that every edge is traversed at least once

• The number of paths sufficient for covering all arcs is always equal or less than the cyclomaticcomplexity

28

For the previous example

29

Three test casesC2. {t=0; z=5; w= 5}C3. {t=0; z=0; w= ‐5}C4. {t=0; z=5; w=‐5}Cover all edges, hence decisions and conditions

V=7‐6+2=3The three paths 1. A‐z‐w‐C‐E (C2) 2. A‐z‐C‐E (C3)3. A‐z‐w‐D‐E (C4)



A

z

w

CD

E

== 0!= 0

<= 0 > 0

How to choose paths

• Pragmatic rule– Experiments show that the the number of errors

increases with increasing of the cyclomaticcomplexity

– Choose a number of paths equal to the cyclomaticcomplexity

– You have a correlation between number of test cases and code complexity

– Which paths?

30

Linearly independent paths:Basis Path Testing

• A good criterion for choosing paths is based on the concept of linear independency

• A maximal set of linearly independent paths is called a basis– It is NOT unique

• A basis contains a number of paths equal to the cyclomatic complexity

• Intuitively, every path in the control graph can be obtained as a linear combination of paths of a basis

• By choosing a basis, a certain extent of reliability is guaranteed with respect of combinations of errors that hide each other

31

Example of paths

32

while (A) {if (B) { C }else { D }E

}F;

Paths. A‐F. A‐B‐D‐E‐A‐F. A‐B‐C‐E‐A‐F

A

B

CD

E

F

1

2 3

4 5

67

Arcs/paths 1 2 3 4 5 6 7

0 0 0 0 0 0 1 1 1 0 1 0 1 1 1 0 1 0 1 1 1

The matrix rank is 3 (the maximum) { } is a basis

Note that A‐B‐D‐E‐A‐B‐C‐E‐A‐F covers all edges BUT…

Remarks• Every path corresponds to a binary row vector (0/1)• Vice versa does not hold• A matrix representing paths has maximum 2E

distinct rows (E = n° of edges)• McCabe has proved that for each graph G:

1. The matrix rank cannot be greater than V(G) (cyclomatic complexity)

2. There exists always a matrix made of paths with rank equal to V(G)

• A basis is any set of paths (rows) with maximum rank

• Note: MacCabe’s result is purely topological:– /*B*/ if(5>7)– /*C*/ i=9;– It is not possible to traverse Edge B-C

• In other cases is necessary to re-iterate the cycles in order to traverse all edges

33

A

B

CD

E

F

1

2 3

4 5

67

Edges traversable only by iterations

34

/*A*/ int z = … ; int t=0;/*B*/ while (t<100)/*C*/ if (t > 20) /*D*/ t=t+90;/*E*/ else t= t+z*z;/*F*/ System.out.println(t);

A

B

C

ED

K

t>=100t<100

t<=20t>20

F

B‐FC‐DRequire at least one iteration in order to be traversed{z=10} A‐B‐C‐E‐K‐B‐F{z= 5} A‐B‐C‐E‐K‐B‐C‐D‐K‐B‐F

SoA‐B‐FA‐B‐C‐D‐K‐B‐Fare not traversable

Fake node

Additional considerations

• Paths coverage method does not consider explicitly loops iterations

• In some cases it is necessary to perform iterations in order to traverse all edges

• It might be interesting to consider some iterations in any case but the number of test cases increases exponentially

• It is important to decide the type of iterations

35

Pragmatic choices for loops

• To limit the number of iterations to n– It is called “loop coverage”

• To execute only certain loops• To limit the number of paths to be traversed based

on weighted edges and function to be maximized– Probability of execution– Resources occupancy

• To limit the number of paths by identifying the paths that define and use program variables (Data Flow testing)– Definition of a variable value– Use of such value in a test

36

Data Flow Testing

37

...A int x,y,a,b;B scanf(“%d %d”,&x, &y);C a=x;D b=yE while (a!=b)F if(a>b)G a=a‐b;H else b=b‐a;I printf(“%d”,a);....

D

E

I

F

a!=b

C

B

HG

a==b

a<=ba>b

For x and y it is not needed to execute the loop (loop coverage = 0)For the definition of a and b the loop has to be iterated once (loop coverage = 1)For the definition and usage of a and b the loop has to be iterated twice (loop coverage = 2)

5. Testing strategies

• Testing and development process• Verification vs. Validation• The V-model• Test levels• Test types• Maintenance testing

38

Testing and development process 1

• The adopted life cycle has a big impact on the testing that is carried out: testing does not exist in isolation

• Test activities are highly related to software development activities

• The way testing is organized must fit the development life cycle or it will fail to deliver its benefit

• If a fully documented software development life cycle is required, the testing must be fully documented

39

Verification vs. Validation

• In every development life cycle, a part of testing is focused on verification testing and a part is focused on validation testing

• Verification:– Evaluation of a work product, component or system to determine

whether it meets the requirements set– Answers the question:

• Is the deliverable built according to the specification?• Validation:

– Evaluation of a work product, component or system to determine whether it meets the user needs and requirements

– Answers the question: • Does the deliverable fit for purpose, e.g., does it provide a solution to

the problem?

40

Good practices for testing

Good testing practices apply to any life cycle model• For every development activity there is a

corresponding testing activity• Each test level has test objectives specific to that

level• The analysis and design of tests for a given test level

should begin during the corresponding development activity

• Testers should be involved in reviewing documents as soon as drafts are available in the development cycle

41

V-model

• The V-model was developed to address some of the problems experienced using the traditional waterfall approach

• Defects were being found too late in the life cycle, as testing was not started until the end of the project

• The V-model provides guidance to begin testing as early as possible in the life cycle

42

03Test.43

RequirementsAnalysis

RequirementsReview

SystemDesign

DesignReview

ArchitectureDesign

ArchitectureReview

ModuleDesign

ModuleReview

Coding

Acceptance testPlanning

Functional testPlanning

Integration testPlanning

Acceptance Testing

SystemTesting

IntegrationTesting

Unit testPlanning

UnitTesting

Design Planning Testing

V-model

Internal activities

Activities performed with the userReview

Four test levels1. Unit testing

– Searches for defects in and verifies the functioning of software components • e.g. modules, programs, objects, classes etc., that are separately testable

2. Integration testing – Tests interfaces between components, interactions to different parts of a

system such as an operating system, file system and hardware or interfaces between systems

3. System testing – Concerned with the behavior of the whole system/product as defined by the

scope of a development project or product – The main focus of system testing is verification against specified requirements

4. Acceptance testing – validation testing with respect to user needs, requirements, and business

processes conducted to determine whether or not to accept the system

44

1: Unit testing

• Also known as component, module, and program testing

• Searches for defects in, and verifies the functioning of software– e.g. modules, programs, objects, classes, etc. that are

separately testable• Can be done in isolation from the rest of the system• Stubs and drivers are used to replace the missing

software and simulate the interface between the software components in a simple manner

45

Stub and Driver

• A stub is called from the software component to be tested

• A driver calls a component to be tested

46

A

B

A

Stub

Driver

B

1: Unit testing (cont.)

• Typically occurs with access to the code being tested

• Can be supported by the development environment– such as a unit test framework or debugging tool – See for example JUnit (http://www.junit.org/)

• Usually involves the programmer who wrote the code

• Defects are fixed as soon as they are found, without formally recording the incidents found

47

Junit (annotation based)

48

Junit assertions

49

Junit test case example

@Testpublic void testClassX_methodY(){

ClassX a = new ClassX();assertEquals(4, a.methodY(3, 1));

}

50

Test-driven development

• To prepare and automate test cases before coding

• Highly iterative • Based on cycles of developing test cases, then

building and integrating small pieces of code, and executing the component tests until they pass

• Used in Extreme Programming

51

2: Integration testing

• Focus: – interfaces among components – Interactions with different parts of a system

• an operating system, file system and hardware

– Interfaces between systems• Within the NASA Apollo project, interface errors were the

75% of all errors !

52

Different levels of integration testing

• Component integration testing– Testing the interactions among software components and

is done after unit testing• System integration testing

– Tests the interactions between different systems and may be done after system testing

– In this case, the developing organization may control only one side of the interface, so changes may be destabilizing

– Worst case: a series of systems that run on different platforms

• The greater the scope of integration, the more difficult it becomes to isolate failures to a specific interface

53

Two extreme approaches

• Big-bang integration testing– All components or systems are integrated simultaneously – After that everything is tested as a whole– Advantage: everything is finished before integration testing starts– Disadvantage: time-consuming and difficult to trace the cause of

failures with this late integration• Incremental integration testing

– All programs are integrated one by one– A test is carried out after each step– Advantage: defects are found early in a smaller assembly when

it is relatively easy to detect the cause– Disadvantage: it can be time-consuming since stubs and drivers

have to be developed and used in the test

54

Incremental testing approaches

• Top-down – Testing takes place from top to bottom, following the

control flow or architectural structure• e.g., starting from the GUI or the main menu

– Components or systems are substituted by stubs• Bottom-up

– Testing takes place from the bottom of the control flow upwards

– Components or systems are substituted by drivers• Functional incremental

– Integration and testing takes place on the basis of the functions or functionality, as documented in the functional specification

55

Example of top-down approach

56

A

B C D

Stub E Stub F G

Stub H

Example of bottom-up approach

57

Driver A

Driver B C Driver

D

E F G

H

3: System testing

• Is concerned with the behavior of the whole system/product• It may include tests based on

– Risks and/or requirements specification – Business processes – Use cases, or other high level descriptions of system behavior – Interactions with the operating system and system resources

• Is most often the final test on behalf of development• Should investigate both functional and non-functional

requirements of the system• System testing requires a controlled test environment with

regard to control of the software versions (configuration management)

• The test environment should correspond to the final target or production environment as much as possible

58

4: Acceptance testing

• It comes after the system testing • Responsibility of the user or customer• Answer questions:

– Can the system be released? – What, if any, are the outstanding (business) risks? – Has development met its obligations?

• Executed in a test environment that isrepresentative of the production environment

• Focused on a validation type of testing• It is not necessarily the final level of testing

– E.g. a large-scale system integration test may come after the acceptance of a system

59

Test types: target of testing

• Test types are a means of clearly defining the objective of a certain test level for a program or project

• A test type is focused on a particular test objective• For example

– Testing of a function to be performed by the component or system

– A non- functional quality characteristic, such as reliability or usability

– The structure or architecture of the component or system – Testing changes, confirming that defects have been fixed

(confirmation testing, or re-testing) and looking for unintended changes (regression testing)

• Depending on its objectives, testing will be organized differently

60

ISO 25010 quality characteristics

ISO 25010 (ex 9126) defines eight quality characteristics and the subdivision of each quality characteristic into a number of sub-characteristics1. Portability2. Maintainability3. Security4. Reliability5. Functional suitability6. Performance efficiency7. Compatibility8. Usability

61

ISO 25010

Functional testing

• The function of a system (or component) is what it does• This is typically described in a requirements specification, a

functional specification, or in use cases• It is difficult to test against undocumented and implicit

requirements– some functions that are assumed to be provided

• Functional tests are based on functions described in documents or understood by the testers

• Functional tests may be performed at all test levels– e.g. test for components may be based on a component

specification• Functional testing considers the specified behavior and is

often also referred to as black-box testing– Even if black-box testing includes also non-functional testing

62

Functionality testing based on ISO 25010

• Functions are those that satisfy stated or implied needs

• The objective is to guarantee the existence of a set of functions and their specified properties

• Functional suitability– Completeness– Correctness– Appropriateness

63

Functionality testing based on ISO 25010 (ex 9126)

• functional suitability: • degree to which a product or system provides functions

that meet stated and implied needs when used under specified conditions

• functional completeness: • degree to which the set of functions covers all the

specified tasks and user objectives• functional correctness:

• degree to which a product or system provides the correct results with the needed degree of precision

• functional appropriateness: • degree to which the functions facilitate the

accomplishment of specified tasks and objectives

64

Non-functional testing

• Quality characteristics or non- functional attributes of the system

• How well or how fast something is done• We need to measure on a scale of measurement

– E.g., time to respond • Non-functional testing is performed at all test

levels

65

Non-functional testing based on ISO 25010

• Portability • Maintainability• Security• Reliability• Performance efficiency• Compatibility• Usability

66

Structural testing

• Is often referred to as white-box or glass-boxbecause we are interested in what is happeninginside the box

• It can occur at any test level– It tends to be mostly applied at unit and integration

testing and generally is less likely at higher test levels (e.g., acceptance testing)

• Good tool support to measure code coverage– At component level, and to a lesser extent at (component)

integration testing– Code coverage: the percentage of executable elements,

e.g., statements or decision outcomes that have been exercised

67

Confirmation and regression testing

• Testing of changes• If you have made a change to the software, the

way it functions, the way it performs (or both) and its structure have changed

• We look at the specific types of tests relating to changes, even though they may include all of the other test types

68

Confirmation testing (re-testing)

• A test run on a code that has been changed in order to fix a defect

• The test is the same that was run before and that causes the defect to emerge

• The test is executed in order to confirm that the defect has been fixed

• It is important to ensure that the test is executed in exactly the same way as it was the first time– Using the same inputs, data and environment

69

Confirmation testing: question

• If the test now passes does this mean that the software is now correct?

• We now know that at least one part of the software is correct– Where the defect was

• The fix may have introduced or uncovered a different defect elsewhere in the software

• The way to detect these unexpected side-effectsof fixes is to do regression testing

70

Regression testing

• Involves executing (all) test cases that have been executed before• The intent is checking that the system has not regressed

– It does not now have more defects in it as a result of some changes• It is common for organizations to have what is usually called a regression

test suite or regression test pack– They are designed to collectively exercise most functions in a system

(but not test each one in detail)– It is appropriate to have a regression test suite at every level of testing– All of the test cases in a regression test suite would be executed every

time a new version of software is produced and this makes them ideal candidates for automation

– Maintenance of a regression test suite should be carried out so it evolves over time in line with the software

71

Maintenance testing

• Once deployed, a system is often in service for years or even decades

• During this time the system and its operational environment is often corrected, changed or extended

• Testing that is executed during this life cycle phase is called maintenance testing

• Note that maintenance testing is different from maintainability testing, which defines how easy it is to maintain the system

72

Maintenance testing

• Usually maintenance testing will consist of two parts

1. Testing the changes 2. Regression tests to show that the rest of the

system has not been affected by the maintenance work

73

Documents

Software Testing part II (white box)santucci/SW_Engineering/Material/05_A_Test_PartII.pdf · White-box testing: general characteristics cont.d • Allows to test parts of the program