Software Testing Copyright, 1999 © Jerzy R. Nawrocki [email protected] Personal Software Process Lecture

Software TestingSoftware Testing

Copyright, 1999 © Jerzy R. Nawrocki

[email protected]@put.poznan.pl

www.cs.put.poznan.pl/jnawrocki/www.cs.put.poznan.pl/jnawrocki/mse/psp/mse/psp/

Personal Software Process Personal Software Process

Lecture 14Lecture 14

J. Nawrocki, PSP, Lecture 14

{ n+=NF; }{ n+=NF; }

END { print n; }END { print n; }

From the previous lecture..From the previous lecture..

• InspectionInspection

(very formal)(very formal)

• Walk-throughWalk-through

(a presentation)(a presentation)

• Personal reviewPersonal review

(DIY)(DIY)



DesignDesignDesignDesign CodeCodeCodeCodeUnitUnit

testtestUnitUnit

testtestI1 I2 I3

Net savings (hours/KLOC):Net savings (hours/KLOC):

II11: 94: 94

II22 : 51 : 51

II33 : : -20-20

Fagan inspectionsFagan inspections



1. Overview (whole team)1. Overview (whole team)

2. Preparation (individual)2. Preparation (individual)

3. Inspection (whole team)3. Inspection (whole team)

4. Rework4. Rework

5. Follow-up5. Follow-up

Design

er

Design

er Implem

.

Implem

.

Mod

erat

or

Mod

erat

or Tester

Tester

Reviewsession

Fagan inspectionsFagan inspections



Are all constants defined?Are all constants defined?

If a queue is being manipulated , can the If a queue is being manipulated , can the execution be interrupted; If so, is queue execution be interrupted; If so, is queue protected by a locking structure?protected by a locking structure?

Are registers being restored on exits?Are registers being restored on exits?

Are all increment counts properly initialised (0 or Are all increment counts properly initialised (0 or 1)?1)?

Are absolutes shown where there should be Are absolutes shown where there should be symbolics?symbolics?

Are all blocks shown in design necessary?Are all blocks shown in design necessary?

Checklist for design inspectionChecklist for design inspection

Ex

Ex

Wr

Wr

Mis

sing

Mis

sing



PR/M/Min L3: the prologue in the REMARKSPR/M/Min L3: the prologue in the REMARKS

section needs expansion.section needs expansion.

LO/W/Maj L172: NAME-CHECK is performed oneLO/W/Maj L172: NAME-CHECK is performed one

time too few.time too few.

DE/W/Min L175: the design should allow for the DE/W/Min L175: the design should allow for the

occurrence of a period in a lastoccurrence of a period in a last

name.name.

Error listError list



CC: Code commentsCC: Code comments

CU: CB usageCU: CB usage

DE: Design errorDE: Design error

IC: Interconnect callsIC: Interconnect calls

LO: LogicLO: Logic

MN: MaintainabilityMN: Maintainability

OT: OtherOT: Other

PE: PerformancePE: Performance

PR: PrologPR: Prolog

Major MinorMajor Minor

M W E M W E M W E M W E

Date ..............Date ..............Code inspection reportCode inspection report

Mod/Mac: .......................Mod/Mac: .......................

TotalTotal



• Questions posed by the author of the design - to Questions posed by the author of the design - to encourage a thorough reviewencourage a thorough review

• Several brief reviews focusing on a part of a Several brief reviews focusing on a part of a work product (part of a design document)work product (part of a design document)

• A sequence of ‘small’ inspectionsA sequence of ‘small’ inspections• One-person reviews and multi-person inspectionsOne-person reviews and multi-person inspections• Indigenous and seeded defectsIndigenous and seeded defects

Active design reviewsActive design reviews

Phased inspectionsPhased inspections


IntroductionIntroduction

Software testing is a critical element of Software testing is a critical element of SQA and represents the ultimate review SQA and represents the ultimate review of specification, design, and coding.of specification, design, and coding.

-- Roger Pressman’97-- Roger Pressman’97


Testing objectivesTesting objectives

Testing = executing a program to find an errorTesting = executing a program to find an error

Quality of a test case: probability of finding an Quality of a test case: probability of finding an as-yet undiscovered error.as-yet undiscovered error.

A successful test uncovers an as-yet A successful test uncovers an as-yet undiscovered error.undiscovered error.

-- Glen Myers’79 -- Glen Myers’79


Dijkstra’s sayingDijkstra’s saying

Testing cannot show the Testing cannot show the absence of defects, it absence of defects, it can only show that can only show that software errors are software errors are present.present.

-- E.W. Dijkstra-- E.W. Dijkstra


Testing effortTesting effort

Testing: ~ 30% - 40% of total effortTesting: ~ 30% - 40% of total effort

Testing mission-critical systems: 70% - Testing mission-critical systems: 70% - 80% of the total effort (!)80% of the total effort (!)

-- Roger Pressman’97-- Roger Pressman’97


Testing principlesTesting principles

• All tests should be traceable to customer All tests should be traceable to customer requirementsrequirements

• Tests should be planned long before testing Tests should be planned long before testing beginsbegins

• The Pareto principle applies to software testingThe Pareto principle applies to software testing• Testing should begin in-the-small and progress Testing should begin in-the-small and progress

toward in-the-largetoward in-the-large• Exhaustive testing is not possibleExhaustive testing is not possible• Some testing should be conducted by an Some testing should be conducted by an

independent third partyindependent third party


A good test ..A good test ..

.. has a high probability of finding an error.. has a high probability of finding an error

.. is not redundant.. is not redundant

.. should be “best of breed”.. should be “best of breed”

.. should be neither too simple nor too .. should be neither too simple nor too complexcomplex


White-box testingWhite-box testing

• Flow chartFlow chart

• The number of independent paths The number of independent paths (cyclomatic complexity - McCabe’76):(cyclomatic complexity - McCabe’76):

V(G) = P + 1V(G) = P + 1

P: the number of predicate nodesP: the number of predicate nodes

• Construct a basis set of independent pathsConstruct a basis set of independent paths

• Prepare test cases: each path in the basis Prepare test cases: each path in the basis set must be takenset must be taken



void PrintGraph(int n){void PrintGraph(int n){ int r, c;int r, c; for(r=0; r<n; r++){for(r=0; r<n; r++){

printf("\n%d:", r);printf("\n%d:", r); for(c=0; c<n; c++)for(c=0; c<n; c++)

if(Connected(r, c))if(Connected(r, c))printf(" %d", c);printf(" %d", c);

}} printf("\n");printf("\n"); }}

0: 1 2 30: 1 2 3

1: 01: 0

2: 02: 0

3: 03: 0

00

11 22 33



void PrintGraph(int n){void PrintGraph(int n){ int r, c;int r, c; for(r=0; r<n; r++){for(r=0; r<n; r++){ printf("\n%d:", r);printf("\n%d:", r); for(c=0; c<n; c++)for(c=0; c<n; c++) if(Connected(r, c))if(Connected(r, c)) printf(" %d", c);printf(" %d", c); }} printf("\n");printf("\n"); }}

r=0;r=0;if r>=n goto .. if r>=n goto ..

printf(“”, r);printf(“”, r);c=0;c=0;if c>=n goto .. if c>=n goto ..

if not Connected goto .. if not Connected goto ..

printf(“”, c) printf(“”, c)

c++; c++;

r++; r++;

printf(“\n”);printf(“\n”);



void PrintGraph(int n){void PrintGraph(int n){ int r, c;int r, c; for(r=0; r<n; r++){for(r=0; r<n; r++){ printf("\n%d:", r);printf("\n%d:", r); for(c=0; c<n; c++)for(c=0; c<n; c++) if(Connected(r, c))if(Connected(r, c)) printf(" %d", c);printf(" %d", c); }} printf("\n");printf("\n"); }}



if not Connected goto .. if not Connected goto ..


c++; c++;

r++; r++;

printf(“\n”);printf(“\n”);V(G) = 3 + 1 = 4V(G) = 3 + 1 = 4





if !Connected goto.. if !Connected goto..


c++; c++;

r++; r++;


AA

GG

BB

CC

DD

EE

FF

A - GA - G







c++; c++;

r++; r++;


AA

GG

BB

CC

DD

EE

FF

A - GA - G

A - B - F - A - GA - B - F - A - G







c++; c++;

r++; r++;


AA

GG

BB

CC

DD

EE

FF

A - GA - G

A - B - F - A - GA - B - F - A - G

A - B - C - E - B - F - A - GA - B - C - E - B - F - A - G







c++; c++;

r++; r++;


AA

GG

BB

CC

DD

EE

FF

A - GA - G

A - B - F - A - GA - B - F - A - G

A - B - C - E - B - F - A - GA - B - C - E - B - F - A - G

A - B - C - D - E - B - F - A - GA - B - C - D - E - B - F - A - G







c++; c++;

r++; r++;


AA

GG

BB

CC

DD

EE

FF

A - GA - G

n = 0n = 0







c++; c++;

r++; r++;


AA

GG

BB

CC

DD

EE

FF

A - B - F - A - GA - B - F - A - G

0 < n and 0 >= n ???0 < n and 0 >= n ???







c++; c++;

r++; r++;


AA

GG

BB

CC

DD

EE

FF

A - B - F - A - GA - B - F - A - G

00

11

n = 2n = 2

A-B-C-E-B-C-D-E-B-F-A-..-A-GA-B-C-E-B-C-D-E-B-F-A-..-A-G

A test case for:A test case for:


Black-box testingBlack-box testing

Targets:Targets:

• incorrect or missing functionsincorrect or missing functions

• interface errorsinterface errors

• errors in data structureserrors in data structures

• performance errorsperformance errors

• initialisation and termination errorsinitialisation and termination errors



Requirements-based:Requirements-based:

• use scenariosuse scenarios

Based on formal specification:Based on formal specification:

• boundary value analysisboundary value analysis



00 11n = 2n = 2

n = 0n = 0

n = 1n = 1 00

00 11n = 2n = 2

n = 3n = 3 00 11 22

n = 3n = 3 00 11 22

n = 3n = 3 00 11 22

n = 3n = 3 0011

22

void PrintGraph( int n )void PrintGraph( int n )







c++; c++;

r++; r++;


AA

GG

BB

CC

DD

EE

FF

00 11n = 2n = 2

n = 0n = 0

n = 1n = 1 00

00 11n = 2n = 2

n = 3n = 3 00 11 22

n = 3n = 3 00 11 22

n = 3n = 3 00 11 22

n = 3n = 3 0011

22

McCabe’s completenessMcCabe’s completeness


Test report templateTest report template

Tester: ........................ Date: ...........Tester: ........................ Date: ...........

Program: ............... Test symbol: .....................Program: ............... Test symbol: .....................

Test objective: ...........................................................Test objective: ...........................................................

......................................................................................................................................................................

Test conditions: .........................................................Test conditions: .........................................................

........................................................................................................................................................................

InputInput Expected outputExpected output Actual outputActual output


Test report templateTest report template

Tester: Tester: J. NawrockiJ. Nawrocki Date: Date: 18.01.0018.01.00

Program: Program: GraphColGraphCol Test symbol: Test symbol: N2FullN2Full

Test objective: Test objective: To check the program for a graphTo check the program for a graph

consisting of 2 connected nodes.consisting of 2 connected nodes.

Test conditions: Test conditions: Use the program N2Full.Use the program N2Full.

InputInput Expected outputExpected output

0: 10: 1

1: 01: 0

Actual outputActual output

0: 10: 1

1: 01: 0


Comparison testingComparison testing

• Multiple implementationsMultiple implementations

• Sometimes interface filters can be Sometimes interface filters can be helpful (two modules are implementing helpful (two modules are implementing not the same but similar systems)not the same but similar systems)


Software testing strategiesSoftware testing strategies

• Verification vs. ValidationVerification vs. Validation

• The V modelThe V model

• Predicting the failure intensity Predicting the failure intensity (logarithmic Poisson execution time (logarithmic Poisson execution time model):model):

f(t) = (1/p)ln(lf(t) = (1/p)ln(l00pt + 1)pt + 1)

f(t) - cumulative number of failures that f(t) - cumulative number of failures that are expected to occur after time tare expected to occur after time t


SummarySummary

Testing is a destructive activity.Testing is a destructive activity.

Testing cannot show the Testing cannot show the absence of defects.absence of defects.

Testing > 30% of total effort.Testing > 30% of total effort.

All tests should be traceable to All tests should be traceable to customer requirements.customer requirements.

White-box testing.White-box testing.

Black-box testing.Black-box testing.


Further readingsFurther readings

W.S. Humphrey, A Discipline for SoftwareW.S. Humphrey, A Discipline for Software

Engineering, Addison-Wesley, Reading,Engineering, Addison-Wesley, Reading,

1995.1995.

R.S. Pressman, Software Engineering.R.S. Pressman, Software Engineering.

A Practitioner’s Approach, McGraw-Hill,A Practitioner’s Approach, McGraw-Hill,

New York, 1997.New York, 1997.

R.A. DeMillo et al., Software Testing andR.A. DeMillo et al., Software Testing and

Evaluation, Benjamin/Cummings, Evaluation, Benjamin/Cummings,

Menlo Park, 1987.Menlo Park, 1987.


Quality assessmentQuality assessment

1. What is your general 1. What is your general impression ? (1 - 6)impression ? (1 - 6)

2. Was it too slow or too fast ?2. Was it too slow or too fast ?

3. Did you learn something 3. Did you learn something important to you ?important to you ?

4. What to improve and how ?4. What to improve and how ?

Documents

Software Testing Copyright, 1999 © Jerzy R. Nawrocki [email protected] Personal Software Process Lecture