Introduction To Software Testing - Aalborg Universitetpeople.cs.aau.dk/~bnielsen/MBTV07/material/intro.pdfIntroduction To Software Testing Brian Nielsen [email protected] Center of

Introduction ToSoftware Testing

Brian Nielsen [email protected] of Embedded Software Systems

Aalborg University, Denmark

C SS10101110110101011011010101110111

CISS

Software development cycle ☺1. Programmer produces code he believes is bug-free.2. Product is tested. 20 bugs are found. 3. Programmer fixes 10 of the bugs and explains to the testing department that the other

10 aren't really bugs.4. Testing department finds that five of the fixes didn't work and discovers 15 new bugs.5. Repeat three times steps 3 and 4. 6. Due to marketing pressure and an extremely premature product announcement based

on overly-optimistic programming schedule, the product is released.7. Users find 137 new bugs.8. Original programmer, having cashed his royalty check, is nowhere to be found.9. Newly-assembled programming team fixes almost all of the 137 bugs, but introduce

456 new ones.10. Original programmer sends underpaid testing department a postcard from Fiji. Entire

testing department quits.11. Company is bought in a hostile takeover by competitor using profits from their latest

release, which had 783 bugs. 12. New CEO is brought in by board of directors. He hires a programmer to redo program

from scratch.13. Programmer produces code he believes is bug-free.

CISS

Autonomous Systems

CISS

Extreme environmental conditions

CISS

Safety Critical Systems

Rotterdam Storm Surge Barrier

CISS

Sensor-networks

CISS

Consumer Electronics

CISS

System Characteristics• Mass production• Limited Hardware (speed, power,memory)• Safety critical• Autonomous• Distributed• One-shot development• Control of physical environment• Mixed signal I/O• Extreme environments• State-based behavior• Real-time constraints (hard and soft)• Technical / scientific algorithms• …

CISS

Spectacular Software Bugs

• ARIANE-5 • http://www.cs.auc.dk/~ask/Undervisning/MVP/ariane5rep.html

• INTEL Pentium II floating-point division 470 Mill US $

• Baggage handling system, Denver 1.1 Mill US $/day for 9 months

• Mars Pathfinder • http://catless.ncl.ac.uk/Risks/19.49.html#subj1

• Radiation theraphy, Therac-25• http://www.cs.auc.dk/~ask/Undervisning/MVP/therac25.ps

• …….

CISS

Ariane 5 crash• An overrun exception stopped both replicated

position calculation computers (SRI’s) at time H0+36.7. Fail-stop was a wrong design decision

• Overrun caused by higher velocity in Ariane 5 compared to Ariane 4

• Overrun occured in a module which was irrelevant after take-off

• Simulations were not run with the new velocity (would have caught the failure)

CISS

Software Qualities

MaintainerGood Documentation

Readable CodeGood Design

Customer

Low CostPortability Increased

productivity

ReliabilityCorrectnessEfficiency

User

FunctionalityEase of useEase of learning

CISS

Costs of Poor Quality• Increased time to find and fix problems• Increased cost to distribute

modifications• Increased customer support• Product liability• Failure in the market place

CISS

Software Quality Assurance• Quality Control

• Process definition and standards • Formal technical review• Test planning and review

• Quality Assurance• Analysis and reporting• Measurement

CISS

Verification vs. Validation:

• Verification: • “are we building the product right?”• The software should conform to its specification

• Validation:• “are we building the right product?”• The software should do what the user really

requires

CISS

Verification vs. Validation:

• Verification: • “are we building the product right?”• The software should conform to its specification

• Validation:• “are we building the right product?”• The software should do what the user really

requires

Assuring that a software system meets a user’s needs

CISS

What is testing?

CISS

A Self-Assessment Test [Myers]

CISS


• “A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.”

CISS


• “A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.”

• Write a set of test cases to test this program

CISS

Test cases for:


CISS

Test cases for:


1. valid scalene triangle ?

CISS

Test cases for:


1. valid scalene triangle ?2. valid equilateral triangle ?

CISS

Test cases for:


1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?

CISS

Test cases for:


1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?

CISS

Test cases for:


1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?

CISS

Test cases for:


1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?

CISS

Test cases for:


1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?7. one side is sum of others ?

CISS

Test cases for:


1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?7. one side is sum of others ?8. 3 permutations of previous ?

CISS

Test cases for:



9. one side larger than sum ofothers ?

CISS

Test cases for:




10. 3 permutations of previous ?

CISS

Test cases for:




10. 3 permutations of previous ?11. all sides = 0 ?

CISS

Test cases for:




10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?

CISS

Test cases for:




10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?

CISS

Test cases for:




10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?14. for each test case: is

expected output specified ?

CISS

Test cases for:




10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?14. for each test case: is

expected output specified ?15. check behaviour after

output was produced ?

CISS

TestingTesting:• to check the quality (functionality, reliability,

performance, …) of an (software) object

-by performing experiments-in a controlled way

CISS




CISS




• In avg. 10-20 errors per 1000 LOC• 30-50 % of development time and cost in embedded software

CISS




• In avg. 10-20 errors per 1000 LOC• 30-50 % of development time and cost in embedded software

What is a Test?

Software under Test

What is a Test?

Software under Test

Test Data

What is a Test?

Software under Test

Test Data Output

What is a Test?

Software under Test

Test Data Output

Correct result?

What is a Test?

Software under Test

Test Data Output

Correct result?

Oracle

What is a Test?

Software under Test

Test Data Output

Correct result?

Oracle

Test Cases

CISS

Types of Testing

unit

integration

system

performancerobustness

functional behaviour

white box black box

Level

Accessibility

Aspect

usability

reliability

CISS

Types of Testing

unit

integration

system

performancerobustness

functional behaviour

white box black box

Level

Accessibility

Aspect

usability

reliability

CISS

Quality-Characteristics (ISO-9126)• Functionality

• Suitability, accuracy, security, compliance, interoperability

• Reliability• maturity, fault tolerance, recoverability

• Usability• understandability, learnability, operability

• Efficiency• time behaviour, resource utilization

• Maintainability• Analysability, changeability, stability, testability

• Portability• Adaptability, intallability, conformance, replaceability

CISS

Quality-Characteristics (ISO-9126)• Functionality

• Suitability, accuracy, security, compliance, interoperability

• Reliability• maturity, fault tolerance, recoverability

• Usability• understandability, learnability, operability

• Efficiency• time behaviour, resource utilization

• Maintainability• Analysability, changeability, stability, testability

• Portability• Adaptability, intallability, conformance, replaceability

⇒ functional testing

⇒ reliability testing

⇒ usability testing

⇒ performance testing

⇒maintainability testing ??

⇒ portability testing ?

CISS

Module testtest driver

Module under test

test stub test stub

CISS

Whitebox Exampleint invoice (int x, int y) {int d1, d2, s;if (x<=30) d2=100;else d2=90;s=5*x + 10 *y;if (s<=200) d1=100;else if (s<=1000) d1 = 95;

else d1 = 80;return (s*d1*d2/10000);

}

Test Cases

CISS


else d1 = 80;return (s*d1*d2/10000);

}

Test Cases

ifd2=100 d2=90

ifd1=100

if

d1=95

d1=80

return

x>30

s>200

s>1000

CISS


else d1 = 80;return (s*d1*d2/10000);

}

Test Cases

ifd2=100 d2=90

ifd1=100

if

d1=95

d1=80

return

x>30

s>200

s>1000

Test Data Expected Output

X=5 Y=5 75

X=31 Y=10 229.5

X=30 Y=100 977.5

CISS

Black box testing

SUT

requirements

input

events

output

y

x

domain testing

CISS

Module IntegrationM1

M6M5

M10

M2 M3 M4

M11 M12

M9M7 M8

”calls””uses”

”interacts”

CISS

Embedded system

Generic Embedded System

Spe

cific

inte

rface

User interface

Interfaces to other systems

Pow

er s

uppl

y

A/D

+ D

/AD

igita

l I/O

Processing Unit

OSI/O-d

river

s Application Software

Pla

ntSystem environment

Act

uato

rsS

enso

rs

CISS

System test

• 2*CRTG (4 channels) 2 * 200 k€

CISS

Test Equipment

• Complete Type Approval Test System (3 M€)

CISS

Who Should Test?

CISS

Who Should Test?

• Developer• Understands the system• But, will test gently• And, is driven by

deadlines

CISS

Who Should Test?


deadlines

CISS

Who Should Test?


deadlines

• Independent tester• Must learn system• But, will attempt to break it• And, is driven by “quality”

CISS

Who Should Test?


deadlines

• Independent tester• Must learn system• But, will attempt to break it• And, is driven by “quality”

CISS

Testing Objectives• To identify as many errors as possible• To bring the software to an acceptable level of

quality• To determine risk of release• Perform testing efficiently, effectively and within

budget and time constraints• Compile a record of software errors for future

error prevention and correction (learning organization).

CISS

Risk Based Test Strategy• Make best possible use of resources by

identifying and prioritizing quality aspects and subsystems• Higher risk ⇒ more testing• No risk ⇒ no testing

• Risk = chance of failure × damage

Use frequencyChance of error being present

ComplexityNew tools/techniquesInexperienced developers

Cost of repairLoss of market shareLegal claim

CISS

Testability = how easy can a program be tested

• operability - it operates cleanly• observability

• the results are easy to see• distinct output is generated for each input• incorrect output is easily identified

• controllability• processing can be controlled• tests can be automated & reproduced

• decomposability - software modules can be tested independently

• simplicity - no complex architecture and logic• stability - few changes are requested during testing• understandability - program is easy to understand

CISS

Testing• Quality assurance activities by running code –

examines functionality in the form used by customers

• Software testing is a formal process carried out by a specialized testing team in which a software unit, several integrated software units or an entire software package are examined by running programs on a computer. All the associated tests are performed according to approved test procedures on approved test cases.

CISS

Testing• Dynamic testing is the process of executing a

program or system with the intent of finding error (Glenford Meyers’ definition)

• Static testing is any activity that aims at finding defects by inspecting, reviewing, walking through, and analyzing any static component of the software (code, documents, and models)

• Debugging is an ad hoc activity performed by individual developers to find and remove bugs from a program.

• Testing is a planned activity

CISS

There is More than Testing

static analysisstatic analysis

reviewingreviewinginspectioninspection

walk-throughwalk-throughdebuggingdebugging

certificationcertification

CMMCMM

quality controlquality controlrequirement managementrequirement managementsoftware processsoftware process

CISS

Lifecycles & testing

CISSimplementation

code

detailed design

requirements

specification

acceptancetest

systemtest

moduletest

unit-test

V - Model

integrationtest

architecturespec

acceptance test spec

system test spec

integration test spec

module test spec

unit test spec

CISS

Introducing, Detecting andRepairing Errors Liggesmeyer 98

CISS

Life Cycle : Testing as a Process

plan

ning

& co

ntro

l

prep

arat

ion

test

gene

ratio

n

test

exec

utio

n

com

plet

ion

T

L

OI

CISS


plan

ning

& co

ntro

l

prep

arat

ion

test

gene

ratio

n

test

exec

utio

n

com

plet

ion

T

L

OI

• Testing as phase of development process vs.

CISS


plan

ning

& co

ntro

l

prep

arat

ion

test

gene

ratio

n

test

exec

utio

n

com

plet

ion

T

L

OI


• Testing as a process itself• with its own phases

• in parallel with development process

CISS


plan

ning

& co

ntro

l

prep

arat

ion

test

gene

ratio

n

test

exec

utio

n

com

plet

ion

T

L

OI


• Testing as a process itself• with its own phases

• in parallel with development process

quality report

implementationspecificationrequirements

CISS

Challenges of Testing

CISS

Challenges of Testing• New embedded systems

• more functionality• increasingly advanced• faster time-to-market• higher quality

CISS



• Testing • more to be tested• more complicated• in less time• more thorough

CISS



• Testing • more to be tested• more complicated• in less time• more thorough

• skilled developers and testers• advanced testing tools and techniques• well organized• using solid development method

CISS

Summary

CISS

Some Testing Principles

CISS

Some Testing Principles• Testing starts during the requirements phase

CISS

Some Testing Principles• Testing starts during the requirements phase• The programmer shall not be the tester

CISS

Some Testing Principles• Testing starts during the requirements phase• The programmer shall not be the tester• A test case specifies the test inputs

and the expected outputs

CISS


and the expected outputs• Test cases shall also cover invalid and unexpected inputs

CISS


and the expected outputs• Test cases shall also cover invalid and unexpected inputs• Test cases shall test that the program does what it should do

and that it does not do what it should not do

CISS



and that it does not do what it should not do• Test cases shall be recorded for reuse

CISS



and that it does not do what it should not do• Test cases shall be recorded for reuse• A test is successful when it detects an error

( but the project manager thinks differently ! )

CISS



and that it does not do what it should not do• Test cases shall be recorded for reuse• A test is successful when it detects an error

( but the project manager thinks differently ! )• No risk, no test

CISS

Lessons• Testing starts when project starts• Have a development approach

• How to develop (waterfall, cyclic, etc) : roles and activities• Clear expectations between testers and developers• Clarity about phases and tasks• Easier planning

• Test execution takes 3 steps• Test, fix, re-test

• Let the programmers find bugs• 80 % of bugs can be found by programmers (but they don’t)

• Every morning discuss the issues found• Testers create test report at end of the day• Next morning discuss which issues have to be solved• Programmer and tester fix issue together

CISS

Non-Structured testing• Start too late• Don’t organize and manage your test process• No test plan (V&V plan)• Unclear responsibilities• Conflicting interests• No fixed or stable specifications• Don’t learn

• from a defect• from a test• from statistics

• Use test tools in a chaotic environment• No configuration management and change control• No standard test procedures• Don’t educate• Do not provide career options

CISS

Model Based Development

CISS

Traditional Software Development

The Waterfall Model

Analyse

Design

Implementation

Testing♦Costly in time-to-market and money♦ Errors are detected late or never♦ Application of FM’s as early as possible

ProblemArea

Runn

ing

Syste

m

REVI

EWS

REVI

EWS

CISS

Models• A model is a simplified representation of the

real world.• Engineers use models to gain confidence in

the adequacy and validity of a proposed design.

• Focus on one or more aspects of interest:• Safety• Liveness• Peak time• Performance

CISS

Modelbased Validation

Design Model SpecificationVerification & Refusal

AnalysisValidation

FORMAL METHODS

ImplementationTesting

UML

Monitoring

CISS



AnalysisValidation

FORMAL METHODS


UML

Monitoring

AutomaticCode generation

CISS



AnalysisValidation

FORMAL METHODS


UML

Monitoring


AutomaticTest generation

CISS



AnalysisValidation

FORMAL METHODS


UML

Monitoring


AutomaticTest generation

AutomaticMonitoringgeneration

CISS

Multiple V-model

• Different objects, objectives, techniques in different stages

• Complexity of ES: functional decomposition and stepwise integration, but tight coordination (e.g independent development of HW and SW)

• Cost: Detect and correct errors earlier => Test earlier

Model Prototype(s)Development Iteration(s) Final Product

design

build

test design

build

test

design

build

test

CISS

END

CISS

Control Systems

CISS

Procedure P; begin while A do begin X; if B then if C then Y else Z else beginV; W end end end

CISS

Testing: A Definition

CISS

Testing: A DefinitionTesting is not:• static analysis• reviewing or inspection• walk-through• debugging• bug fixing• auditing• quality control• validation of the

specification

CISS

Testing: A DefinitionTesting is not:• static analysis• reviewing or inspection• walk-through• debugging• bug fixing• auditing• quality control• validation of the

specification

But testing:• is one of the

instrumentsfor measuring quality

• increases value of productby establishing confidencein its quality

• helps in assessing risk of putting product into operation

CISS

Verification & Validation

CISS

Verification & Validation• Verification :

CISS


building the product right

• concerns development process

• intermediate products

• e.g. checking the testability of the specification

CISS






• Validation :

CISS






• Validation :

building the right product

• whether final product meets requirements and needs

• e.g. by reviewing the specification

Documents

Introduction To Software Testing - Aalborg Universitetpeople.cs.aau.dk/~bnielsen/MBTV07/material/intro.pdfIntroduction To Software Testing Brian Nielsen [email protected] Center of