Introduction To Software Testing - Aalborg...

Introduction ToSoftware Testing

Brian Nielsen bnielsen@cs.auc.dkCenter of Embedded Software Systems

Aalborg University, Denmark

C SS10101110110101011011010101110111

Software development cycle ☺1. Programmer produces code he believes is bug-free.2. Product is tested. 20 bugs are found. 3. Programmer fixes 10 of the bugs and explains to the testing department that the other

10 aren't really bugs.4. Testing department finds that five of the fixes didn't work and discovers 15 new bugs.5. Repeat three times steps 3 and 4. 6. Due to marketing pressure and an extremely premature product announcement based

on overly-optimistic programming schedule, the product is released.7. Users find 137 new bugs.8. Original programmer, having cashed his royalty check, is nowhere to be found.9. Newly-assembled programming team fixes almost all of the 137 bugs, but introduce

456 new ones.10. Original programmer sends underpaid testing department a postcard from Fiji. Entire

testing department quits.11. Company is bought in a hostile takeover by competitor using profits from their latest

release, which had 783 bugs. 12. New CEO is brought in by board of directors. He hires a programmer to redo program

from scratch.13. Programmer produces code he believes is bug-free.

Autonomous Systems

Extreme environmental conditions

Safety Critical Systems

Rotterdam Storm Surge Barrier

Sensor-networks

Consumer Electronics

System Characteristics• Mass production• Limited Hardware (speed, power,memory)• Safety critical• Autonomous• Distributed• One-shot development• Control of physical environment• Mixed signal I/O• Extreme environments• State-based behavior• Real-time constraints (hard and soft)• Technical / scientific algorithms• …

Spectacular Software Bugs

• ARIANE-5 • http://www.cs.auc.dk/~ask/Undervisning/MVP/ariane5rep.html

• INTEL Pentium II floating-point division 470 Mill US $

• Baggage handling system, Denver 1.1 Mill US $/day for 9 months

• Mars Pathfinder • http://catless.ncl.ac.uk/Risks/19.49.html#subj1

• Radiation theraphy, Therac-25• http://www.cs.auc.dk/~ask/Undervisning/MVP/therac25.ps

• …….

Ariane 5 crash• An overrun exception stopped both replicated

position calculation computers (SRI’s) at time H0+36.7. Fail-stop was a wrong design decision

• Overrun caused by higher velocity in Ariane 5 compared to Ariane 4

• Overrun occured in a module which was irrelevant after take-off

• Simulations were not run with the new velocity (would have caught the failure)

Software Qualities

MaintainerGood Documentation

Readable CodeGood Design

Customer

Low CostPortability Increased

productivity

ReliabilityCorrectnessEfficiency

FunctionalityEase of useEase of learning

Costs of Poor Quality• Increased time to find and fix problems• Increased cost to distribute

modifications• Increased customer support• Product liability• Failure in the market place

Software Quality Assurance• Quality Control

• Process definition and standards • Formal technical review• Test planning and review

• Quality Assurance• Analysis and reporting• Measurement

Verification vs. Validation:

• Verification: • “are we building the product right?”• The software should conform to its specification

• Validation:• “are we building the right product?”• The software should do what the user really

requires

Verification vs. Validation:

• Verification: • “are we building the product right?”• The software should conform to its specification

• Validation:• “are we building the right product?”• The software should do what the user really

requires

Assuring that a software system meets a user’s needs

What is testing?

A Self-Assessment Test [Myers]

• “A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.”

• Write a set of test cases to test this program

Test cases for:

1. valid scalene triangle ?

Test cases for:

1. valid scalene triangle ?2. valid equilateral triangle ?

Test cases for:

1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?

Test cases for:

1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?

Test cases for:

1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?

Test cases for:

1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?

Test cases for:

1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?7. one side is sum of others ?

Test cases for:

1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?7. one side is sum of others ?8. 3 permutations of previous ?

Test cases for:

9. one side larger than sum ofothers ?

Test cases for:

10. 3 permutations of previous ?

Test cases for:

10. 3 permutations of previous ?11. all sides = 0 ?

Test cases for:

10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?

Test cases for:

10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?

Test cases for:

10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?14. for each test case: is

expected output specified ?

Test cases for:

10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?14. for each test case: is

expected output specified ?15. check behaviour after

output was produced ?

TestingTesting:• to check the quality (functionality, reliability,

performance, …) of an (software) object

-by performing experiments-in a controlled way

• In avg. 10-20 errors per 1000 LOC• 30-50 % of development time and cost in embedded software

What is a Test?

Software under Test

What is a Test?

Software under Test

Test Data

What is a Test?

Software under Test

Test Data Output

What is a Test?

Software under Test

Test Data Output

Correct result?

What is a Test?

Software under Test

Test Data Output

Correct result?

Oracle

What is a Test?

Software under Test

Test Data Output

Correct result?

Oracle

Test Cases

Types of Testing

integration

system

performancerobustness

functional behaviour

white box black box

Accessibility

Aspect

usability

reliability

Types of Testing

integration

system

performancerobustness

functional behaviour

white box black box

Accessibility

Aspect

usability

reliability

Quality-Characteristics (ISO-9126)• Functionality

• Suitability, accuracy, security, compliance, interoperability

• Reliability• maturity, fault tolerance, recoverability

• Usability• understandability, learnability, operability

• Efficiency• time behaviour, resource utilization

• Maintainability• Analysability, changeability, stability, testability

• Portability• Adaptability, intallability, conformance, replaceability

Quality-Characteristics (ISO-9126)• Functionality

• Suitability, accuracy, security, compliance, interoperability

• Reliability• maturity, fault tolerance, recoverability

• Usability• understandability, learnability, operability

• Efficiency• time behaviour, resource utilization

• Maintainability• Analysability, changeability, stability, testability

• Portability• Adaptability, intallability, conformance, replaceability

⇒ functional testing

⇒ reliability testing

⇒ usability testing

⇒ performance testing

⇒maintainability testing ??

⇒ portability testing ?

Module testtest driver

Module under test

test stub test stub

Whitebox Exampleint invoice (int x, int y) {int d1, d2, s;if (x<=30) d2=100;else d2=90;s=5*x + 10 *y;if (s<=200) d1=100;else if (s<=1000) d1 = 95;

else d1 = 80;return (s*d1*d2/10000);

Test Cases

else d1 = 80;return (s*d1*d2/10000);

Test Cases

ifd2=100 d2=90

ifd1=100

return

s>1000

else d1 = 80;return (s*d1*d2/10000);

Test Cases

ifd2=100 d2=90

ifd1=100

return

s>1000

Test Data Expected Output

X=5 Y=5 75

X=31 Y=10 229.5

X=30 Y=100 977.5

Black box testing

requirements

events

output

domain testing

Module IntegrationM1

M2 M3 M4

M11 M12

M9M7 M8

”calls””uses”

”interacts”

Embedded system

Generic Embedded System

User interface

Interfaces to other systems

Processing Unit

OSI/O-d

s Application Software

ntSystem environment

System test

• 2*CRTG (4 channels) 2 * 200 k€

Test Equipment

• Complete Type Approval Test System (3 M€)

Who Should Test?

• Developer• Understands the system• But, will test gently• And, is driven by

deadlines

Who Should Test?

deadlines

Who Should Test?

deadlines

• Independent tester• Must learn system• But, will attempt to break it• And, is driven by “quality”

Who Should Test?

deadlines

• Independent tester• Must learn system• But, will attempt to break it• And, is driven by “quality”

Testing Objectives• To identify as many errors as possible• To bring the software to an acceptable level of

quality• To determine risk of release• Perform testing efficiently, effectively and within

budget and time constraints• Compile a record of software errors for future

error prevention and correction (learning organization).

Risk Based Test Strategy• Make best possible use of resources by

identifying and prioritizing quality aspects and subsystems• Higher risk ⇒ more testing• No risk ⇒ no testing

• Risk = chance of failure × damage

Use frequencyChance of error being present

ComplexityNew tools/techniquesInexperienced developers

Cost of repairLoss of market shareLegal claim

Testability = how easy can a program be tested

• operability - it operates cleanly• observability

• the results are easy to see• distinct output is generated for each input• incorrect output is easily identified

• controllability• processing can be controlled• tests can be automated & reproduced

• decomposability - software modules can be tested independently

• simplicity - no complex architecture and logic• stability - few changes are requested during testing• understandability - program is easy to understand

Testing• Quality assurance activities by running code –

examines functionality in the form used by customers

• Software testing is a formal process carried out by a specialized testing team in which a software unit, several integrated software units or an entire software package are examined by running programs on a computer. All the associated tests are performed according to approved test procedures on approved test cases.

Testing• Dynamic testing is the process of executing a

program or system with the intent of finding error (Glenford Meyers’ definition)

• Static testing is any activity that aims at finding defects by inspecting, reviewing, walking through, and analyzing any static component of the software (code, documents, and models)

• Debugging is an ad hoc activity performed by individual developers to find and remove bugs from a program.

• Testing is a planned activity

There is More than Testing

static analysisstatic analysis

reviewingreviewinginspectioninspection

walk-throughwalk-throughdebuggingdebugging

certificationcertification

CMMCMM

quality controlquality controlrequirement managementrequirement managementsoftware processsoftware process

Lifecycles & testing

CISSimplementation

detailed design

requirements

specification

acceptancetest

systemtest

moduletest

unit-test

V - Model

integrationtest

architecturespec

acceptance test spec

system test spec

integration test spec

module test spec

unit test spec

Introducing, Detecting andRepairing Errors Liggesmeyer 98

Life Cycle : Testing as a Process

• Testing as phase of development process vs.

• Testing as a process itself• with its own phases

• in parallel with development process

• Testing as a process itself• with its own phases

• in parallel with development process

quality report

implementationspecificationrequirements

Challenges of Testing

Challenges of Testing• New embedded systems

• more functionality• increasingly advanced• faster time-to-market• higher quality

• Testing • more to be tested• more complicated• in less time• more thorough

• skilled developers and testers• advanced testing tools and techniques• well organized• using solid development method

Summary

Some Testing Principles

Some Testing Principles• Testing starts during the requirements phase

Some Testing Principles• Testing starts during the requirements phase• The programmer shall not be the tester

Some Testing Principles• Testing starts during the requirements phase• The programmer shall not be the tester• A test case specifies the test inputs

and the expected outputs

and the expected outputs• Test cases shall also cover invalid and unexpected inputs

and the expected outputs• Test cases shall also cover invalid and unexpected inputs• Test cases shall test that the program does what it should do

and that it does not do what it should not do

and that it does not do what it should not do• Test cases shall be recorded for reuse

and that it does not do what it should not do• Test cases shall be recorded for reuse• A test is successful when it detects an error

( but the project manager thinks differently ! )

and that it does not do what it should not do• Test cases shall be recorded for reuse• A test is successful when it detects an error

( but the project manager thinks differently ! )• No risk, no test

Lessons• Testing starts when project starts• Have a development approach

• How to develop (waterfall, cyclic, etc) : roles and activities• Clear expectations between testers and developers• Clarity about phases and tasks• Easier planning

• Test execution takes 3 steps• Test, fix, re-test

• Let the programmers find bugs• 80 % of bugs can be found by programmers (but they don’t)

• Every morning discuss the issues found• Testers create test report at end of the day• Next morning discuss which issues have to be solved• Programmer and tester fix issue together

Non-Structured testing• Start too late• Don’t organize and manage your test process• No test plan (V&V plan)• Unclear responsibilities• Conflicting interests• No fixed or stable specifications• Don’t learn

• from a defect• from a test• from statistics

• Use test tools in a chaotic environment• No configuration management and change control• No standard test procedures• Don’t educate• Do not provide career options

Model Based Development

Traditional Software Development

The Waterfall Model

Analyse

Design

Implementation

Testing♦Costly in time-to-market and money♦ Errors are detected late or never♦ Application of FM’s as early as possible

ProblemArea

Models• A model is a simplified representation of the

real world.• Engineers use models to gain confidence in

the adequacy and validity of a proposed design.

• Focus on one or more aspects of interest:• Safety• Liveness• Peak time• Performance

Modelbased Validation

Design Model SpecificationVerification & Refusal

AnalysisValidation

FORMAL METHODS

ImplementationTesting

Monitoring

AnalysisValidation

FORMAL METHODS

Monitoring

AutomaticCode generation

AnalysisValidation

FORMAL METHODS

Monitoring

AutomaticTest generation

AnalysisValidation

FORMAL METHODS

Monitoring

AutomaticTest generation

AutomaticMonitoringgeneration

Multiple V-model

• Different objects, objectives, techniques in different stages

• Complexity of ES: functional decomposition and stepwise integration, but tight coordination (e.g independent development of HW and SW)

• Cost: Detect and correct errors earlier => Test earlier

Model Prototype(s)Development Iteration(s) Final Product

design

test design

design

Control Systems

Procedure P; begin while A do begin X; if B then if C then Y else Z else beginV; W end end end

Testing: A Definition

Testing: A DefinitionTesting is not:• static analysis• reviewing or inspection• walk-through• debugging• bug fixing• auditing• quality control• validation of the

specification

Testing: A DefinitionTesting is not:• static analysis• reviewing or inspection• walk-through• debugging• bug fixing• auditing• quality control• validation of the

specification

But testing:• is one of the

instrumentsfor measuring quality

• increases value of productby establishing confidencein its quality

• helps in assessing risk of putting product into operation

Verification & Validation

Verification & Validation• Verification :

building the product right

• concerns development process

• intermediate products

• e.g. checking the testability of the specification

• Validation :

building the right product

• whether final product meets requirements and needs

• e.g. by reviewing the specification

Introduction To Software Testing - Aalborg...

Documents

Brian Nielsen Arne Skou - Aalborg Universitetpeople.cs.aau.dk/~bnielsen/TOV07/lektioner/cunit-intro-07.pdf · Brian Nielsen Arne Skou {bnielsen ... A set of “Frameworks” for programming

September 29, 20021 Algorithms and Data Structures Lecture V Simonas Šaltenis Aalborg University simas@cs.auc.dk

Introduction to Distributed Systems - Aalborg Universitetpeople.cs.aau.dk/~bnielsen/DS-E08/material/intro.pdf · Introduction to Distributed Systems ... real-time, distributed systems

NXT HWNXT HW Sensors and Actuators - people.cs.aau.dkpeople.cs.aau.dk/~bnielsen/sw5-es-io.pdf · NXT HWNXT HW Sensors and Actuators Brian NielsenBrian Nielsen bnielsen@cs.aau.dk

October 23, 20031 Algorithms and Data Structures Lecture X Simonas Šaltenis Aalborg University simas@cs.auc.dk

Model Based Testing -- FSM based testing - …people.cs.aau.dk/~bnielsen/TOV07/lektioner/tov-fsm-test.pdf · Model Based Testing--FSM based testing Brian Nielsen {bnielsen}@cs.aau.dk

November 5, 20011 Algorithms and Data Structures Lecture VIII Simonas Šaltenis Nykredit Center for Database Research Aalborg University simas@cs.auc.dk

Introduction to FSM-modelling and testpeople.cs.aau.dk/~bnielsen/TOV07/material/fsm-intro.pdf · Introduction to FSM-modelling and test Brian Nielsen bnielsen@cs.auc.dk Department

September 12, 20021 Algorithms and Data Structures Lecture III Simonas Šaltenis Nykredit Center for Database Research Aalborg University simas@cs.auc.dk

October 1, 20011 Algorithms and Data Structures Lecture III Simonas Šaltenis Nykredit Center for Database Research Aalborg University simas@cs.auc.dk

Design for Systemadministrationpeople.cs.aau.dk/~bnielsen/sw6-07/material/fischer-design.pdf · teknologier, software-systemer, arbejdsmetoder • Undgå én medarbejder pr. system

October 16, 20031 Algorithms and Data Structures Lecture IX Simonas Šaltenis Aalborg University simas@cs.auc.dk

September 15, 20031 Algorithms and Data Structures Simonas Šaltenis Aalborg University simas@cs.auc.dk

October 3, 20021 Algorithms and Data Structures Lecture VII Simonas Šaltenis Nykredit Center for Database Research Aalborg University simas@cs.auc.dk

Introduction: Historical linguistics as a ...gerlin.phil-fak.uni-koeln.de/kvh/pub/pub03/Heusinger2003_WIT Intro.pdfIntroduction: Historical linguistics as a transdisciplinary field

Introduction to Numerical Algebraic Geometryjan/Articles/intro.pdfIntroduction to Numerical Algebraic Geometry AndrewJ.Sommese1?,JanVerschelde2??,andCharlesW.Wampler3 1 Department

Introduction: Software Testing and Quality Assuranceagurfink/stqam.w18/assets/pdf/W01P1-Intro.pdfIntroduction: Software Testing and Quality Assurance Software Testing, Quality Assurance,

September 17, 2001 Algorithms and Data Structures Lecture II Simonas Šaltenis Nykredit Center for Database Research Aalborg University simas@cs.auc.dk

Microsoft .NET Remoting - people.cs.aau.dkpeople.cs.aau.dk/~bnielsen/DS-E08/material/NetRemoting.pdf · 2008. 9. 23. · – not assigned to specific client – Singleton: framework

Early Performance Testing of Distributed Software ...people.cs.aau.dk/~bnielsen/DSE06/papers/p94-denaro.pdf · Early Performance Testing of Distributed Software Applications*