Theory and Practice of Software Testing

Chapter 13 (Pressman)Software Testing Strategies

Testing

To check the quality of something by performing experiments in controlled way

Definition

Myers “Testing is the process of executing a program with the intent of finding errors.”

[ISO] “Testing is a technical operation that consists of the determination of one or more characteristics of a given product, process, or service according to a specified procedure.”

[Pol] “Testing is a process of planning, preparationand measurement aimed at establishing thecharacteristics of an information system anddemonstrating the difference between the actual and the required status.”

Why are we Testing ?

Testing as measurement for software quality Software quality as a measure to predict future operation:

reliability, dependability expected value, expected costs, probability of failure

Risk probability of failure expected cost of failure

systemtest

acceptancetest

spec construction deliver exploit & use destroyaccept

quality?test it !

A Strategic Approach to SW Testing

Verification and validation (V&V) Verifications: Are we building the

product right? Validation: Are we building the right

product?

More on V & V

ValidationThe process of checking whether the system being built is what the customer expects or needs

VerificationThe process of checking any deliverable of the software process meets defined needs or requirements

Verification is only as good as the validity of the model

on which it is based

Verification is only as good as the validity of the model

on which it is based

Verification vs. TestingVerification: Formal manipulation prove properties performed on the model

Testing: experimentation show error Performed on actual system

Testing can only show the presence of errors, not their

absence

Testing can only show the presence of errors, not their

absence

Organizing for SW Testing

A software testing strategy for conventional software architectures Spiral model V model

SW Testing Strategy : Spiral Model

System Testing

System eng.

validation Testing

Requirements

Integration Testing

Design

Unit Testing

Code

Requirementsspecification

Systemspecification

Systemdesign

Detaileddesign

Module andunit codeand tess

Sub-systemintegrationtest plan

Systemintegrationtest plan

Acceptancetest plan

ServiceAcceptance

testSystem

integration testSub-system

integration test

V-Model

Types of Testing

unit

integration

system

performancerobustness

functional behaviour

white box

black box

Level of detail

Accessibility

Characteristics

usability

reliability

module

security

Unit testing

interface, local data structures, boundary conditions, independent paths, error handling

From Textbook: Ian Sommerville, Software

Engineering, 6th Edition

Unit Testing Process

Design testcases

Prepare testdata

Run programwith test data

Compare resultsto test cases

Testcases

Testdata

Testresults

Testreports

Goals of Unit Testing

Verify code against interface and functional specifications

Check both “good” and “bad” inputs against design and code implementation

Check all possible paths of execution through code

Integration testing Tests complete systems or subsystems

composed of integrated components Integration testing should be black-box

testing with tests derived from the specification

Main difficulty is localising errors Incremental integration testing reduces

this problem

Incremental integration testing

T3

T2

T1

T4

T5

A

B

C

D

T2

T1

T3

T4

A

B

C

T1

T2

T3

A

B

Test sequence1

Test sequence2

Test sequence3

Integration testing strategies Top-down integration Bottom-up integration Regression testing Smoke testing

An initial set of tests that shows if a new version of the application performs well enough for further testing

Used for time-critical projects Assess the project on a frequent basis

Approaches to Integration Testing

Top-down testing Start with high-level system and integrate

from the top-down replacing individual components by stubs where appropriate

Depth-first vs. breadth-first Bottom-up testing

Integrate individual components in levels until the complete system is created

In practice, most integration involves a combination of these strategies

Top-down testing

Bottom-up testingM

Top-down Vs. Bottom-up Architectural validation

Top-down integration testing is better at discovering errors in the system architecture

System demonstration Top-down integration testing allows a limited

demonstration at an early stage in the development Test implementation

Often easier with bottom-up integration testing Test observation

Problems with both approaches. Extra code may be required to observe tests

Guideline for Integration testing

Integrate the component that implement the most frequently used functionality

Perform regression testing for existing and new functionality

Validation testing

Validation test criteria All functional, all nonfunctional

Configuration review (auditing) Ensures that all system elements

have been properly documented Alpha/beta testing

System testing (1) Recovery testing

Forces the software to crash and shows the adequacy of recovery

E.g., use the back button after submitting the credit card information

Security testing Tests adequacy of system protection

mechanisms E.g., how does the website authenticate

users?

System testing (2) Stress testing/load testing

Subjecting the system to varying and maximum loads to evaluate the resulting performance

E.g., how can you overwhelm the system?

Performance testing Used to evaluate system performance

under normal and heavy usage E.g., how long should a task take?

More on Stress testing Exercises the system beyond its maximum

design load. Stressing the system often causes defects to

come to light

Stressing the system test failure behaviour. Systems should not fail catastrophically. checks for unacceptable loss of service or data

Particularly relevant to distributed systems exhibit severe degradation as a

network becomes overloaded