Stan Kurkovsky

Software EngineeringSoftware Engineering

Software TestingSoftware Testing

Based on Software Engineering, 7th Edition by Ian Sommerville

Stan Kurkovsky

ObjectivesObjectives

•• To discuss the distinctions between validation testing and defecTo discuss the distinctions between validation testing and defect testingt testing•• To describe the principles of system and component testingTo describe the principles of system and component testing•• To describe strategies for generating system test casesTo describe strategies for generating system test cases•• To understand the essential characteristics of tool used for tesTo understand the essential characteristics of tool used for test t

automationautomation

Stan Kurkovsky

The testing processThe testing process

•• Component testing Component testing •• Testing of individual program components;Testing of individual program components;•• Usually the responsibility of the component developer (except soUsually the responsibility of the component developer (except sometimes for metimes for

critical systems);critical systems);•• Tests are derived from the developerTests are derived from the developer’’s experience.s experience.

•• System testingSystem testing•• Testing of groups of components integrated to create a system orTesting of groups of components integrated to create a system or subsub--

system;system;•• The responsibility of an independent testing team;The responsibility of an independent testing team;•• Tests are based on a system specification.Tests are based on a system specification.

Componenttesting

Systemtesting

Software developer Independent testing team

Stan Kurkovsky

Testing process goalsTesting process goals

•• Validation testingValidation testing•• To demonstrate to the developer and the system customer that theTo demonstrate to the developer and the system customer that the software software

meets its requirements;meets its requirements;•• A successful test shows that the system operates as intended.A successful test shows that the system operates as intended.

•• Defect testingDefect testing•• To discover faults or defects in the software where its behaviorTo discover faults or defects in the software where its behavior is incorrect or is incorrect or

not in conformance with its specification;not in conformance with its specification;•• A successful test is a test that makes the system perform incorrA successful test is a test that makes the system perform incorrectly and so ectly and so

exposes a defect in the system.exposes a defect in the system.

Design testcases

Prepare testdata

Run programwith test da ta

Compare resultsto test cases

Testcases

Testdata

Testresults

Testreports

Stan Kurkovsky

Testing policiesTesting policies

•• Only exhaustive testing can show a program is free from defects.Only exhaustive testing can show a program is free from defects.However, exhaustive testing is impossible,However, exhaustive testing is impossible,

•• Testing policies define the approach to be used in selecting sysTesting policies define the approach to be used in selecting system tests:tem tests:•• All functions accessed through menus should be tested;All functions accessed through menus should be tested;•• Combinations of functions accessed through the same menu should Combinations of functions accessed through the same menu should be be

tested;tested;•• Where user input is required, all functions must be tested with Where user input is required, all functions must be tested with correct and correct and

incorrect input.incorrect input.

Stan Kurkovsky

System testingSystem testing

•• Involves integrating components to create a system or subInvolves integrating components to create a system or sub--system.system.•• May involve testing an increment to be delivered to the customerMay involve testing an increment to be delivered to the customer..•• Two phases:Two phases:

•• Integration testingIntegration testing -- the test team have access to the system source code. the test team have access to the system source code. The system is tested as components are integrated.The system is tested as components are integrated.

•• Release testingRelease testing -- the test team test the complete system to be delivered as a the test team test the complete system to be delivered as a blackblack--box.box.

Stan Kurkovsky

Integration testingIntegration testing

•• Involves building a system from its components and testing it foInvolves building a system from its components and testing it for r problems that arise from component interactions.problems that arise from component interactions.

•• TopTop--down integrationdown integration•• Develop the skeleton of the system and populate it with componenDevelop the skeleton of the system and populate it with components.ts.

•• BottomBottom--up integrationup integration•• Integrate infrastructure components then add functional componenIntegrate infrastructure components then add functional components.ts.

•• To simplify error localization, systems should be incrementally To simplify error localization, systems should be incrementally integrated.integrated.

T3

T2

T1

T4

T5

A

B

C

D

T2

T1

T3

T4

A

B

C

T1

T2

T3

A

B

Test sequence 1 Test sequence 2 Test sequence 3

Stan Kurkovsky

Release testingRelease testing

•• The process of testing a release of a system that will be distriThe process of testing a release of a system that will be distributed to buted to customers.customers.

•• Primary goal is to increase the supplierPrimary goal is to increase the supplier’’s confidence that the system s confidence that the system meets its requirements.meets its requirements.

•• Release testing is usually blackRelease testing is usually black--box or functional testingbox or functional testing•• Based on the system specification only;Based on the system specification only;•• Testers do not have knowledge of the system implementation.Testers do not have knowledge of the system implementation.

•• Black box testingBlack box testingIeInput test data

OeOutput test results

System

Inputs causinganomalousbehavior

Outputs which revealthe presence ofdefects

Stan Kurkovsky

Testing guidelinesTesting guidelines

•• Testing guidelines are hints for the testing team to help them cTesting guidelines are hints for the testing team to help them choose hoose tests that will reveal defects in the systemtests that will reveal defects in the system•• Choose inputs that force the system to generate all error messagChoose inputs that force the system to generate all error messages;es;•• Design inputs that cause buffers to overflow;Design inputs that cause buffers to overflow;•• Repeat the same input or input series several times;Repeat the same input or input series several times;•• Force invalid outputs to be generated;Force invalid outputs to be generated;•• Force computation results to be too large or too small.Force computation results to be too large or too small.

•• Use casesUse cases•• Use cases can be a basis for deriving the tests for a system. ThUse cases can be a basis for deriving the tests for a system. They help ey help

identify operations to be tested and help design the required teidentify operations to be tested and help design the required test cases.st cases.•• From an associated sequence diagram, the inputs and outputs to bFrom an associated sequence diagram, the inputs and outputs to be created e created

for the tests can be identified.for the tests can be identified.

Stan Kurkovsky

Performance testingPerformance testing

•• Part of release testing may involve testing the emergent propertPart of release testing may involve testing the emergent properties of a ies of a system, such as performance and reliability.system, such as performance and reliability.

•• Performance tests usually involve planning a series of tests whePerformance tests usually involve planning a series of tests where the re the load is steadily increased until the system performance becomes load is steadily increased until the system performance becomes unacceptable.unacceptable.

Stress testingStress testing•• Exercises the system beyond its maximum design load. Stressing tExercises the system beyond its maximum design load. Stressing the he

system often causes defects to come to light.system often causes defects to come to light.•• Stressing the system test failure behavior. Systems should not fStressing the system test failure behavior. Systems should not fail ail

catastrophically. Stress testing checks for unacceptable loss ofcatastrophically. Stress testing checks for unacceptable loss of service or service or data.data.

•• Stress testing is particularly relevant to distributed systems tStress testing is particularly relevant to distributed systems that can hat can exhibit severe degradation as a network becomes overloaded.exhibit severe degradation as a network becomes overloaded.

Stan Kurkovsky

Component testingComponent testing

•• Component or unit testing is the process of testing individual cComponent or unit testing is the process of testing individual components omponents in isolation.in isolation.

•• It is a defect testing process.It is a defect testing process.•• Components may be:Components may be:

•• Individual functions or methods within an object;Individual functions or methods within an object;•• Object classes with several attributes and methods;Object classes with several attributes and methods;•• Composite components with defined interfaces used to access theiComposite components with defined interfaces used to access their r

functionality.functionality.

Object class testingObject class testing•• Complete test coverage of a class involvesComplete test coverage of a class involves

•• Testing all operations associated with an object;Testing all operations associated with an object;•• Setting and interrogating all object attributes;Setting and interrogating all object attributes;•• Exercising the object in all possible states.Exercising the object in all possible states.

•• Inheritance makes it more difficult to design object class testsInheritance makes it more difficult to design object class tests as the as the information to be tested is not localized.information to be tested is not localized.

Stan Kurkovsky

Interface testingInterface testing

Interface typesInterface types•• Parameter interfacesParameter interfaces

•• Data passed from one procedure Data passed from one procedure to another.to another.

•• Shared memory interfacesShared memory interfaces•• Block of memory is shared Block of memory is shared

between procedures or functions.between procedures or functions.

•• Procedural interfacesProcedural interfaces•• SubSub--system encapsulates a set of system encapsulates a set of

procedures to be called by other procedures to be called by other subsub--systems.systems.

•• Message passing interfacesMessage passing interfaces•• SubSub--systems request services systems request services

from other subfrom other sub--systems.systems.

Interface errorsInterface errors•• Interface misuseInterface misuse

•• A calling component calls another A calling component calls another component and makes an error in component and makes an error in its use of its interface e.g. its use of its interface e.g. parameters in the wrong order.parameters in the wrong order.

•• Interface misunderstandingInterface misunderstanding•• A calling component embeds A calling component embeds

assumptions about the behavior of assumptions about the behavior of the called component which are the called component which are incorrect.incorrect.

•• Timing errorsTiming errors•• The called and the calling The called and the calling

component operate at different component operate at different speeds and outspeeds and out--ofof--date date information is accessed.information is accessed.

•• Objectives are to detect faults due to interface errors or invalObjectives are to detect faults due to interface errors or invalid id assumptions about interfaces.assumptions about interfaces.

•• Particularly important for objectParticularly important for object--oriented development as objects are oriented development as objects are defined by their interfaces.defined by their interfaces.

Stan Kurkovsky

Test case designTest case design

•• Involves designing the test cases (inputs and outputs) used to tInvolves designing the test cases (inputs and outputs) used to test the est the system.system.

•• The goal of test case design is to create a set of tests that arThe goal of test case design is to create a set of tests that are effective in e effective in validation and defect testing.validation and defect testing.

•• Design approaches:Design approaches:•• RequirementsRequirements--based testing;based testing;•• Partition testing;Partition testing;•• Structural testing.Structural testing.

Stan Kurkovsky

Requirements based testingRequirements based testing

•• A general principle of requirements engineering is that requiremA general principle of requirements engineering is that requirements ents should be testable.should be testable.

•• RequirementsRequirements--based testing is a validation testing technique where you based testing is a validation testing technique where you consider each requirement and derive a set of tests for that reqconsider each requirement and derive a set of tests for that requirement.uirement.

Stan Kurkovsky

Partition testingPartition testing

•• Input data and output results often fall into different classes Input data and output results often fall into different classes where all where all members of a class are related.members of a class are related.

•• Each of these classes is an Each of these classes is an equivalence partitionequivalence partition or domain where the or domain where the program behaves in an equivalent way for each class member.program behaves in an equivalent way for each class member.

•• Test cases should be chosen from each partition.Test cases should be chosen from each partition.

Stan Kurkovsky

Structural testingStructural testing

•• Sometime called whiteSometime called white--box testing.box testing.•• Derivation of test cases according to program structure. KnowledDerivation of test cases according to program structure. Knowledge of the ge of the

program is used to identify additional test cases.program is used to identify additional test cases.•• Objective is to exercise all program statements (not all path Objective is to exercise all program statements (not all path

combinations).combinations).

Stan Kurkovsky

Path testingPath testing

•• The objective of path testing is to ensure that the set of test The objective of path testing is to ensure that the set of test cases is cases is such that each path through the program is executed at least oncsuch that each path through the program is executed at least once.e.

•• The starting point for path testing is a program flow graph thatThe starting point for path testing is a program flow graph that shows shows nodes representing program decisions and arcs representing the fnodes representing program decisions and arcs representing the flow of low of control.control.

•• Statements with conditions are therefore nodes in the flow graphStatements with conditions are therefore nodes in the flow graph..

Stan Kurkovsky

Test automationTest automation

•• Testing is an expensive process phase. Testing workbenches proviTesting is an expensive process phase. Testing workbenches provide a de a range of tools to reduce the time required and total testing cosrange of tools to reduce the time required and total testing costs.ts.

•• Systems such as JUnit support the automatic execution of tests.Systems such as JUnit support the automatic execution of tests.•• Most testing workbenches are open systems because testing needs Most testing workbenches are open systems because testing needs are are

organizationorganization--specific.specific.•• They are sometimes difficult to integrate with closed design andThey are sometimes difficult to integrate with closed design and analysis analysis

workbenches.workbenches.

Stan Kurkovsky

Key pointsKey points

•• Testing can show the presence of faults in a system; it cannot pTesting can show the presence of faults in a system; it cannot prove rove there are no remaining faults.there are no remaining faults.

•• Component developers are responsible for component testing; systComponent developers are responsible for component testing; system em testing is the responsibility of a separate team.testing is the responsibility of a separate team.

•• Integration testing is testing increments of the system; releaseIntegration testing is testing increments of the system; release testing testing involves testing a system to be released to a customer.involves testing a system to be released to a customer.

•• Use experience and guidelines to design test cases in defect tesUse experience and guidelines to design test cases in defect testing.ting.•• Interface testing is designed to discover defects in the interfaInterface testing is designed to discover defects in the interfaces of ces of

composite components.composite components.•• Equivalence partitioning is a way of discovering test cases Equivalence partitioning is a way of discovering test cases -- all cases in a all cases in a

partition should behave in the same way.partition should behave in the same way.•• Structural analysis relies on analyzing a program and deriving tStructural analysis relies on analyzing a program and deriving tests from ests from

this analysis.this analysis.•• Test automation reduces testing costs by supporting the test proTest automation reduces testing costs by supporting the test process with cess with

a range of software tools.a range of software tools.