View
250
Download
0
Embed Size (px)
Citation preview
Software Engineering Software Testing Slide 1
Software Engineering
Software Testing
The material is this presentation is based on the following references and other internet resources:
•Ian Sommerville, Software Engineering (Seventh Edition), Addison-Wesley, 2004.
•Roger Pressman, Software Engineering, A Practitioner Approach, 6th ed., McGraw Hill, 2005.
Software Engineering Software Testing Slide 2
Objectives• To discuss the distinctions between validation
testing and defect testing
• To describe the principles of system and component testing
• To describe strategies for generating system test cases
• To understand the essential characteristics of tool used for test automation
Software Engineering Software Testing Slide 3
Topics covered• System testing
• Component testing
• Test case design
• Test automation
Software Engineering Software Testing Slide 4
Testability• Operability—it operates cleanly• Observability—the results of each test case are
readily observed• Controllability—the degree to which testing can be
automated and optimized• Decomposability—testing can be targeted• Simplicity—reduce complex architecture and logic to
simplify tests• Stability—few changes are requested during testing• Understandability—of the design
Software Engineering Software Testing Slide 5
What is a “Good” Test?
• A good test has a high probability of finding an errorA good test has a high probability of finding an error
• A good test is not redundant.A good test is not redundant.
• A good test should be “best of breed” A good test should be “best of breed”
• A good test should be neither too simple nor too complexA good test should be neither too simple nor too complex
• A good test has a high probability of finding an errorA good test has a high probability of finding an error
• A good test is not redundant.A good test is not redundant.
• A good test should be “best of breed” A good test should be “best of breed”
• A good test should be neither too simple nor too complexA good test should be neither too simple nor too complex
Software Engineering Software Testing Slide 6
What Testing Shows
errors
requirements conformance
performance
an indicationof quality
Software Engineering Software Testing Slide 7
Who Tests the Software?
DeveloperUnderstands the system but, will test "gently“ and, is driven by "delivery"
independent testerMust learn about the system, but, will attempt to break it and, is driven by quality
Software Engineering Software Testing Slide 8
Testing phases
Componenttesting
Systemtesting
Software developer Independent testing team
Software Engineering Software Testing Slide 9
Testing Strategy• We begin by ‘testing-in-the-small’ and move toward ‘testing-
in-the-large’
• For conventional software
– The module (component) is our initial focus
– Integration of modules follows
• For OO software
– our focus when “testing in the small” changes from an individual module (the conventional view) to an OO class that encompasses attributes and operations and implies communication and collaboration
Software Engineering Software Testing Slide 10
Testing Strategyunit test integration
test
systemtest
validationtest
Software Engineering Software Testing Slide 11
The testing process• Component testing
– Testing of individual program components;
– Usually the responsibility of the component developer (except sometimes for critical systems);
– Tests are derived from the developer’s experience.
• System testing– Testing of groups of components integrated to create a
system or sub-system;
– The responsibility of an independent testing team;
– Tests are based on a system specification.
Software Engineering Software Testing Slide 12
Defect testing• The goal of defect testing is to discover defects
in programs
• A successful defect test is a test which causes a program to behave in an anomalous way
• Tests show the presence not the absence of defects
Software Engineering Software Testing Slide 13
Testing process goals• Validation testing
– To demonstrate to the developer and the system customer that the software meets its requirements.
– A successful test shows that the system operates as intended.
• Defect testing– To discover faults or defects in the software where its
behavior is incorrect or not in conformance with its specification.
– A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system.
Software Engineering Software Testing Slide 14
The software testing process
Design testcases
Prepare testdata
Run programwith test data
Compare resultsto test cases
Testcases
Testdata
Testresults
Testreports
Software Engineering Software Testing Slide 15
Testing policies• Only exhaustive testing can show a program is free
from defects. However, exhaustive testing is impossible,
• Testing policies define the approach to be used in selecting system tests:– All functions accessed through menus should be tested;
– Combinations of functions accessed through the same menu should be tested;
– Where user input is required, all functions must be tested with correct and incorrect input.
Software Engineering Software Testing Slide 16
Exhaustive Testing
loop < 20 Xloop < 20 X
Software Engineering Software Testing Slide 17
Selective Testing
loop < 20 Xloop < 20 X
Selected pathSelected path
Software Engineering Software Testing Slide 18
Component testing• Component or unit testing is the process of
testing individual components in isolation.
• It is a defect testing process.
• Components may be:– Individual functions or methods within an object;
– Object classes with several attributes and methods;
– Composite components with defined interfaces used to access their functionality.
Software Engineering Software Testing Slide 19
Unit Testing
test cases
results
Software engineer
moduleto betested
Software Engineering Software Testing Slide 20
Unit Testing
modulemoduleto beto betestedtested
Interfacelocal data structuresboundary conditionsindependent pathserror handling paths
test cases
results
Software Engineering Software Testing Slide 21
Object class testing• Complete test coverage of a class involves
– Testing all operations associated with an object;
– Setting and interrogating all object attributes;
– Exercising the object in all possible states.
• Inheritance makes it more difficult to design object class tests as the information to be tested is not localised.
Software Engineering Software Testing Slide 22
Interface testing• Objectives are to detect faults due to interface
errors or invalid assumptions about interfaces.
• Particularly important for object-oriented development as objects are defined by their interfaces.
Software Engineering Software Testing Slide 23
Interface testing
B
C
Testcases
A
Software Engineering Software Testing Slide 24
Interface types• Parameter interfaces
– Data passed from one procedure to another.
• Shared memory interfaces– Block of memory is shared between procedures or
functions.
• Procedural interfaces– Sub-system encapsulates a set of procedures to be called by
other sub-systems.
• Message passing interfaces– Sub-systems request services from other sub-systems
Software Engineering Software Testing Slide 25
Interface testing guidelines• Design tests so that parameters to a called procedure
are at the extreme ends of their ranges.
• Always test pointer parameters with null pointers.
• Design tests which cause the component to fail.
• Use stress testing in message passing systems.
• In shared memory systems, vary the order in which components are activated.
Software Engineering Software Testing Slide 26
Integration Testing Strategies
Options:• the “big bang”
approach• an incremental
construction strategy
Software Engineering Software Testing Slide 27
Integration testing• Involves building a system from its components and
testing it for problems that arise from component interactions.
• Top-down integration– Develop the skeleton of the system and populate it with
components.
• Bottom-up integration– Integrate infrastructure components then add functional
components.
• To simplify error localisation, systems should be incrementally integrated.
Software Engineering Software Testing Slide 28
Top Down Integration
AA
BB
CC
DD EE
FF GG
top module is tested with stubs
stubs are replaced one at a time, "depth first"
as new modules are integrated, some subset of tests is re-run
Software Engineering Software Testing Slide 29
Bottom-Up Integration
drivers are replaced one at atime, "depth first"
worker modules are grouped intobuilds and integrated
AA
BB
CC
DD EE
FF GG
clustercluster
Software Engineering Software Testing Slide 30
Sandwich Testing
Top modules aretested with stubs
Worker modules are grouped intobuilds and integrated
AA
BB
CC
DD EE
FF GG
cluster
Software Engineering Software Testing Slide 31
System testing• Involves integrating components to create a system or
sub-system.
• May involve testing an increment to be delivered to the customer.
• Two phases:– Integration testing - the test team have access to the system
source code. The system is tested as components are integrated.
– Release testing - the test team test the complete system to be delivered as a black-box.
Software Engineering Software Testing Slide 32
Object-Oriented Testing• begins by evaluating the correctness and consistency of the
OOA and OOD models
• testing strategy changes
– the concept of the ‘unit’ broadens due to encapsulation
– integration focuses on classes and their execution across a ‘thread’ or in the context of a usage scenario
– validation uses conventional black box methods
Software Engineering Software Testing Slide 33
OOT Strategy• class testing is the equivalent of unit testing
– operations within the class are tested– the state behavior of the class is examined
• integration applied three different strategies– thread-based testing—integrates the set of classes
required to respond to one input or event– use-based testing—integrates the set of classes
required to respond to one use case– cluster testing—integrates the set of classes
required to demonstrate one collaboration
Software Engineering Software Testing Slide 34
Incremental integration testing
T3
T2
T1
T4
T5
A
B
C
D
T2
T1
T3
T4
A
B
C
T1
T2
T3
A
B
Test sequence 1 Test sequence 2 Test sequence 3
Software Engineering Software Testing Slide 35
Testing approaches• Architectural validation
– Top-down integration testing is better at discovering errors in the system architecture.
• System demonstration– Top-down integration testing allows a limited
demonstration at an early stage in the development.
• Test implementation– Often easier with bottom-up integration testing.
• Test observation– Problems with both approaches. Extra code may be
required to observe tests.
Software Engineering Software Testing Slide 36
Black-box testing
IeInput test data
OeOutput test results
System
Inputs causinganomalousbehaviour
Outputs which revealthe presence ofdefects
Software Engineering Software Testing Slide 37
Testing guidelines• Testing guidelines are hints for the testing team to
help them choose tests that will reveal defects in the system– Choose inputs that force the system to generate all error
messages;
– Design inputs that cause buffers to overflow;
– Repeat the same input or input series several times;
– Force invalid outputs to be generated;
– Force computation results to be too large or too small.
Software Engineering Software Testing Slide 38
Use cases• Use cases can be a basis for deriving the tests
for a system. They help identify operations to be tested and help design the required test cases.
• From an associated sequence diagram, the inputs and outputs to be created for the tests can be identified.
Software Engineering Software Testing Slide 39
Test case design• Involves designing the test cases (inputs and
outputs) used to test the system.
• The goal of test case design is to create a set of tests that are effective in validation and defect testing.
• Design approaches:– Requirements-based testing;– Partition testing;– Structural testing.
Software Engineering Software Testing Slide 40
OOT—Test Case DesignBerard [BER93] proposes the following approach:
1. Each test case should be uniquely identified and should be explicitly associated with the class to be tested,
2. The purpose of the test should be stated,
3. A list of testing steps should be developed for each test and should contain [BER94]:
a list of specified states for the object that is to be tested
b. a list of messages and operations that will be exercised as a consequence of the test
c. a list of exceptions that may occur as the object is tested
d. a list of external conditions (i.e., changes in the environment external to the software that must exist in order to properly conduct the test)
e. supplementary information that will aid in understanding or implementing the test.
Software Engineering Software Testing Slide 41
Test Case Design
OBJECTIVE to uncover errors
CRITERIA in a complete manner
CONSTRAINT with a minimum of effort and time
Software Engineering Software Testing Slide 42
Requirements based testing• A general principle of requirements
engineering is that requirements should be testable.
• Requirements-based testing is a validation testing technique where you consider each requirement and derive a set of tests for that requirement.
Software Engineering Software Testing Slide 43
Software Testing
Methods
Strategies
white-boxmethods
black-box methods
Software Engineering Software Testing Slide 44
Black-Box Testing
requirementsrequirements
eventseventsinputinput
outputoutput
Software Engineering Software Testing Slide 45
Black-Box Testing• How is functional validity tested?• How is system behavior and performance tested?• What classes of input will make good test cases?• Is the system particularly sensitive to certain input
values?• How are the boundaries of a data class isolated?• What data rates and data volume can the system
tolerate?• What effect will specific combinations of data have
on system operation?
Software Engineering Software Testing Slide 46
Equivalence Partitioning
useruserqueriesqueries
mousemousepickspicks
outputoutputformatsformats
promptsprompts
FKFKinputinput
datadata
Software Engineering Software Testing Slide 47
Boundary Value Analysis
useruserqueriesqueries
mousemousepickspicks
outputoutputformatsformats
promptsprompts
FKFKinputinput
datadata
outputoutputdomaindomaininput domaininput domain
Software Engineering Software Testing Slide 48
Partition testing• Input data and output results often fall into
different classes where all members of a class are related.
• Each of these classes is an equivalence partition or domain where the program behaves in an equivalent way for each class member.
• Test cases should be chosen from each partition.
Software Engineering Software Testing Slide 49
Equivalence partitioning
System
Outputs
Invalid inputs Valid inputs
Software Engineering Software Testing Slide 50
Equivalence partitions
Between 10000 and 99999Less than 10000 More than 99999
999910000 50000
10000099999
Input values
Between 4 and 10Less than 4 More than 10
34 7
1110
Number of input values
Software Engineering Software Testing Slide 51
Search routine specificationprocedure Search (Key : ELEM ; T: SEQ of ELEM; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition-- the sequence has at least one elementT’FIRST <= T’LAST
Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)
or -- the element is not in the array( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
Software Engineering Software Testing Slide 52
Search routine - input partitions• Inputs which conform to the pre-conditions.
• Inputs where a pre-condition does not hold.
• Inputs where the key element is a member of the array.
• Inputs where the key element is not a member of the array.
Software Engineering Software Testing Slide 53
OOT Methods: Random Testing• Random testing
– identify operations applicable to a class– define constraints on their use– identify a minimum test sequence– generate a variety of random (but valid) test
sequences
Software Engineering Software Testing Slide 54
OOT Methods: Partition Testing• Partition Testing
– reduces the number of test cases required to test a class in much the same way as equivalence partitioning for conventional software
– state-based partitioning• categorize and test operations based on their ability to change the
state of a class
– attribute-based partitioning• categorize and test operations based on the attributes that they use
– category-based partitioning• categorize and test operations based on the generic function each
performs
Software Engineering Software Testing Slide 55
Sample Equivalence Classes• Valid dataValid data
– user supplied commandsuser supplied commands– responses to system promptsresponses to system prompts– file namesfile names• computational datacomputational data
• physical parametersphysical parameters• bounding valuesbounding values• initiation valuesinitiation values
• output data formattingoutput data formatting• responses to error messagesresponses to error messages• graphical data (e.g., mouse picks)graphical data (e.g., mouse picks)
• Invalid dataInvalid data– data outside bounds of the programdata outside bounds of the program– physically impossible dataphysically impossible data– proper value supplied in wrong placeproper value supplied in wrong place
Software Engineering Software Testing Slide 56
Testing guidelines (sequences)• Test software with sequences which have only
a single value.
• Use sequences of different sizes in different tests.
• Derive tests so that the first, middle and last elements of the sequence are accessed.
• Test with sequences of zero length.
Software Engineering Software Testing Slide 57
Search routine - input partitions
Software Engineering Software Testing Slide 58
Structural testing• Sometime called white-box testing.
• Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases.
• Objective is to exercise all program statements (not all path combinations).
Software Engineering Software Testing Slide 59
Structural testing
Componentcode
Testoutputs
Test data
DerivesTests
Software Engineering Software Testing Slide 60
Binary search - equiv. partitions• Pre-conditions satisfied, key element in array.• Pre-conditions satisfied, key element not in
array.• Pre-conditions unsatisfied, key element in array.• Pre-conditions unsatisfied, key element not in array.• Input array has a single value.• Input array has an even number of values.• Input array has an odd number of values.
Software Engineering Software Testing Slide 61
Binary search equiv. partitions
Mid-point
Elements < Mid Elements > Mid
Equivalence class boundaries
Software Engineering Software Testing Slide 62
Binary search - test cases
Software Engineering Software Testing Slide 63
White-Box Testing
... our goal is to ensure that all statements and conditions havebeen executed at least once ...
Software Engineering Software Testing Slide 64
Path testing• The objective of path testing is to ensure that
the set of test cases is such that each path through the program is executed at least once.
• The starting point for path testing is a program flow graph that shows nodes representing program decisions and arcs representing the flow of control.
• Statements with conditions are therefore nodes in the flow graph.
Software Engineering Software Testing Slide 65
Binary search flow graph
elemArray [mid] != key
elemArray [mid] > key elemArray [mid] < key
1
2
3
4
5
6
7
8
9
14 10
11
12 13
bottom > top while bottom <= topbottom > top
elemArray[mid] = key
Software Engineering Software Testing Slide 66
Independent paths• 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14
• 1, 2, 3, 4, 5, 14
• 1, 2, 3, 4, 5, 6, 7, 11, 12, 5, …
• 1, 2, 3, 4, 6, 7, 2, 11, 13, 5, …
• Test cases should be derived so that all of these paths are executed
• A dynamic program analyser may be used to check that paths have been executed
Software Engineering Software Testing Slide 67
Why Cover?• logic errors and incorrect assumptions are
inversely proportional to a path's execution probability
• we often believe that a path is not likely to be executed; in fact, reality is often counter intuitive
• typographical errors are random; it's likely that untested paths will contain some
Software Engineering Software Testing Slide 68
Basis Path Testing
First, we compute the cyclomatic complexity:
or
number of enclosed areas + 1
In this case, V(G) = 4
number of simple decisions + 1
Software Engineering Software Testing Slide 69
Cyclomatic Complexity
A number of industry studies have indicated that the higher V(G), the higher the probability or errors.
V(G)V(G)
modulesmodules
modules in this range are modules in this range are more error pronemore error prone
Software Engineering Software Testing Slide 70
Basis Path Testing
Next, we derive the Next, we derive the independent paths:independent paths:
Since V(G) = 4,Since V(G) = 4,there are four pathsthere are four paths
Path 1: 1,2,3,6,7,8Path 1: 1,2,3,6,7,8Path 2: 1,2,3,5,7,8Path 2: 1,2,3,5,7,8Path 3: 1,2,4,7,8Path 3: 1,2,4,7,8Path 4: 1,2,4,7,2,4,...7,8Path 4: 1,2,4,7,2,4,...7,8
Finally, we derive testFinally, we derive testcases to exercise these cases to exercise these paths.paths.
11
22
3344
55 66
77
88
Software Engineering Software Testing Slide 71
Basis Path Testing Notes
you don't need a flow chart, you don't need a flow chart, but the picture will help when but the picture will help when you trace program pathsyou trace program paths
count each simple logical test, count each simple logical test, compound tests count as 2 or compound tests count as 2 or moremore
basis path testing should be basis path testing should be applied to critical modulesapplied to critical modules
Software Engineering Software Testing Slide 72
Control Structure Testing• Condition testing — a test case design method
that exercises the logical conditions contained in a program module
• Data flow testing — selects test paths of a program according to the locations of definitions and uses of variables in the program
Software Engineering Software Testing Slide 73
Loop Testing
Nested Nested LoopsLoops
ConcatenatedConcatenated Loops Loops Unstructured Unstructured
LoopsLoops
Simple Simple looploop
Software Engineering Software Testing Slide 74
Loop Testing: Simple Loops
• Minimum conditions—Simple LoopsMinimum conditions—Simple Loops
• skip the loop entirely
• only one pass through the loop
• two passes through the loop
• m passes through the loop m < n
• (n-1), n, and (n+1) passes through the loop
where n is the maximum number of allowable passes
Software Engineering Software Testing Slide 75
Loop Testing: Nested LoopsNested LoopsNested Loops
Start at the innermost loop. Set all outer loops to their minimum iteration parameter values.
Test the min+1, typical, max-1 and max for the innermost loop, while holding the outer loops at their minimum values.
Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this step until the outermost loop has been tested.
Software Engineering Software Testing Slide 76
Loop Testing: Nested Loops (cont.)
Concatenated Loops
If the loops are independent of one another
then treat each as a simple loop
else* treat as nested loops
endif*
Software Engineering Software Testing Slide 77
Release testing• The process of testing a release of a system that will
be distributed to customers.
• Primary goal is to increase the supplier’s confidence that the system meets its requirements.
• Release testing is usually black-box or functional testing– Based on the system specification only;
– Testers do not have knowledge of the system implementation.
Software Engineering Software Testing Slide 78
Performance testing• Part of release testing may involve testing the
emergent properties of a system, such as performance and reliability.
• Performance tests usually involve planning a series of tests where the load is steadily increased until the system performance becomes unacceptable.
Software Engineering Software Testing Slide 79
Stress testing• Exercises the system beyond its maximum design
load. Stressing the system often causes defects to come to light.
• Stressing the system test failure behaviour.. Systems should not fail catastrophically. Stress testing checks for unacceptable loss of service or data.
• Stress testing is particularly relevant to distributed systems that can exhibit severe degradation as a network becomes overloaded.
Software Engineering Software Testing Slide 80
Test automation• Testing is an expensive process phase. Testing
workbenches provide a range of tools to reduce the time required and total testing costs.
• Systems such as Junit support the automatic execution of tests.
• Most testing workbenches are open systems because testing needs are organisation-specific.
• They are sometimes difficult to integrate with closed design and analysis workbenches.
Software Engineering Software Testing Slide 81
A testing workbench
Dynamicanalyser
Programbeing tested
Test resultsTest
predictions
Filecomparator
Executionreport
Simulator
Sourcecode
Testmanager
Test data Oracle
Test datagenerator
Specification
Reportgenerator
Test resultsreport
Software Engineering Software Testing Slide 82
Key points• Testing can show the presence of faults in a system; it
cannot prove there are no remaining faults.
• Component developers are responsible for component testing; system testing is the responsibility of a separate team.
• Integration testing is testing increments of the system; release testing involves testing a system to be released to a customer.
• Use experience and guidelines to design test cases in defect testing.
Software Engineering Software Testing Slide 83
Key points• Interface testing is designed to discover defects in the
interfaces of composite components.
• Equivalence partitioning is a way of discovering test cases - all cases in a partition should behave in the same way.
• Structural analysis relies on analysing a program and deriving tests from this analysis.
• Test automation reduces testing costs by supporting the test process with a range of software tools.