Upload
halcyong3285
View
599
Download
1
Embed Size (px)
Citation preview
Software Testing Techniques
2
Introduction• Many aspects to achieving software quality
– Formal reviews (of both the software process and the various stages of development), audits, documentation, etc.
– Unit testing– Integration testing– Verification
• Does the module meet the specifications
– Validation• Does the product meet the requirements
3
Introduction
• A Critical element of the Software Quality Assurance
• Represents a critical review of Specifications, Design and Coding
• Destructive rather than Constructive (try to break the system)
• Major objective is to find errors not to show the absence of errors (as distinct from Verification and Validation)
4
Objectives
• A good test case is one that has a high probability of finding an as-yet undiscovered error
• A Successful test is one that uncovers an as-yet undiscovered error
5
Principles• All tests should be traceable to customer
requirements• Tests should be planned long before testing begins• The Pareto principle applies to Testing
– Typically, 80% of the errors come from 20% of the modules
• Testing should begin ‘‘in the small’’ and progress towards “in the large”
• Exhaustive Testing is not possible, but, – if time permits, conduct multiple failure mode testing
• Test plans must receive independent review
6
Testability
• The ease with which a computer program can be tested .
7
Characteristics for Testability
• Operability– the better it works ,
the more efficiently it can be tested
• The system has few bugs• No bugs block the execution of tests• The product evolves in functional stages
8
Characteristics for Testability
• Observability– what you see is what you test
• Distinct output for each input• System states and variables visible during
execution• Past system states and variables are visible• All factors affecting the output are visible• Incorrect output is easily identified• Internal errors are automatically detected and
reported
9
Characteristics for Testability• Controllability
– the better we can control the software ,the more testing can be automated
• All possible outputs can be generated through some combination of input
• All code is executable through some combination of input
• Input and Output formats are consistent and structured
• All sequences of task interaction can be generated• Tests can be conveniently specified and reproduced
10
Characteristics for Testability
• Decomposability– By controlling the scope of testing , isolate
problems and perform smarter retesting
• The Software system is built from independent modules
• Software modules can be tested independently
– While this is very important, it does not obviate the need for integration testing
11
Characteristics for Testability
• Simplicity– the less there is to test ,
the more quickly we can test it
• Functional simplicity• Structural simplicity• Code simplicity
12
Characteristics for Testability
• Stability– the fewer the changes , the
fewer disruptions to testing• Changes are infrequent
• Changes are controlled
• Changes do not invalidate existing tests
• The software recovers well from failures
13
Characteristics for Testability
• Understandability– the more information we have ,
the smarter we will test
• The design is well understood• Dependencies between internal, external and
shared components well understood• Changes to design are well communicated• Technical documentation is instantly accessible,
well-organized, specific and accurate
14
Types of Testing
• White-Box Testing– Knowing the internal workings of a product,
tests are conducted to ensure that “all all gears mesh”gears mesh”
• Black-Box Testing– Knowing the specified function that a
product has been designed to perform , tests are conducted to demonstrate that each function is fully operational (note: this is still different from validation)
15
White Box Testing
• Uses the control structure of the procedural design to derive test cases
• Guarantees that all independent paths within a module have been exercised at least once
• Exercises all loops at their boundaries and within their operational bounds
• Exercises internal data structures to assure their validity - again, at their boundaries and with their operational bounds
16
Control Structure Testing
• Attacks the control flow of the program
• Provides us with a logical complexity measure of a procedural design
• Use this measure as a guide for defining a Basis set of execution paths
• Test cases derived to exercise the Basis set are guaranteed to execute every statement in the program at least once
17
Basis Path Testing• Basis path testing is a white-Box testing
technique first proposed by Tom McCabe.
• The basis path method enables the test case designer to derive a logical complexity measure of a procedural design use this measure as a guide for defining a basis set of execution paths.
• Test cases which derived to exercise, are guaranteed to execute every stmt in the program at least one time during testing.
18
Basis path testing(contd..)
• A Flow Graph notation(Simple notation for representationof control flow)
– represents the control flow of the program – each node in the graph represents one or
more procedural statements– Any procedural design representation can be
translated into a flow graph
19
Flow Graph Notation
20
Basis Path Testing ( contd.)• Flow Graph
1
2
4 3
6 5
7a
7b
8
21
Basis Path Testing ( contd.)
• Cyclomatic Complexity
– Quantitative measure of the complexity of a program
– is the number of independent paths in the basis set of a program
– Upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once
22
Basis Path Testing ( contd.)
• Cyclomatic Complexity calculation
V (G) = E -N+ 2 = P + 1 = No. of regions in the graph
where E = no. of edges, N = no. of nodes, and P = no. of predicate nodes
• For the previous example
– Independent paths
path 1 : 1 - 8path 2 : 1 - 2
- 3 - 7b - 1 - 8 path 3 : 1 - 2 - 4 - 6 - 7a - 7b - 1 - 8 path 4 : 1- 2 - 4 - 5 - 7a - 7b - 1 - 8
– Cyclomatic complexity = 11 - 9 + 2 = 3 + 1 = 4
23
Basis Path Testing ( contd.)
• Prepare test cases that will force execution of each independent path in the Basis Path set
• Each test case is executed and compared to expected results
24
Example
25
Example
26
Condition Testing
• Exercises all the logical conditions in a module
• Types of possible errors – Boolean variable error– Boolean Parenthesis error– Boolean Operator error– Arithmetic expression error
27
Types of Condition Testing
• Branch Testing – the TRUE and FALSE branches of the
condition and every simple condition in it are tested
• Domain Testing– for every Boolean expression of n
variables , all of 2n possible tests are required
28
Data Flow Testing• Assume functions do not modify their
arguments or global variables. Then define– DEF ( S ) = { X | Statement S contains a
definition of X }– USE ( S ) = { X | Statement S contains a
use of X }– Definition - Use chain ( DU chain )
• [ X , S , S ‘ ] , where X DEF ( S ) and X USE ( S ‘ ) and the definition of X in S is live at S ’
• Every DU chain to be covered at least once
29
Kinds of Loops
30
Loop Testing• Focus is on the validity of loop constructs• Simple loop ( n is the max. no. of allowable passes )
– Skip the loop entirely– Only one pass through the loop– Two passes– m passes , where m < n – n-1 , n , n+1 passes
• Nested loop– Start at innermost loop– Conduct simple loop test for this loop– Move outwards one loop at a time
31
Loop Testing ( contd.)
• Concatenated loops– Multiple simple loop tests if independent– Nested loop approach if dependent
• Unstructured loops– Should be restructured into a combination of
simple and nested loops
32
Black Box Testing• Focus is on the functional requirements of the
software• Uncovers errors such as
– Incorrect or missing functions– Interface errors– Errors in data structures – Performance errors– Initialization and Termination errors
• Unlike White Box Testing , this is performed at later stages of testing
33
• How is functional validity tested?• How is system behavior and performance tested?• What classes of input will make good test cases?• Is the system particularly sensitive to certain input
values?• How are the boundaries of a data class isolated?• What data rates and data volume can the system
tolerate?• What effect will specific combinations of data have on
system operation?
34
Graph Based Testing
• Identify all objects modeled by the software• Identify the relationships that connect these
objects• Create an Object-Relationship graph
– node– node weights– links– link weights
35
Graph Testing ( contd.)
• Example graph
newfile
Document window
Document text
is represented as containsAttributes :
start dimension Background color
text color
allowsediting
of
menu select generates
generation < 1 sec
36
Equivalence Partitioning• It is a Black box testing method that divides,
Input domain of a program into classes of data from which test cases are derived
• Goal is to design a single test case that uncovers classes of errors , thereby reducing the total number of test cases to be developed
• Each class represents a set of valid or invalid states for input conditions
37
Equivalence Partitioning ( contd.)
• Test case design is based on an evaluation of equivalence classes for an input condition– range specified , one valid and two invalid
equivalence classes– requires a specific value , one valid and two invalid
equivalence classes– specifies a member of a set , one valid and one
invalid equivalence classes– is boolean , one valid & one invalid equivalence
class
38
Equivalence Partitioning ( cont. )• Example
Automatic Banking– area code : input condition , boolean
input condition , range [200,999]– prefix : input condition , range >200, no 0’s, < 1000– suffix : input condition , value -- 4 digits– password : input condition , boolean
input condition , value -- 6 char str– command : input condition , set
39
Boundary Value Analysis• Greater number of errors tend to occur at the
boundaries of the input domain• Select test cases that exercise bounding values• Input condition
– range , test cases are just below min and just above max
– set , test cases are minimum and maximum values, if ordered
• The above guidelines are also applied to output conditions– example
• outputs that produce minimum and maximum values in the output range
40
Comparison Testing• Multiple copies of the software are constructed
in case of critical applications– Example: Shuttle Flight Control Software
• Each version is independently built from the specs by different teams
• Test cases derived from other BB Testing techniques are provided as input to each version
• Outputs are compared and versions validated• Not fool proof
41
Other Cases• GUI testing
– See text for partial list of things to test• Client Server
– Often distributed -- complicates testing– Additional emphasis on non-interference among
clients• Documentation
– Use black box techniques• Real-time
– Beyond the scope of this course
42
Summary• Destructive rather than constructive• Main goal is to find errors not to prove the
absence of errors• White Box Testing
– control structure testing– Condition testing– Data flow testing– Loop testing
• Black Box Testing - Functional requirements– Graph based testing – Equivalence partitioning– Boundary Value testing– Comparison testing
43
CLCS Example
Software Vendor Version Platform Facility
IRIX (UNIXoperating system)
Silicon GraphicsIncorporated (SGI)
6.2 SGI Indigo 2,SGI Indy,SGI Challenge
SDE,LCC-X
IRIX (UNIXoperating system)
Silicon GraphicsIncorporated (SGI)
6.3 SGI O2 SDE,LCC-X
VxWorks (GatewayOS)
VxWorks 5.2 SDS Gateway,CS Gateways
SDE,LCC-X
Table 1.1: Juno Baselined COTS Software
44
CLCS Example
data.
Step Description Expected Results1. Turn on SDE1 network hardware and PC’s Blinky lights start blinking on the
network devices, PC’s execute poweron self tests, boots OS
2. Turn on the sde1net workstation, wait for it to finishbooting (login screen will be displayed), then turn onthe sde1boot workstation and wait for it to finishbooting.
POST (Power On Self Test) testsoccur, Operating system startprocedures initiate, Login screensappear.
3. Turn on all remaining HCI workstations and thesde1ddp1 machine.
POST (Power On Self Test) testsoccur, Operating system startprocedures initiate, Login screensappear.
45
CLCS Example
16. Initiate data acquisition at the sde1hci7 workstation.In the Dnav master menu, select “Global Apps”, thenselect “Start receive process”, then select “GW toHCI JUNO_DDP_8”
The System messages windowindicates that the Start receiveprocess is executing, no unexpectederrors are displayed in the consolewindow.
17. Start data display. In the Dnav master menu, select“Shuttle”, then select any of the following:
Wind SpeedWind Direction PAD AWind Direction PAD BTemperature
The command is accepted (as shownin the System messages and consolewindows), the appropriate display(s)are started and are regularly updated.
18. Stop data display at the workstation. Select quit fromdisplay menu(s)
Display windows are closed.
46
Test Results
Number Title Opened
During
Criticality Date
Opened
Current
Status
Juno-11 Telnet from sde1hci1 to
sde1ddp-r failed
System
Test
Major 4/14/97 Open
Juno-12 Remote delog written
into wrong directory
System
Integration
Major 4/15/97 Open
Juno-13 Application displays
CPU intensive
System
Integration
Minor 4/15/97 Open
Juno-14 Telnet to SDS Gateway
not working
System
Test
Minor 4/22/97 Open
Juno-15 Error received when
attempting to start
receive process
System
Test
Major 4/22/97 Open
47
Lessons Learned• The configuration management process
was not complete, multiple baselines of some software components existed.
• A CLCS software sustaining process needs to be defined and implemented as soon as possible.
• Requirements buy-off process needs to be refined.
• Hardware identification could be improved.