Embed Size (px)
Citation preview
1
Test cases generation starting from Requirements
Filippo Ricca DISI, Università di Genova, Italy
2
White vs. Black box testing A white box testing is based upon
explicit knowledge of the SUT and its structure Called also structural testing Statement coverage testing is an example
The aim is to create enough testcases to ensure every statement is executed at least once
A black box testing approach will device testcases without any knowledge of the SUT or any aspect of its structure Called also functional testing Usually testcases are generated starting from
requirements/specifications
Inputs Outputs
Inputs Outputs
Exhaustive testing
3
By ``exhaustive" testing we mean testing the given program against every element in its input domain P: I-domain output
The large size of the input domain prevents a tester from exhaustively testing the program under test against all possible inputs ``exhaustive" testing = infeasible
Strategies to select “good inputs” are needed!
Test generation techniques
4
Just a look!
Equivalence partitioning
Test selection using equivalence partitioning allows a tester to subdivide the input domain into a small number of sub-domains
The equivalence classes are created assuming that the SUT exhibits the same behavior on all elements
The entire set of inputs can be divided into at least two subsets: expected, or legal inputs (E) and the unexpected, or illegal inputs (U).
Each of the two subsets, can be further subdivided into subsets
5
Input domains
Wordcount Example
Consider a wordCount method that: takes a word w and a filename f as
input and returns the number of occurrences of w in the text contained in the file named f
An exception is raised if there is no file with name f
6
begin
String w;
File f;
Input w, f
if (not exists(f))
{raise exception; return(0);}
If (length(w)==0)
return(0);
If (empty(f))
return(0);
return(getCount(w,f));
end
Input: xxy, [in xxy vi sono xxy tuple t.c.] Output: 2
file
Wordcount equivalence classes
7
Equivalence class w f
E1 non-null exists, not empty
E2 non-null does not exist
E3 non-null exists, empty
E4 null exists, not empty
E5 null does not exist
E6 null exists, empty
Null – not null not exist, exist-empty, exist-not empty
Junit example
8
public class WordCountTester extends TestCase {
public void testE1() { WordCount WC = new WordCount(); String w=“xxy”; File f; OS.fill(f); assertEquals(2, WC.wordCount(w, f)); }
public void testE2() { // file doesn't exist WordCount WC = new WordCount();
String w=“xxy”; File f; \\ OS.fill(f); try { WC.wordCount(w, f); fail(“method should launch the exception!!"); } catch(FileNotFound e){ assertTrue(true); // OK } }
[in xxy vi sono xxy tuple t.c.]
null
Equivalence partitioning and types
9
Eq. Classes Example
One class with values inside the range and two with values outside the range.
speed ∈[60..90] {50}, {75}, {92}
Eq. Classes Example
At least one containing all legal strings and one all illegal strings based on any constraints.
firstname: string
{{ε}, {Sue}, {Loooong Name}}
Eq. Classes Example
Each value in a separate class
autocolor:{red, blue, green}
{{red,} {blue}, {green}}
up:boolean {{true}, {false}}
Eq. Classes Example
One class containing all legal arrays, the empty array, and one containing a larger than expected array.
int [ ] aName: new int[3];
{[ ]}, {[-10, 20]}, {[-9, 0, 12, 15]}
enumeration
Unidimentional vs. Multidimentional partitioning
Unidimentional: one way to partition the input domain is to consider one input variable at a time. Thus each input variable leads to a partition of the input domain
Multidimensional: another way is to consider all the variables at the same time. This procedure creates one partition consisting of several equivalence classes Multidimensional partitioning leads to a large number of
equivalence difficult to manage manually. Many classes so created might be infeasible
10
Partitioning Example
Consider an application that requires two integer inputs x and y
Each of these inputs is expected to lie in the following ranges: 3≤x≤7 and 5≤y≤9
11
E1: x<3 E2: 3≤x≤7 E3: x>7 y ignored E4: y<5 E5: 5≤y≤9 E6: y>9 x ignored
E1: x<3, y<5
E2: x<3, 5≤y≤9 E3: x<3, y>9
E4: 3≤x≤7, y<5 E5: 3≤x≤7, 5≤y≤9 E6: 3≤x≤7, y>9 E7: x>7, y<5 E8: x>7, 5≤y≤9 E9: x>7, y>9
Multidimentional
Unidimentional
6 vs. 9!
Boundary value analysis (BVA)
Boundary value analysis is a test selection technique that targets faults in applications at the boundaries of equivalence classes Experience indicates that programmers make
mistakes in processing values at and near the boundaries of equivalence classes
12
X
98 100 998 1000
99 999 x x * * * * 99≤X≤999
BVA: Procedure
1) Partition the input domain
using unidimensional
partitioning
2) Identify the boundaries for each partition
3) Select test data such that each boundary value
occurs in at least one test input
13
14
BVA: FindPrice Example 1. Create equivalence classes
Assuming that an item code must be in the range 99...999 and quantity in the range 1..100
Equivalence classes for code: E1: Values less than 99 E2: Values in the range E3: Values greater than 999
Equivalence classes for qty: E4: Values less than 1 E5: Values in the range E6: Values greater than 100
Code = 127 Quantity = 3
Price 87 euro
15
BVA: FindPrice Example 2. Identify boundaries
• Equivalence classes and boundaries for findPrice • Boundaries are indicated with an x • Points near the boundary are marked *
E1 E2
E3
98 100 998 1000
99 999 x x * * * *
E4 E5
E6
0 2 99 101
1 100 x x * * * *
code
quantity
16
BVA: FindPrice Example 3. Construct test set
Test selection based on the boundary value analysis technique requires that tests must include, for each variable, values at and around the boundary. Consider the following test set:
T={ t1: (code=98, qty=0), t2: (code=99, qty=1), t3: (code=100, qty=2), t4: (code=998, qty=99), t5: (code=999, qty=100), t6: (code=1000, qty=101)
}
Illegal values of code and qty included
Predicates Predicates arise from requirements in a variety of
applications Example: A boiler is to be shut down when a or b is
true or the boiler is in degraded mode and the steam meter fails
17
1. The water level in the boiler is below X lbs. (a) 2. The water level in the boiler is above Y lbs. (b) 3. A water pump has failed. (c) 4. A pump monitor has failed. (d) 5. Steam meter has failed. (e)
Boiler in degraded mode when either is true.
Predicate E=a+b+(c+d)e
Goal of predicate Testing
The goal of predicate-based test generation is to generate tests from a predicate p that guarantee the detection of any error that belongs to a class of errors in the coding of p
18
SUT impelment E=a+b+(c+d)e
E=a+b+(c+d)e
T (t, f, t, f, f)
=?
Junit 4.x example
19
import org.junit.*; import static org.junit.Assert.*;
public class BoilerTest {
boiler Boiler = new boiler(); @Parameters public static int[][] data = new int[][] { {T, T, T, T, T, T}, {T, T, T, T, T, F}, ... {F, F, F, F, F, F}, };
@Parameter(0) public boolean expected; @Parameter(1) public boolean a; @Parameter(2) public boolean b; @Parameter(3) public boolean c; @Parameter(4) public boolean d; @Parameter(2) public boolean e;
@Test public void testBoiler() { assertEquals(expected, boiler.shutDown(a,b,c,d,e)); } ... }
truth table
Decision Tables Decision table is based on logical relationships
just as the truth table It is a tool that helps us look at the “complete”
combination of conditions It is often used in SE for different purposes:
to produce specifications for complex systems for documentation to diagnose software/hardware problems based upon
symptoms described Ex. technical support company (by phone)
for testing
Components of a Decision Table
C1
C2
C3
a1
a2
a3
a4
a5
T T T T F F F F
T T F F T T F F
T F T F T F T F
x x x x
x x
x x
x x x x
x x
conditions
actions
values of conditions
actions taken
R1 R2 R3 R4 R5 R6 R7 R8
rules
Read a Decision Table by columns of rules. R1 says when all conditions are T, then actions a1, a2, and a5 occur or should occur
Conditions in Decision Table
The conditions in the decision table may take on any number of values When it is boolean, then the decision table is just like a truth table
set of conditions
Conditions and actions in the decision table are interpreted in different way according to the context Testing
conditions are interpreted as the inputs, and the actions as outputs
to diagnose software/hardware problems conditions are interpreted as symptoms while actions are steps
to conduct to resolve the problem …
Printer troubleshooter
23
Printer troubleshooter Rules
Conditions
Printer does not print Y Y Y Y N N N N
A red light is flashing Y Y N N Y Y N N
Printer is unrecognised Y N Y N Y N Y N
Actions
Check the power cable X
Check the printer-computer cable X X
Ensure printer software is installed X X X X
Check/replace ink X X X X
Check for paper jam X X
A technical support company writes a decision table to diagnose printer problems based upon symptoms described to them over the phone from their clients
symptoms
problems
Triangle Problem Example Consider a program that, given the length of 3 sides,
determines whether the 3 sides can: (i) form a triangle and if yes (ii) what type of triangle
equilateral, isosceles, or scalene
The inputs are a, b, c sides (each between 1 and 200) We know that if the inputs satisfy the following conditions
then the 3 sides form a triangle: a < b + c b < a + c c < a + b
Triangle Problem Example
1. a < b + c 2. b < a + c 3. c < a + b
4. a = b 5. a = c 6. b = c
1. Not triangle
2. Scalene 3. Isosceles 4. Equilateral 5. “impossible”
F T T T T T T T T T T - F T T T T T T T T T - - F T T T T T T T T
- - - T T T T F F F F - - - T T F F T T F F - - - T F T F T F T F
X X X
X X X X X X X X
Note the Impossible cases
Pick input <a, b, c> for each of the columns
the “-” represents “don’t care”
R1 R11 R4
How Many Test Cases for Triangle Problem?
Three classes: “invalid situation” --- not a triangle:
There are 3 cases Note the “-” entries, which represents “don’t care,” when it is determined
that the input sides <a, b, c> do not form a triangle
“valid” triangle situation: There are 5 cases
impossible: There are 3 cases
So, for valid values of a, b, and c, we need to come up with 8 sets of <a, b, c> to test the “Rules”. R1 <3, 1, 1> - Not triangle R4 <3, 3, 3> - Equilateral …
Use cases (1)
Use cases are a (informal) way to define system software requirements
A use case describes a sequence of actions performed by a system (under development) to provide a result to a person
Ex. Login: 1. the system asks for the student ID and password 2. the student enters ID and password 3. the system shows the main page (3a) ID and password are not correct. Goto 1
27
Use cases (2)
Use cases tell: the customer what to expect; the developer what to code; the technical writer what to document, and; the tester what to test.
Developers can begin creating test cases as soon as use cases are available, before any code is written! This is one of the advantages of black box testing
28
Use case diagrams
29
Use case diagrams provide the big picture: Each use case represents a
chunk of functionality that will be implemented
Ovals represent use cases Stick figures represent
"actors” Humans or other systems
lines represent communication between an actor and a use case
Use cases are described textually by a template
University course registration system
System
Register For Courses Use case 1. Logon: the system asks for, and the Student enters, the student ID
and password 2. Select 'Create a Schedule’: the system displays the functions
available to the student. The student selects "Create a Schedule." 3. Obtain Course Information: The system retrieves a list of available
courses from the Course Catalog System and displays the list to the Student
4. Select Courses: the Student selects four primary courses and two alternate courses
5. Submit Schedule: The student indicates that the schedule is complete. The system verifies that the Student has the necessary prerequisites for each course
6. Display Completed Schedule The system displays the schedule and the confirmation number for the schedule 30
Flow of events
The most important part of a use case for generating test cases is the flow of events
The basic flow of events should cover what "normally" happens when the use case is performed
The alternate flows of events covers behavior of an optional or exceptional character Ex. errors
31
Basic flow
32
1. Logon 2. Select 'Create a Schedule’ 3. Obtain Course Information 4. Select Courses 5. Submit Schedule 6. Display Completed Schedule
Normal flow! The schedule was ok ....
Alternate Flows (Register for Courses)
1A. Unidentified Student: In Step 1 if student ID and password are not valid, an error message is displayed. The use case ends
2A. Quit: The Course Registration System allows the student to quit at any time during the use case. The partial schedule is saved in the system. The use case ends
3A. Unfulfilled Prerequisites, Course Full: In Step 5 of the Basic Flow, if the system determines that prerequisites for a selected course are not satisfied or that the course is full the system will not enroll the student in the course. A message is displayed. The use case return at Step 4, Select Courses
4A. Course Catalog System Unavailable In Step 3 if the system is down, a message is displayed and the use case ends.
5A. Course Registration Closed: If the registration for courses has been closed, a message is displayed, and the use case ends 33
Scenario A scenario is an instance of a use
case, or a complete "path" through the use case
Following the basic flow would be one scenario
Following the basic flow plus alternate flow 1A would be another.
The basic flow plus alternate flow 2A would be a third, and so on
34 scenarios will be used as the basis for creating test cases!
Scenarios
35
Scenario 1 Basic flow
Scenario 2 Basic flow Alt 1
Scenario 3 Basic flow Alt 1 Alt 2
Scenario 4 Basic flow Alt 3
Scenario 5 Basic flow Alt 3 Alt 1
Scenario 6 Basic flow Alt 3 Alt 1 Alt 2
Scenario 7 Basic flow Alt 4
Scenario 8 Basic flow Alt 3 Alt 4
All the scenarios
Three-step process
• For each use case, generate a full set scenarios
• For each scenario, identify at least one quasi-test case and the conditions that will make it "execute."
• For each test case, identify the data values with which to test.
Three-step process for generating test cases
from a use case:
36
Use Case
Scenario 1
quasi-test case 1
quasi-test case 2
TC1
TC2
TC3
Scenario 2 Scenario 3
Step One: Generate Scenarios Read the use-case textual description and identify each
combination of main and alternate flows and create a scenario matrix
37
Scenario name Starting flow alternate
1 succesful registration Basic
2 Unidentified student Basic A1
3 User quits Basic A2
4 Course catalog system unavailable
Basic A4
5 Registration closed Basic A5
6 Cannot enroll Basic A3
no nested alternate flows!
Step Two: Identify Quasi-Test Cases
Rule of thumb: at least one quasi-test case for each scenario, but there will probably be more … 3A. Unfulfilled Prerequisites, Course Full
Re-read the use-case and find the conditions or data elements required to execute the various scenarios. For the “Register for Course” use case, conditions
would be student ID, password, courses selected, prerequisites fulfilled etc.
To document the quasi-test cases, a matrix format is useful 38
Quasi-test case 1 Quasi-test case 2
Test Case Matrix format
39
• V indicates valid • I invalid • n/a means that it is not necessary to supply a data value
• each row in the matrix should have at least one I indicating an invalid condition being tested!
data elements/conditions
Step Three: Identify Data Values to Test
The final step is to substitute actual data values for the Is and Vs
A number of techniques can be used for identifying data values equivalence partitioning boundary value analysis predicate based test generation …
40
Test case Matrix with values
41
42
Marathon GUI testing Tool Marathon:
is a tool that helps writing System/GUI tests for Java/Swing applications
consists of a recorder, player, and an editor records tests script in Python
It is possible to use it without knowing Python …
allows to insert assertions in the script easily Using a specific GUI
Shows testing results using JUnit’s control bar Red/green
http://www.marathontesting.com/Marathon.html
43
Marathon example
A calculator application was developed
We want to write tests for the calculator Ex. add two values
Steps: 1. create a new Marathon
project 2. create a new empty test
(a test is a Python script file)
3. build the test (recording)
44
Test script
• click on the record button in the toolbar – the red circle
• enter “12 + 4 =”, the result is displayed (16)
• press control right-click in the text area, to open the contextual menu
• select Assert Text
• stop recording (the button with a red square), the script now looks like:
• save the script
ifwindow('CalculatorExample'):click('1')click('2')click('+')click('4')click('=')assert_p('TextField','Text','16')close()
45
Run the test
• Select the “JUnit” tab • Click the test view, you should
see your new created test • Click on the “Run all tests”
button, in the JUnit’s control bar
• and … • get the green bar …
Possible exercises at the exam Partitioning example
Ex. Wordcount
Boundary value analysis Ex. FindPrice
Decision table Ex. Triangle problem
Given a use case applying the three step process to derive some testcases Ex. Register for courses
46
47
References (used to prepare these slides)
Slides of the book “Foundations of software testing” by Aditya P. Mathur http://www.cs.purdue.edu/homes/apm/
foundationsBook/InstructorSlides.html Generating Test Cases From Use Cases
http://www.ibm.com/developerworks/rational/library/content/RationalEdge/jun01/GeneratingTestCasesFromUseCasesJune01.pdf
Testing “Multiple Conditions” with Decision Table Technique (Wkshop 2009) http://www.marathontesting.com/Marathon.html
Wikipedia
