Click here to load reader
Upload
nadina-adelia-indrawan
View
86
Download
0
Embed Size (px)
DESCRIPTION
blacbox
Citation preview
Black box testing
Readings: Ron Patton chap 5, Daniel Galin chap 9.5.1With material from: R. Binder, Testing ObjectOriented Systems
What is BlackBox testing ?
● Testing without having an insight into the details of underlying code
input outputUnit
● Testing based on specification
What is BlackBox testing ?
● Advantages● no need for source code● can be used for unit as well as systemlevel testing
●Disadvantages● don't test hidden functions
●Approaches:● Error guessing● Equivalence Partitioning● Boundary Value Analysis● Decision tables● CauseEffect Graphing● Tests derived from functional requirements (e.g. use cases)
Error Guessing (1)
● Adhoc approach based on experience● Approach:
1. Make list of possible errors or errorprone situations➔ error model
2. Design testcases to cover error model● Develop and maintain your own error models
Error Guessing (2)
● Example sorting array function● Error model:
– empty array– already sorted array– reversesorted array– large unsorted array– ...
● Generate test cases for these situations
Equivalence Partitioning
● Partition input domain in equivalence classes according to specification
● Ideally ECs should be – such that each input is a class – disjointed– elements of same class mapped to output similarly
● Difficult in practice to find perfect ECs– use of heuristics
Heuristics for Identifying Equivalence Classes
For each external input:
1. if specifies range of valid values, define ● 1 valid EC (within the range) ● 2 invalid EC (one outside each end of the range)
2. if specifies a number (N) of valid values, define ● 1 valid EC and ● 2 invalid ECs (none and more than N)
3. if specifies a set of valid values, define ● 1 valid EC (within set) and ● 1 invalid EC (outside set)
Heuristics for Identifying Equivalence Classes
For each external input:
4. if there is a reason to believe that program handle valid input differently, define
● 1 valid EC per valid input
5. if specifies a must be situation, define ● 1 valid EC (satisfy must) and ● 1 invalid EC (don't satisfy must)
6. if there is a reason to believe that elements in an EC are not handled in an identical manner by the program,
● subdivide EC into smaller Ecs.
Equivalence Class Partitioning
● Consider creating an equivalence partition that handle the default, empty, blank, null, zero, or none conditions.
Equivalence Classes Partitioning Triangle Example (1)
● Specification– input three integers (sides of a triangle: a, b, c)
● each side must be a positive number less or equal to 20.– output type of the triangle:
● Equilateral if a = b = c● Isosceles if 2 pairs of sides are equals● Scalene if no pair of sides is equal or ● NotATriangle if a >= b + c, b >= a + c, or c >= a + b
Equivalence Classes Partitioning Triangle Example (2)
According to heuristic #5
Input Condition Valid EC Invalid ECSides (a, b, c) All > 0 & <= 20 (1) a>20 (2)
b>20 (3)c>20 (4)
a <= 0 (5)b <= 0 (6)c <= 0 (7)
Equivalence Classes Partitioning Triangle Example (3)
EC #1 too broad can be subdivided (heuristic #6) Based on the treatment to data handling of data
1.1. a, b, c such that the triangle is equilateral1.2. a, b, c such that the triangle is isosceles1.3. a, b, c such that the triangle is scalene1.4. a, b, c such that it's not a triangle
Based on input1.5. a = b = c1.6. a = b, a <> c1.7. a = c, a <> b1.8. b = c, a <> b1.9. a <> b, a <> c, b <> c
Based on triangle property 1.10. a, b, c such that a >= b + c 1.11. a, b, c such that b >= a + c 1.12. a, b, c such that c >= a + b
Equivalence Classes Partitioning – Identifying test cases
1. Until all valid ECs have been covered by test cases, – write a new test case that cover as many of the
uncovered valid ECs as possible
2. Until all invalid ECs have been covered by test cases,
– write a test case that covers one, and only one of the uncovered invalid ECs
Equivalence Classes Partitioning Triangle Example (4)
● Test cases for triangle
EC a b c Expected Output1.1, 1.5 12 12 12 Equilateral1.2, 1.6 6 6 7 Isosceles1.3, 1.9 9 3 7 Scalene
1.4, 1.10 8 2 3 Not a triangle2 30 15 63 10 100 994 7 14 215 5 10 116 12 10 107 8 5 1
Equivalence Classes Partitioning Nextdate (1)
● Input: month, day, year representing a date– 1 <= month <= 12– 1 <= day <= 31– 1812 <= year <= 2012
● Output: date of the day after the input date– Must consider leap years– Leap year if divisible by 4, and not a century year– Century year is leap years if multiple of 400
Equivalence Classes Partitioning Nextdate (2)
● Equivalence classesInput Condition Valid Ecs Invalid Ecs
month 1 <= month <= 12 month < 1month > 12
day 1 <= day <= 31 day < 1day > 31
year 1812 <= year <= 2012 year < 1812year > 2012
●Refinement can be done based on how input is treated● Day incremented by 1 unless it's last day of a month
● Depend on month (30, 31, February)● At end of month next day is 1, month is incremented ● At end of year, both days and months are reset to 1, and year
incremented● Problem of leap year
Equivalence Classes Partitioning Nextdate (3)
● Decomposition of valid ECsInput Condition Valid Ecsmonth 30 days (1)
31 days (2)February (3)
day 1 <= day <= 28 (4)day = 29 (5)day = 30 (6)day = 31 (7)
year year = 1900 (8)1812 <= year <= 2012 andyear <> 1900 andyear mod 4 = 0 (9)1812 <= year <= 2012 andyear mod 4 <> 0 (10)
Better coverage obtained by selecting test cases according Cartesian product of ECs➔ 36 test cases.
Equivalence Classes Partitioning Nextdate (4)
● Test cases
ID Ecs month day year output1 1, 4, 8 6 14 1900 15/06/19002 1, 4, 9 6 14 1912 15/06/19123 1, 4, 10 6 14 1913 15/06/19134 1, 5, 8 6 29 1900 30/06/19005 1, 5, 9 6 29 1912 30/06/19126 1, 5, 10 6 29 1913 30/06/19137 1, 6, 8 6 30 1900 01/07/19008 1, 6, 9 6 30 1912 01/07/19129 1, 6, 10 6 30 1913 01/07/1913
10 1, 7, 8 6 31 1900 Error11 1, 7, 9 6 31 1912 Error12 1, 7, 10 6 31 1913 Error
Equivalence Classes Partitioning Problems
● Specification doesn't always define expected output for invalid testcases
● Strongly typed languages eliminate the need for the consideration of some invalid inputs
● Bruteforce of defining a test case for every combination of the inputs ECs – provides good coverage, but – impractical when number of inputs and associated classes is
large
Boundary Value Analysis
● Errors tend to occur near extreme values (boundaries)
● BVA enhances Equivalence Partitioning by– selecting elements just on, and just beyond the
borders of each EC– deriving test cases by considering ECs from output as
well
Boundary Conditions
● Situation at the edge of the planned operational limits of the software
● Data types with boundary conditions
– Numeric, Character, Position, Quantity, Speed, Location, Size● Boundary condition characteristics
– First/Last, Start/Finish, Min/Max, Over/Under, Empty/Full, Slowest/Fastest, Largest/Smallest, NextTo/FarthestFrom, Shortest/Longest, Soonest/Latest, Highest/Lowest
Boundary Value Analysis Example
● Examples of boundary values for nextdate
Input Boundary Valuemonth 0, 1, 12, 13day 0, 1, 30, 31, 32year 1811, 1812, 2012, 2013
Boundary Value Analysis Guidelines
● For each boundary condition– include boundary value in at least one valid test case– include value just beyond boundary in at least one invalid test
case
● Use same guidelines for each output condition– include test cases with input such that output at boundaries are
produced
Boundary Value Analysis
● SubBoundary Conditions
– internal to software (not defined in specification)– examples:
● Powersoftwo: bit, nibble, byte, word, kilo, mega, giga, tera● ASCII table
Boundary Value Analysis Character ASCII Code
/ 47 lower bound 0 48
1 492 503 514 525 536 547 558 56
upper bound 9 57: 58A 65a 97
If numbers are expected (conversion from ascii), '/' and ':' are possible boundary values
Decision Table
● Ideal for situations where:– combinations of actions taken under varying set of conditions– conditions depends on input variables– response produced doesn't depend on the order in which input
variables are set or evaluated, and– response produced doesn't depend on prior input or output
Decision Table Format
Conditions Combination of conditions (variants)
Actions Selected actions
Decision Table Development
1. Identify decision variables and conditions
2. Identify resultant actions to be selected or controlled
3. Identify which action should be produced in response to particular combinations of actions
Example
● Suppose the following insurance renewal rules – 0 claims, age<=25: raise by $50– 0 claims, age>25: raise by $25– 1 claim, age<=25: raise by $100, send letter– 1 claim, age>25: raise by $50– 2, 3 or 4 claims, age<=25: raise by $400, send letter– 2, 3 or 4 claims, age>25: raise by $200, send letter– more than 5 reclamations: cancel policy
Decision Table – Triangle Example● Actions
– finding it is not a triangle
– finding it is a scalene triangle
– finding it is an isosceles triangle
– finding it is an equilateral triangle
– finding it is an error
– finding variant impossible
● Decision variables: sides a, b, c
● Conditions
– a < b+c
– b < a+c
– c < a+b
– a=b
– a=c
– b=c
– a > 20
– b > 20
– c > 20
Decision Table – Triangle Example1 2 3 4 5 6 7 8 9 10 11 12 13 14
c1: a<b+c N Y Y Y Y Y Y Y Y Y Y c2: b<a+c N Y Y Y Y Y Y Y Y Y c3: c<a+b N Y Y Y Y Y Y Y Y c4: a=b Y Y Y Y N N N N c5: a=c Y Y N N Y Y N N c6: b=c Y N Y N Y N Y N c7: a>20 N N N N N N N N N N N Y c8: b>20 N N N N N N N N N N N Y c9: c>20 N N N N N N N N N N N Ya1: not a triangle X X Xa2: Scalene Xa3: Isosceles X X Xa4: Equilateral Xa5: impossible X X Xa6: error X X X
● Attention for inconsistencies
Decision Table – Don't Care condition
● Don't Care condition– May be true or false without changing the action– Simplifies the decision table– Corresponds to different implementation cases:
● inputs are necessary but have no effect for the variant● inputs may be omitted but have no effect if supplied● TypeSafe exclusion
● several conditions are defined for a non binary decision variable, which can hold only one value at a time
Decision Table – Can't Happen & Don't know conditions
Attention not to confuse a Don't Care condition with– Can't Happen Condition reflects assumption that
● some inputs are mutually exclusive,● some inputs can't be produced by the environment, or● implementation is structured so as to prevent evaluation
– Don't Know Condition – reflects an incomplete model● Usually indication of misspecification
– tests needed to exercise these undefined cases
Decision Table – Test generation strategies
● AllExplicit variants
– a test case for each explicit variant● AllVariants
– a test case for each implicit variant– variants for n conditions
● AllTrue
– a test case for each variant with an outcome (action)● AllFalse
– a test case for each variant without outcome
2n
Decision Table – Test generation Triangle example
● AllExplicit VariantsVariant a b b Expected output
1 4 1 2 Not a Triangle2 1 4 2 Not a Triangle3 1 2 4 Not a Triangle4 5 5 5 Equilateral5 Impossible6 Impossible7 2 2 3 Isosceles8 Impossible9 2 3 2 Isosceles10 3 2 2 Isosceles11 3 4 5 Scalene12 21 6 12 Error13 6 45 17 Error
Decision Table – Test generation strategies
● EachCondition/AllConditions
– Heuristic to reduce number of variants● Set of variants includes
– For each variable, a variant such that the variable is made true with all other variables being false, and
● One variant such that all variables are true (and logic), or ● One variant such that all variables are false (or logic)
Decision Table – EachCondition/AllConditions
S = P or Q or R
P Q R S (action)F F T xF T F xT F F xF F F
Decision Table – EachCondition/AllConditions
S = P and Q and R
P Q R S (action)F F TF T FT F FT T T x
Decision Table – EachCondition/AllConditions
Z = (A and B and (not C)) or (A and D)
A B C D Z (action)T F T -F T T -F F F -T T F - xT - - FF - - TT - - T x
Develop variants for (A and B and (not C)) and variants for (A and D)
● Values must be assigned to Don't Care variables
● randomly or● by suspicion
Decision Table – EachCondition/AllConditions
Z = (A and B and (not C)) or (A and D)
A B C D Z (action)T F T TF T T FF F F FT T F F xT T F FF T T TT T T T x
● Example of variants used for test suite generation
Decision Table – EachCondition/AllConditions
● Provides a compact table● number of tests increase linearly with number of
product terms● Assumes independence of condition evaluation,
and absence of default actions
Decision Table – Binary Decision Diagram Determinants
Binary Decision Diagram (BDD) – compact representation of a truth table
1. Construct BDD from the truth table
a)Create a decision tree from the truth table• Nodes represent Boolean variables• Left branch is always false• Right branch is always true• Each leaf represents resultant value for condition on
the path from root to leaf
Decision Table – Binary Decision Diagram Determinants
Example: Z = A and (B or C)
A
B
A
B
C C C C
0 0 0 0 0 1 1 1
Decision Tree
Decision Table – Binary Decision Diagram Determinants
b) Reduce decision tree to a BDD• From left to right, replace leaf nodes with equivalent
constants or variables and prune the branchesA
B
A
B
C C C C
0 0 0 0 0 1 1 1
Decision Table – Binary Decision Diagram Determinants
AA
B
C C
0 1 1 1
0
Decision Table – Binary Decision Diagram Determinants
AA
B
C
1 1
0
C
Decision Table – Binary Decision Diagram Determinants
AA
B
1
0
C
variant 1
variant 2 variant 3
Decision Table – Binary Decision Diagram Determinants
2. Map BDD into BDD determinant table• BDD determinant – root to leaf path in BDD
BDD Variant A B C Z1 F - - F2 T F C C3 T T - T
Decision Table – Binary Decision Diagram Determinants
3. Generate BDD test suite
BDD Variant A B C Z1 F - -2 T F F2 T F T x3 T T - x
● Values must be assigned to Don't Care variables
● randomly or● by suspicion
Decision Table – Variable Negation
● Strategy designed to reveal faults resulting from Don't Care implementation
– BDD determinant strategy does not explicitly address Don't Care variables
● Specification must be given as sumofproducts Boolean formula ((a11 and a12 ... and a1k) or (a21 and a22 ... and a2l) or ...)
● Produces a test suite with:
– one variant for each product term such that no other product term is true (unique true points)
– one variant for each term that results when each literal in each product term is negated such that the function evaluates to false (near false points)
Decision Table – Variable Negation
● Example: Z = A and (B or C) = (A and B) or (A and C)
1. Unique true points– (A and B) true, (A and C) false {A=T, B=T, C=F}
– (A and B) false, (A and C) true {A=T, B=F, C=T}
Decision Table – Variable Negation
● Example: Z = A and (B or C) = (A and B) or (A and C)
2. Near false points– (A and B)
● (~A and B) {A=F, B=T, C=T}● (A and ~B) {A=T, B=F, C=F}
– (A and C) ● (~A and C) {A=F, B=F, C=T}● (A and ~C) {A=T, B=F, C=F}
Causeeffect graphing
● Systematic way to aid selecting a highyield set of test cases– to identify/analyze relationships in a decision table– generate boolean formula
● Nodes:– Cause distinct instance of input condition or EC– Effect observable result of change in system state– Intermediate node combination of causes
representing an expression on input conditions
Causeeffect graphing layout
Causes
........
Intermediate nodes Effects
Valid links from● cause to intermediate node● cause to effects● intermediate node to intermediate node● intermediate node to effect
Causeeffect graphing – type of links
^
direct cause
AND
v OR
~ NOT
Causeeffect graphing example
A file update depends on the value in two fields. The value in field 1 must be an ``A'' or ``B''. The value in field 2 must be a digit. In this situation the file update is made. If the value in field 1 is incorrect, error message X12 is issued. If the value in field 2 is incorrect, error message X13 is issued.
Causeeffect graphing example
● Causes
– c1: character in column 1 is ''A''– c2: character in column 1 is ''B''– c3: character in column 2 is a digit
● Effects
– e1: update made– e2: message X12 issued– e3: message X13 issued
Causeeffect graphing – constraints between causes
c1
c2E
c1
c2I
c1
c2R
c1
c2O
Exclusive – c1 and c2 can not be true simultaneously
Exclusive – one of c1 and c2 must be true (can't be both false)
Require – c2 must be true for c1 to be true
One and only One – one and only one of c1 and c2 can be true at a given time
Causeeffect graphing – constraints between effects
e1
e2Mask – e1 forced to false if e2 M
Test generation from CauseEffect graphs (1)
● Divide the specification into workable pieces.● Define separate graphs for each piece.
1. Identify Causes and Effects
2. Deduce Logical Relationships and Constraints – represent as graph
3. Draw a decision table
4. Convert each variant in the decision table into a test case.
Test generation from CauseEffect graphs (2)
● To find the unique combination of conditions● Work with a single effect (output) at a time
1. Set effect to true (1) state
2. Look backward for all combination of inputs which will force effect to the true state (constraints limit number of combination)
3. Create a column in decision table for each combination
4. Determine states of all other effects for each combination
Boolean Formula Generation from CauseEffect graphs
● Working from effects toward causes1. Transcribe nodetonode formulas from the graph
– write formula for each effect and its predecessors– For each intermediate node
● write formula for intermediate node and its predecessors
2. Derive the complete Boolean formula
– replace intermediate variables by substitution until the effect formula contains only cause variables
– factor and rewrite formula into sumofproducts form
Boolean Formula Generation from CauseEffect graphs
A
B
C
D
E
F
G
H
I
~
~
^v
v
^
•I = H ∨ G•H=F C∨•F=~A B∧•G=D ~E∧
I = (~A B) C (D ~E∧ ∨ ∨ ∧ )
Test Cases derivation from requirements
● Involve clarification and restatement of the requirements● such that requirements are testable
– Obtain a point form formulation ● enumerate single requirements● group related requirements
– For each requirement● One test case showing requirement functioning● One test case trying to falsify
– e.g. but trying something forbidden● Test boundaries and constraints when possible
Test Cases derivation from requirements
● Example: Requirements for a Video Rental system● 1. The system shall allow hiring and returning of films
– 1.1. If a film is available for hire then it may be lent to a customer.
● 1.1.1 A film is available for hire until all copies have been simultaneously borrowed.
– 1.2. If a film was unavailable for hire, then returning the film makes it available.
– 1.3. The return date is established when the film is lent and must be shown when the film is returned.
– 1.4. It must be possible for an inquiry on a hired film to reveal the current borrower.
– 1.5. An inquiry on a member will reveal any films they currently have on hire.
Test Cases derivation from requirements
● Test situations for requirement 1.1● Attempt to borrow an available film. ● Attempt to borrow an unavailable film.
● Test situations for requirement 1.1.1
● Attempt to borrow a film for which there are multiple copies, all of which have been hired.
● Attempt to borrow a film for which all copies but one have been hired.
Test Cases derivation from requirements
● Test situations for requirement 1.2● Borrow an unavailable film.
● Return a film and borrow it
● Test situations for requirement 1.3● Borrow a film, return it and check dates● Check date on a nonreturned film
Test Cases derivation from requirements
● Test situations for requirement 1.4 ● Inquiry on hired film.
● Inquiry on returned film.
● Inquiry on a film which has been just returned.
● Test situations for requirement 1.5 ● Inquiry on member with no films.
● Inquiry on member with 1 film.
● Inquiry on member with multiple films.
Test Cases derivation fromUse Cases
● For all use cases:
1. Develop a Graph of scenarios
2. Determine all possible scenarios
3. Analyze and rank scenarios
4. Generate test cases from scenarios to meet a coverage goal
5. Execute test cases
Scenario GraphGenerated from a use case
● Nodes correspond to point where system waits for an event– environment event, system reaction
● There is a single starting node
● End of use case is finish node
● Edges correspond to events occurrence
– May include conditions
– Special looping edge
● Scenario: – path from starting node to a finish node
Scenario GraphTitle: User loginActors: User Precondition: System is ON1: User inserts a Card2: System asks for Personal Identification Number (PIN)3: User types PIN4: System validates USER identification5: System displays a welcome message to USER6: System ejects CardAlternatives: 1a: Card is not regular 1a1: System emits alarm1a2: System ejects Card4a: User identification is invalid AND number of attempts < 4 4a.1 Go back to Step 24b: User identification is invalid AND number of attempts >= 44b.1: System emits alarm4b.2: System ejects CardPostcondition: User is logged in
Scenarios
● Paths from start to finish
● Number of time loops can be taken need to be restricted for finite number of scenarios
Id Events Description1 1 2 3 4 5 – 6 User login with regular card,
right PIN on first trial. Normal course of events
2 1 1a.1 – 1a.2 User login with non regular Card3 1 2 3 4 2 – 3 User login with regular card,
4 5 – 6 wrong PIN on first trial,the right PIN on second trial
4 1 2 3 4 2* 3 User login with regular card, 4 4b.1 – 4b.2 wrong PIN on first, second,
third and fourth trials
Scenario Ranking
● If too many scenarios to test all● Ranking may be based on critically and
frequency● Always include main scenario
– should be tested first
Test Cases generation
● According to coverage goal. E.g.– All branches in graph of scénarios (minimal coverage
goal)– All Scenarios– n most critical scenarios
Example of Test Case
Test Case: TC1
Goal: Test the main course of events for the ATM system.
Scenario Reference: 1
Setup: Create a Card #2411 with PIN #5555 as valid user identification, System is ON
Course of test case
Pass criteria: User is logged in
External Event Reaction Comment1 User insert Card #2411 System asks for Personal
Identification Number (PIN)2 User types PIN #5555 System validates USER identification
System displays a welcome message to USER