HELSINKI UNIVERSITY OF TECHNOLOGY
T-76.5613 Software Testing and Quality Assurance
Lecture 5, 24.9.2007
Black-Box Testing Techniques
Juha ItkonenSoberIT
2Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Black-box test case design techniques
Basic techniquesEquivalence partitioning Boundary value analysis
Function testingTesting heuristicsState transition testingSpecification testingError guessing and Failure modesScenario testing
Soap opera testing
3Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Equivalence partitioning and boundary value analysis (i.e. domain testing)
To avoid unnecessary testing, partition (divide) the range of inputs into groups of equivalent tests. Then treat an input value from the equivalence class as representative of the full group.We treat two tests as equivalent if they are so similar to each other that it seems pointless to test both.Boundaries mark the point or zone of transition from one equivalence class to another. These are good members of equivalence classes to use because the program is more likely to fail at a boundary.
invalid valid invalid
0 1 100 101
4Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Why not test only the boundaries?
If you do boundaries only, you have covered all the partitions as well
technically correct and may be OK if everything works correctlyboundaries can be analyzed wrongly, there can be hidden boundaries
if the test fails, is the whole partition wrong, or is a boundary in the wrong place – have to test mid-partition anyway
testing only extremes may not give confidence for typical use scenarios (especially for business people or users)boundaries may be harder (more costly) to set up
5Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Exercise: Design test cases Use equivalence classes and boundaries
6Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Exercise: Design test cases Some sample test cases
All: just one slide, no slides, many slidesCurrent slide: first, last, middle, selected, not selectedSelection: no selection, one slide, many slides, dis-continuous selection, all slides, …Slides:
One, first, lastSemicolon separated, two, many, consecutive not consecutiveNormal ranges, open ranges (e.g. ‘-5’ or ‘4-’), all slides, multiple ranges, duplicate and intersecting rangesRanges that exceed the number of slides, etc. Invalid slide numbersEmpty, invalid strings
Slides per page: One, two, four, six, odd number, big even number
Different amount of slides in the presentation (one, odd, even) Order: Horizontal and vertical
Combinations with other settings?
7Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Function testing
Testing one function at a time“Functional unit testing”
Can be performed based on various informationGoing through the user interface or other interface commandsUser manual (draft)Functional specifications and requirements
Various techniques can be utilized for testing each function
Equivalence classes, boundary values, combinations, …
8Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Function testing tasks
Build a list of functionsIdentify each function and sub-function
Determine how to recognize correct behaviourIdentify variables, equivalence classes, and boundariesIdentify environmental variables
that may constrain the function under test
Check that the function does what it's supposed to do and does not do what it's not supposed to do
9Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Testing individual functions
Input variables of the functionOutputs of the functionPossible scope of the function
E.g. apply to selection, page, whole document
Options of the functionSettings, configuration, etc.
Circumstances under which the function behaves differently
Modes, or status of the systemDifferent ways of performing the same functionOther functions that interact with the tested function
10Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Benefits of function testing
Functional coverageCover every feature and optionFunctional specification or function list
Useful for initial testing of the productIntroduction to the capabilities of the productReveals serious problems that should be addressed early
11Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Shortcomings of function testing as a primary method
Misses interaction and relationships between functions
Feature interactionsmisses exploration of the benefits offered by the program
Is not necessarily powerful for validationWhether the software actually solves the customer’s problemWhether the users can actually achieve the benefitsEmphasizes isolated testing of individual itemsLoad, performance, reliability, etc. not covered
Easily focuses on the typical caseswithout considering boundaries, special cases or tests of special data
12Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Test Heuristics
A heuristicA rule of thumbA way of making an educated guessUseful rules that are not guaranteed to lead to the right result
Heuristics can be used to generate ides for good testsHeuristics are suggestions that must be used wisely
Do not blindly follow heuristics without understanding them
Experienced testers collect heuristics that help them make good guesses when designing tests
13Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Examples of general testing heuristics
Test at the boundariesTest exceptional casesTest with realistic data and scenariosTest every error messageAvoid redundant testsTest configurations that are different from the programmers’Run tests that are annoying to set up…
14Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
CICCUPP – Consistency heuristics help recognizing defects in software
Consistent with History: Present function behavior is consistent with past behavior.Consistent with our Image: Function behavior is consistent with an image that the organization wants to project. Consistent with Comparable Products: Function behavior is consistent with that of similar functions in comparable products.Consistent with Claims: Function behavior is consistent with what people say it’s supposed to be.Consistent with User’s Expectations: Function behavior is consistent with what we think users want.Consistent within Product: Function behavior is consistent with behavior of comparable functions or functional patterns within the product.Consistent with Purpose: Function behavior is consistent with apparent purpose.
Suorce: Black Box Software Testing, Cem Kaner
15Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
State transition testing
Uses a model that shows:states the software may occupytransitions between the states events which cause the transitionsactions that result from the transitions
Wait for card Wait for PINCancel
Return card
Card inserted
Ask for PIN
Invalid PIN
Ask for PIN
Valid PIN
Ask amount
16Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
State transition table
Current state Event Action Next state
Wait for card Card inserted Ask for PIN Wait for PIN
Wait for pin Cancel Return card Wait for card
Wait for pin Valid PIN Ask amount Wait for Amount
Wait for pin Invalid PIN Ask for PIN Wait for pin
Wait for amount … … …
… … … …
17Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
State transition testing coverage criteria
Simple criteriaCover all statesCover all eventsCover all transitionsCover all paths (infinite if loops possible)
Switch coverageAll single transitions (0-switch)Transition pairs (1-switch)Transition triples (2-switch)etc.
A more ”complete” test set will test for possible invalid transitions
use state transition table to identify invalid transitions
18Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
State transition table with invalid transitions
Current state Event Action Next state
Wait for card Card inserted Ask for PIN Wait for pin
Wait for card Cancel Wait for card
Wait for card Valid PIN Wait for card
Wait for card Invalid PIN Wait for card
Wait for card Valid amount Wait for card
Wait for card Sufficient balance Wait for card
Wait for card Insufficient balance Wait for card
Wait for pin Card inserted Wait for pin
Wait for pin Cancel Return card Wait for card
Wait for pin Valid PIN Ask amount Wait for amount
Wait for pin Invalid PIN Ask for PIN Wait for pin
Wait for pin Valid amount Wait for pin
Wait for pin Sufficient balance Wait for pin
Wait for pin Insufficient balance Wait for pin
Wait for amount Card inserted Wait for amount
Wait for amount Cancel Wait for amount
Wait for amount Valid PIN Wait for amount
Wait for amount Invalid PIN Wait for amount
Wait for amount Valid amount Get amount Check balance
Wait for amount Sufficient balance Wait for amount
Wait for amount Insufficient balance Wait for amount
Check balance Card inserted Check balance
Check balance Cancel Check balance
Check balance Valid PIN Check balance
Check balance Invalid PIN Check balance
Check balance Valid amount Check balance
Check balance Sufficient balance Give money Dispense notes
Check balance Insufficient balance Return card Wait for card
… … … …
19Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Specification testing
Testing based on some sort of specification document
Requirements specificationFunctional specificationUser manualEtc.
Discovering what claims are being made testing them against the product
20Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Specification testing examples
Using context-free questionsWho? What? When? Where? How? Why?
How does the spec say what it saysAmbiguity
Are multiple interpretations possible?
Adequacy Does it provide enough information for programming, documentation and testing?
CompletenessTo what extent does it cover the feature set, use cases, etc.?
Whenever there is ambiguity, there is a strong opportunity for a defect
21Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Error guessing and Failure model analysis
Always worth includingComplements systematic techniquesCan find some faults that systematic techniques can missIs based on experience and previous or typical defects
22Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Error guessing: deriving test cases
People are not that imaginative about committing bugsThere tends to be the same bugs as usuallyLearning from typical errors
ConsiderPast failuresIntuitionExperienceBrain storming
”What is the craziest thing we can do?”
Lists and failure models in literature, defect catalogs
Create a defect catalog of your own List the most typical defects in your context
23Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Test CatalogsTest catalogs are refined defect catalogsTest catalog is a check list of typical tests for a certain situationBased on experience on typical errors that developers make
Textual input/Numbersthe largest number that is supposed to be accepted the smallest number that is supposed to be accepted one larger than the largest number that is supposed to be accepted one smaller than the smallest number that is supposed to be accepted
M achine boundaries127128 (signed byte overflow) 255256 (unsigned byte overflow) 32767 (32 767) 32768 (32 768) (signed 16-bit overflow) 65535 (65 535) 65536 (65 536) (unsigned 16-bit overflow) 2147483647 (2 147 483 647) 2147483648 (2 147 483 648) (signed 32-bit word overflow) 4294967295 (4 294 967 295) 4294967296 (4 294 967 296) (unsigned 32-bit word overflow.)
24Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Example of Failure Model Testing James A. Whittaker: How to Break Software
User interface attacksApply inputs that force all error messagesApply inputs that force the software to establish default valuesOverflow input buffersForce data structure to store too many or too few valuesInvestigate alternative ways to modify internal data constraintsEtc.
System interface attacksFill the system to its capacityForce media to be busy or unavailableVary file permissionsVary file content or corrupt fileEtc.
25Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Scenario testing
A scenario is a hypothetical story, used to help a person think through a complex problem or system.Scenario Testing is a testing approach that is based on scenarios.Testing scenarios are complicated and realistic stories of real usage of the systemScenarios are carefully designed and documentedScenarios can be executed manually or automated
The goal is to focus on business needs and realistic use situations, instead of technical details and mechanical testing
26Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Good Testing Scenarios
Based on a story about how the program is usedmotivations of the people involved.
MotivatingA stakeholder with influence would push to fix a revealed defect
CredibleNot only could happen in the real world; stakeholders would believe that something like it probably will happen
ComplexComplex use of the program Complex environment Complex set of data.
Results are easy to evaluateEspecially important for scenarios because they are complex
Cem
Kaner. “An Introduction to Scenario Testing”, STQE, October, 2003.http://www.kaner.com/pdfs/ScenarioIntroVer4.pdf
27Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Scenarios can be used to
Learn the productConnect testing to documented requirementsBring requirements-related issues to the surfaceExpose failures to deliver desired benefitsExplore expert use of a programMake a bug report more motivating
Cem
Kaner. “An Introduction to Scenario Testing”, STQE, October, 2003.http://www.kaner.com/pdfs/ScenarioIntroVer4.pdf
28Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Soap Opera Testing
Extreme Scenario TestingBuild a scenario based on real-life experienceExaggerate each aspect of it
for each variable, substitute a more extreme valueif a scenario can include a repeating element, repeat it lots of timesMake environment more hostile
increase or decrease memory, printer resolution, video resolution, etc.
Make the situation as complicated as possible
Create a real-life story that combines all of the elements into a test case narrative.