3/12/2013
1
To Infinity & Beyond! Using Combinatorial Testing to Handle Complexity
Presented By:Andy Tinkham
Magenic Technologies
Testing Everything is Impossible
The number of test cases might as well be infinite
Interacting variables have a multiplicative effect
3/12/2013
2
Consider this screen…
How many test cases are needed to ensure that this all works
correctly?
What about when additional settings
are added to the poll creation process?
What about all the rest of the
functionality of this app?
Additional complexity
• MyVote runs on iOS, Windows 8, and Windows Phone 8 (and multiple devices for those systems!)
• Users can log in with Twitter, Facebook, and Microsoft ID
• Testing every combination of these is already over 18000 test cases
3/12/2013
3
We need ways to manage this
Many ways we could handle this
Give upTest everything (delaying release while we run all 18000+ test cases)
Choose a subset of tests to run & hope we chose well
Automate the tests
Combine tests to get a minimal test set that covers the interactions
One way…
• Combinatorial testing
• Simply multiplying out the test cases results in a lot of duplication
• Many bugs require interactions of variables
• Solution: Create set of tests to cover all pairs of variables interacting
3/12/2013
4
When to use combinatorial tests
• You’ve got a set of interacting variables that result in a large number of test cases
• You understand these variables well
• The variables are independent or you can explicitly define the dependencies
Pitfalls
• Choosing the wrong variables or values can leave gaps in your testing
• Relying solely on combinatorial testing may leave important combinations untested
• In at least 1 academic study, combinatorial testing didn’t do much better than random testing
3/12/2013
5
Variables
• What might vary in a test case
• Focus on one piece of functionality not whole system (use multiple models)
• Use domain analysis skills
• Look for anything that has an impact on the results for the functionality you’re testing
• May already have identified many as part of previous test design efforts
Identifying variables
Input values
User Input
Values from other
components
Function parameters
Data structures
Branch points
Environment
Configuration
Presence or absence of
Application settings
User settings
Global configuration
Data conditions
User role
Data state
3/12/2013
6
Identifying values
• Analyze variable list one by one
• Identify default values
• Identify equivalence classes for possible values
• Name equivalence classes rather than choosing representative values for now
3/12/2013
7
Example List for Add a Poll
• Question: Present, Not Present
• NumberOfAnswers: 0, 1, 2, 3, 4, 5
• CanSelectMultipleAnswers: Yes, No
• Image: None, JPG, GIF, PNG
• StartTime: None, Past, Present, Future
• EndTime: None, Past, Present, Future
Making the set
• List each value of the first variable
– Question: Present, Not Present
• Then add the second variables values so that each pair of values occurs
– NumberOfAnswers: 0, 1, 2, 3, 4, 5
3/12/2013
8
Question NumberOfAnswers
CanSelectMultipleAnswers
Image StartTime EndTime
Present
Present
Present
Present
Present
Present
Not Pres
Not Pres
Not Pres
Not Pres
Not Pres
Not Pres
Question NumberOfAnswers
CanSelectMultipleAnswers
Image StartTime EndTime
Present 0
Present 1
Present 2
Present 3
Present 4
Present 5
Not Pres 0
Not Pres 1
Not Pres 2
Not Pres 3
Not Pres 4
Not Pres 5
3/12/2013
9
Continuing the set
• Repeat for the remaining variables, keeping pair relationships up as you go
• When you can’t get a needed pair, first look for partial rows and set unknown values if possible
• If no partial row where you can set the needed value, add a new row
– Mark currently non‐needed values as unknown
Question NumberOfAnswers
CanSelectMultipleAnswers
Image StartTime EndTime
Present 0 Yes None None None
Present 1 No GIF Past Past
Present 2 Yes JPG Present Present
Present 3 No PNG Future Future
Present 4 Yes … … …
Present 5 No
Not Pres 0 No
Not Pres 1 Yes
Not Pres 2 No
Not Pres 3 Yes
Not Pres 4 No
Not Pres 5 Yes
3/12/2013
10
Creating the set
For a detailed description of the manual process, check out the Addendum to Chapter 3 in Kaner, Bach & Pettichord’s Lessons Learned in
Software Testing
Automating the process
• Luckily, we don’t have to do this by hand!
• Many tools available to create the test set for us– Bach’s AllPairs tool (satisfice.com)
– Hexawise
– Microsoft’s PICT
• Let’s look at PICT
3/12/2013
11
PICT
• Free tool, developed at Microsoft
• Windows based– Other platforms have similar tools, see http://pairwise.org for more info
• Command‐line based
• Reads in text files containing variables, values & context then outputs test cases
• See References slide for download & documentation links
Setting up variables in PICT
• Create a text file
• List each variable on a separate line, followed by a :
• List the values after the :, separated by commas
3/12/2013
12
Example
Question: Present, Not Present
Number Of Answers: 0, 1, 2, 3, 4, 5
Can Select Multiple Answers: Yes, No
Image: None, GIF, JPG, PNG
StartTime: None, Past, Present, Future
EndTime: None, Past, Present, Future
Output
Question Number Of Answers
Can Select Multiple Answers
Image Start Time End Time
Not Present 1 Yes PNG None Present
Not Present 5 No None Past None
Present 5 Yes GIF Present Future
Present 5 No JPG Future Past
Not Present 2 No JPG None Future
Not Present 3 Yes GIF Past Past
Present 0 No GIF None None
Present 4 No GIF Future Present
Not Present 0 Yes None Future Future
Present 1 No PNG Past Future
… (16 more) … … … … …
3/12/2013
13
Adding constraints
• Can’t have one time stamp be None while the other is set
• Don’t edit generated files – you might lose coverage of all the pairs
• Instead, add constraints to PICT model
3/12/2013
14
Example Model File
Question: Present, Not PresentNumber Of Answers: 0, 1, 2, 3, 4, 5Can Select Multiple Answers: Yes, NoImage: None, GIF, JPG, PNGStartTime: None, Past, Present, FutureEndTime: None, Past, Present, Future
IF [StartTime] = "None" THEN [EndTime] = "None";IF [EndTime] = "None" THEN [StartTime] = "None";
Output
Question Number Of Answers
Can Select Multiple Answers
Image Start Time End Time
Not Present 4 Yes PNG Past Past
Present 5 No JPG Present Future
Not Present 0 Yes None Future Future
Not Present 4 No GIF None None
Present 2 Yes None Present Present
Present 5 No PNG Future Present
Not Present 4 Yes JPG Past Present
Present 3 Yes None None None
Present 0 No GIF Past Past
Not Present 3 No JPG Future Past
… (19 more) … … … … …
3/12/2013
15
Positive & negative values
• Not all the test cases generated are expected to pass due to invalid values
– Question Not Present
– Answers 0 & 1
• We may not want multiple invalid values in a test case
• Solution: Tell PICT which values are invalid
Example Model
Question: Present, ~Not PresentNumber Of Answers: ~0, ~1, 2, 3, 4, 5Can Select Multiple Answers: Yes, NoImage: None, GIF, JPG, PNGStartTime: None, Past, Present, FutureEndTime: None, Past, Present, Future
IF [StartTime] = "None" THEN [EndTime] = "None";IF [EndTime] = "None" THEN [StartTime] = "None";
3/12/2013
16
Output
Question Number Of Answers
Can Select Multiple Answers
Image Start Time End Time
… (13 more) … … … … …
Present 3 Yes JPG Past Future
Present 5 Yes GIF None None
Present 4 No None None None
Present 4 No JPG Present Future
Present 4 No JPG Future Present
~Not Present 4 Yes PNG Past Past
Present ~0 Yes None Future Future
~Not Present 2 No JPG Past Present
~Not Present 3 Yes GIF None None
… (14 more) … … … … …
Weighting values
• Not every value occurs in the real world with the same frequency
• Once pairs are satisfied, PICT still has some “don’t care” values in table
• “Don’t care” values are filled in with random selections from the variable
• We can weight the more likely variables so PICT chooses them more often
3/12/2013
17
Example Model
Question: Present, ~Not PresentNumber Of Answers: ~0, ~1, 2, 3, 4 (10), 5Can Select Multiple Answers: Yes, No (10)Image: None, GIF, JPG (10), PNGStartTime: None, Past, Present, FutureEndTime: None, Past, Present, Future
IF [StartTime] = "None" THEN [EndTime] = "None";IF [EndTime] = "None" THEN [StartTime] = "None";
Seeding Test Cases
• There may be some test combinations that we want to ensure are included– Default options– Most common configuration
• If more than 2 variables involved in desired combination, PICT may not automatically generate that test
• We can give the test to PICT as a starting point
3/12/2013
18
Creating seed file
• Make a separate file• Format is same as output from PICT
– Lines tab‐delimited– First line has variable names in order– Remaining lines have desired test case values
• Seeded test cases can specify all values for test or just some (PICT will populate the remaining values
• File name is passed in to PICT command line call with /e: option
Example Seed File
Question Number Of Answers
Can Select Multiple Answers
Image Start Time End Time
Not Present 4 Yes PNG Past Past
Present No GIF
3/12/2013
19
Generating tests
• Install PICT
• Add directory with PICT.exe to PATH environment variable
• From a Command Prompt, run
pict.exe model_file.txt [options] > output_file.txt
PICT Options
• /o:N ‐ Order of combinations (default: 2)
• /d:C ‐ Separator for values (default: ,)
• /a:C ‐ Separator for aliases (default: |)
• /n:C ‐ Negative value prefix (default: ~)
• /e:file ‐ File with seeding rows
• /r[:N] ‐ Randomize generation, N ‐ seed
• /c ‐ Case‐sensitive model evaluation
• /s ‐ Show model statistics
3/12/2013
20
Test Automation with Combinatorics
• Generated test cases can be fed in to custom‐built parser & executed
• Using names for equivalence classes allows you to generate data on the fly
– Pick different values every time
– Find data that matches needs in current environment
• Variable values can also be keyword actions
References
• PICT download http://download.microsoft.com/download/f/5/5/f55484df‐8494‐48fa‐8dbd‐8c6f76cc014b/pict33.msi
• PICT documentation http://msdn.microsoft.com/en‐us/library/cc150619.aspx
• More information on pair‐wise testinghttp://pairwise.org
• Bach & Schroeder’s comparison of pairwise & randomly selected test caseshttp://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
3/12/2013
21
Contact Information
Andy TinkhamMagenic [email protected]://magenic.com
http://ohours.org/andytinkhamhttp://testerthoughts.com
http://twitter.com/andytinkham