1

Lee Copelandlee@sqe.com

© SQE 2012

AdvancedTest Case Design

Using PairWise Testing

Topics

• The Problem

• The Magic Solution

• Choosing Tests: Microsoft’s PICT Program

• Conclusion

2

2

The Problem

• Consider the following situations

• What do they have in common?

3

Situation #1

• A web site must operate correctly with:– Internet Explorer 6, 7, and 8, Firefox 3,0,

3.5, and 3.6, and Chrome 8, 9, and 10

– Using different plug-ins – Adobe Acrobat, RealPlayer, or none

– Running on different client operating systems – Windows XP, Vista, and 7

– Receiving pages from different servers –IIS and Apache

– Running on different server operating systems – Windows Server 2003, 2008, and Linux

4

3

Situation #2

• A bank has a new data processing system that

is ready for testing. This bank has:

– Different kinds of customers – consumers,

very important consumers, businesses,

and non-profits

– Different kinds of accounts – checking,

savings, mortgages, consumer loans, and

commercial loans

– They operate in different states, each with

different regulations – California, Nevada,

Utah, Idaho, Arizona, and New Mexico

5

Situation #3

6

m(p){…}

m(p){…} m(p){…} m(p){…}

X

W

ZY

A

B DC

{..obj.m(p)..}

{..obj.m(p)..}

{..obj.m(p)..} {..obj.m(p)..}

m(p)

SR

P

Q

4

The Problem

• What do these very different situations all have in common?

–

–

–

7

THE Problem In Testing

• We must, somehow, select a reasonably sized subset that we could test given our resource constraints. What are some ways of choosing such a subset?

–

–

–

8

5

Possible Solutions

• Don’t test at all. Simply give up because the number of input combinations, and thus the number of test cases, is just too great

• Test all combinations [once], but delay the project so it misses its market window so that everyone quits from stress, or the company goes out of business

• Choose one or two tests and hope for the best

9

Possible Solutions

• Choose the tests you have already run, perhaps as part of programmer-led testing. Incorporate them into a formal test plan and run them again

• Choose the tests that are easy to create and run. Ignore whether they provide useful information about the quality of the product

• Make a list of all the combinations and choose the first few

10

6

Possible Solutions

• Make a list of all the combinations and choose a random subset

• By magic, choose a specially selected, fairly small subset that finds a great many defects– more than you would expect

from such a subset

11

Topics

• The Problem

• The Magic Solution

• Choosing Tests: Microsoft’s PICT Program

• Conclusion

12

7

The Magic Solution

• The magic is not to attempt to test all the combinations of all the values of all combinations of variables but to test all combinations of all the values of all pairs of variables

• This significantly reduces the number of tests that must be created and run …

• while still finding many more defects than you would expect

13

The Magic Solution

14

% Test Cases Executed0%0%

100%

100%

% D

efe

cts

Fo

un

d

8

The Magic Solution

15

% Test Cases Executed0%0%

100%

100%%

De

fec

ts F

ou

nd

The Magic Solution

16

% Test Cases Executed0%0%

100%

100%

% D

efe

cts

Fo

un

d

9

The Magic Solution

17

% Test Cases Executed0%0%

100%

100%%

De

fec

ts F

ou

nd

The Magic Solution

18

% Test Cases Executed0%0%

100%

100%

% D

efe

cts

Fo

un

d

10

Published Findings

• In a case study published by Brownlie of AT&T regarding the testing of a local-area network-based electronic mail system, pairwise testing detected 28 percent more defects than their original plan of developing and executing 1,500 test cases (later reduced to 1,000 because of time constraints) and took 50 percent less effort

19

Published Findings

• A study by the National Institute of Standards and Technology published by Wallace and Kuhn on software defects in recalled medical devices reviewed 15 years of defect data. They concluded that 98 percent of the reported software flaws could have been detected by testing all pairs of parameter settings

20

11

Published Findings

• Kuhn and Reilly analyzed defects recorded in the Mozilla web browser database. They determined that pairwise testing would have detected 76 percent of the reported errors

21

The Promise

1. “Pairwise testing protects against pairwise bugs

2. while dramatically reducing the number of tests to perform

3. which is especially cool because pairwise bugs represent the majority of combinatoric bugs

4. and such bugs are a lot more likely to happen than ones that only happen with more variables

5. Plus, the availability of tools means you no longer need to create tests by hand.”

- James Bach, “Pairwise Testing”

22

12

A Problem To Solve

23

Skype

Tools |Options |Privacy

A Problem To Solve

24

13

A Problem To Solve

• Allow calls from – anyone, contact list (2)

• Receive video from – anyone, contact list, no one (3)

• Allow IMs from – anyone, contact list (2)

• Show video to – contact list, no one (2)

• Keep history for – no history, 2 weeks, 1 month, 3 months, forever (5)

• Show online status – yes, no (2)

• Accept cookies – yes, no (2)

25

2 x 3 x 2 x 2 x 5 x 2 x 2 = 480 combinations

A Problem To Solve

• But, I don’t have time for 480 tests (and neither do you) so let’s use pairwise testing

26

14

Topics

• The Problem

• The Magic Solution

• Choosing Tests: Microsoft’s PICT Program

• Conclusion

27

PICT Program

• Microsoft has developed a tool to generate all the pair combinations

• Microsoft’s algorithm is described in“Pairwise Testing in the Real World: Practical Extensions to Test Case Generators” by Jacek Czerwonka

• PICT is a free tool you can download and use

28

15

PICT Program

29

Clickhere

http://msdn.microsoft.com/en-us/testing/bb980925

PICT Program

• … and download pict33.msi (a Microsoft Installer Package)

• Save it

• Execute it to install. (It installs in the Program Files folder)

• PICTHelp.html has the instructions (They are very helpful)

30

16

PICT Program

• Prepare the input by entering the data into Notepad

31

PICT Program

• Run pict.exe

• Specify it’s input and output. For example:

pict SkypeExampleForPICT.txt > SkypeTestCasesFromPICT.txt

32

17

PICT Program

• Open the output with Excel to make it look nice

33

PICT Program

34

AllowCalls ReceiveVideo AllowIMs ShowVideo KeepHistory ShowStatus AcceptCookies

ContactList ContactList Anyone ContactList OneMonth Yes Yes

Anyone Anyone ContactList NoOne ThreeMonths No No

ContactList NoOne Anyone NoOne Forever No Yes

Anyone ContactList ContactList NoOne NoHistry Yes No

ContactList NoOne ContactList ContactList TwoWeeks Yes No

Anyone NoOne Anyone ContactList ThreeMonths Yes Yes

ContactList Anyone Anyone ContactList NoHistry No Yes

Anyone ContactList ContactList ContactList Forever Yes No

Anyone NoOne ContactList NoOne OneMonth No No

ContactList Anyone Anyone NoOne Forever Yes No

Anyone Anyone Anyone NoOne TwoWeeks No Yes

ContactList ContactList ContactList NoOne ThreeMonths No Yes

Anyone Anyone Anyone ContactList OneMonth No No

Anyone ContactList ContactList NoOne TwoWeeks No Yes

ContactList NoOne ContactList ContactList NoHistry Yes No

18

PICT Program

• Again, note the good news – we can test all pairs of the 7 inputs in only 16 test cases (rather than the total combinations of 480)

35

PICT Program

• PICT has a nice feature the other approaches don’t have – it allows for selection constraints to be specified

36

19

Topics

• The Problem

• The Magic Solution

• Choosing Tests: Microsoft’s PICT Program

• Conclusion

37

The Dangers

1. Pairwise testing will not be effective if you choose the wrong input test data values

2. Pairwise testing will not be effective if you don’t have a good enough oracle. We often assume defects can be immediately observed. In fact, there may be slow internal corruption occurring

3. High-use (including defaults) or high-risk combinations probably don’t get enough attention

38

20

The Dangers

4. We rarely understand the n-wise connectivity within programs we are testing, so it’s not clear that pairwise testing is an appropriate choice

39

Note: These dangers are not exclusive to pairwise testing.

They affect all testing efforts

The Future

• We could put pairwise testing on a more stable “scientific” basis if we were to conduct real, statistically valid experiments of its use and contribution to testing. Perhaps you’d like to begin this effort

• Or, we could just continue to use it and hope for the best

40

21

Thank You

• Thanks for joining with me today. I am honored to have been invited to speak

• If I can be of assistance, or if you’d just like to chat, please contact me at

lee@sqe.com

41

References

• Bach, James, Cem Kaner, and Brett Pettichord. Lessons Learned in Software

Testing.

• Bach, James. “Pairwise Testing: A Best Practice That Isn’t”.

• Copeland, Lee. A Practitioner’s

Guide to Software Test Design.

• Czerwonka, Jacek. “Pairwise Testing in the Real World: Practical Extensions to Test Case Generators”.

42

22

References

• Kuhfeld, Warren. “Orthogonal Arrays: A Library of 117,000”

43

http://support.sas.com/techsup/technote/ts723.html

44