Orthogonal Array Orthogonal Array Testing StrategyTesting Strategy

Abstract

Orthogonal Array Testing Strategy (OATS) is a

systematical, statistical way of testing pair-wise

interactions by deriving suitable small set of test cases

from a large number of scenarios. The testing strategy

can be used to reduce the number of test combinations

and provide maximum coverage with a minimum number

of test cases. OATS utilizes an array of values

representing variable factors that are combined pair-wise

rather than representing all combinations of factors and

levels.

Abstract

Being intelligent about which test cases are selected can

make the difference between “endlessly executing” over

“executing a concise, defined set of test cases”. OATS

steps involve identifying, determining, finding, mapping,

choosing, and transcribing. As an end result, orthogonal

array testing provides a means to select a test set that

guarantees, creates, exercises, and is simplistic, deriving

a suitable small set of test cases from a large number of

scenarios with broad coverage.

• Definition for Orthogonal Array Testing Strategy (OATS)

• The OAT Theory

• Simple OAT Example– Pairwise Combination of Parameters

• OATS Example 2

• Why Use Orthogonal Array Testing?

• Advantages of OATS

• Disadvantages of OATS

Outline

• Selecting the Right Test Set with OATS

• Mistakes to Avoid

– Applying OATS manually

– Focusing OATS on the wrong areas of applications

– Using OATS for minimal testing efforts

– Using OATS for High-Risk Applications

– Picking the Wrong Parameters to Combine

Outline

• How to Utilize and Apply OATS?

• What Can OATS Provide?

• Definition of Pairwise

• Pairwise Facts

• Where to get Pairwise?

• Summary

• References

• Resourceful Websites

Outline

Definition of Orthogonal Array Testing Strategy

• Orthogonal Array Testing Strategy (OATS) is a systematical, statistical way of testing pairwise interactions by deriving a suitable small set of test cases from a large number of scenarios.

The OAT Theory

• OAT can be used to reduce the number of combinations and provide maximum coverage with a minimum number of test cases.

• OAT is an array of values in which each column represents a variable - factor that can take a certain set of values called levels. Each row represents a test case.

• In OAT, the factors are combined pairwise rather than representing all possible combinations of factors and levels.

• Orthogonal arrays are two dimensional arrays of numbers which possess the interesting quality that by choosing any two columns in the array you receive an even distribution of all the pairwise combinations of values in the array.

Simple OAT Example

• Consider three parameters – A, B, and C.

• Parameters A, B, and C each have positive values – 1, 2, and 3.

• Testing all combinations of the three parameters would involve executing a total of 27 test cases.

Simple OAT Example (cont’d)

• With the way programming works, a fault will most likely depend on the values of two parameters, not three.

• The fault might occur for each of these three test cases: A=1, B=1, C=1, A=1, B=1, C=2, and A=1, B=1, C=3.

• Not necessary to run all 27 scenarios, 9 test cases is suffice.

• The 9 scenarios outlined in Table 1 (on the next screen) address all possible pairs within the three parameters.

Pair-wise Combination of Parameters

• All possible pairwise combinations between parameters A and B, B and C, and C and A are displayed in Table 1.

Orthogonal Array – Example 2(More Complex)

• Orthogonal Array has 3 columns, representing 3 variables.

• Utilize a three level-alphabet – Red, Green, and Blue.

• Next, organize combinations into a table (next slide).

Orthogonal Array – Example 2All Combinations for Three Variables of Three Levels Each

  A B C

1 Red Red Red

2 Red Red Green

3 Red Red Blue

4 Red Green Red

5 Red Green Green

6 Red Green Blue

7 Red Blue Red

8 Red Blue Green

9 Red Blue Blue

10 Blue Red Red

11 Blue Red Green

12 Blue Red Blue

13 Blue Green Red

14 Blue Green Green

15 Blue Green Blue

16 Blue Blue Red

17 Blue Blue Green

18 Blue Blue Blue

19 Green Red Red

20 Green Red Green

21 Green Red Blue

22 Green Green Red

23 Green Green Green

24 Green Green Blue

25 Green Blue Red

26 Green Blue Green

27 Green Blue Blue

Orthogonal Array – Three Variables of Three Levels Each

• For each pair of columns, AB, AC, and BC, each pair of colors appears exactly three times.

• To save testing effort, the appearance of each pair has been reduced to once.

Orthogonal Array – Example 2

All-Pairs Array, Three Variables of Three Levels Each

  A B C

2 Red Red Green

4 Red Green Red

9 Red Blue Blue

12 Blue Red Blue

14 Blue Green Green

16 Blue Blue Red

19 Green Red Red

24 Green Green Blue

26 Green Blue Green

Why Use Orthogonal Array Testing?

• Being intelligent about which test cases you choose can make all the difference between:

- Endlessly executing tests that just aren’t likely to find bugs and don’t increase your confidence in the system.

- Executing a concise, well-defined set of tests that are likely to uncover most (not all) of the bugs and that give you a great deal more comfort in the quality of your software.

Advantages of OATS

• Utilizes variable pair combination

• Provides uniformly distributed coverage

• Useful for integration testing of software components

• Useful for testing configurable options on an application web page allowing font style background colorpage layout selections

Disadvantages of OATS

• Can only be applied at the initial stage of the product/process design system.

• There are some situations whereby OA techniques are not applicable, such as a processes involving influencing factors that vary in time and cannot be quantified exactly.

• Can be overwhelming to use the first time; improves with use on each project.

• Arrays can be difficult to construct.

Selecting the Right Test Set with OATS

• OATS provides a means to select a test set that:

1. Guarantees testing the pair-wise combinations of all the selected variables.

2. Creates an efficient and concise test set with many fewer test cases than testing all combinations of all variables.

Selecting the Right Test Set with OATS (cont’d)

3. Creates a test set that has an even distribution of all pair-wise combinations.

4. Exercises some of the complex combinations of all the variables.

5. Is simpler to generate and less error prone than test sets created by hand.

Mistakes to Avoid

1. Applying OATS manually.

2. Focusing the testing effort on the wrong area of the application.

3. Using OATS for minimal testing efforts.

4. Using OATS for high-risk applications.

5. Picking the wrong parameters to combine.

Applying OATS manually

• Manual entry of OATS test parameters is time consuming and prone to inaccuracies.

• Utilize free orthogonal array freeware, to avoid manually applying OATS.

• Orthogonal freeware can generate various array sizes.

Focusing the testing effort on the wrong area of the application

• Focus OATS on the critical areas of the application.

• Evaluate where applying OATS is most feasible.

• It is important to evaluate where the application testing effort will be most effective and efficient to apply.

Using OATS for minimal testing efforts

• There are times when OATS might ask for more test cases than is actually necessary.

• Determine if budget and time allows for a “full testing effort” or a “good enough testing effort”.

• If utilize OATS-generated test case combination, omit unnecessary tests.

Using OATS for high-risk applications

• Testing medical device systems or other life-critical applications require more than just OATS.

• Other testing techniques must be used to complement OATS.

• OATS is used to minimize the test combination inputs.

• Not a technique for exhaustive testing.

Picking the Wrong Parameters to Combine

• Understand how the application functions.

• Enlist the help of:– Subject Matter Experts (SMEs)– Developers– Business Analysts

• Pick the right parameter combinations based on feedback from SME, Developers, and BAs.

How to Utilize and Apply OATS?

• Steps to use OATS are outlined below:

1. Identify independent variables

2. Determine the maximum number of values

3. Find a suitable array with the smallest number of runs

4. Map the factors and values onto the array

5. Choose values for any remaining level

6. Transcribe the runs into test cases

What Can OATS Provide?

• OATS provides a means to select a test set that:

1. Guarantees testing the pair-wise combinations of all selected variables.

2. Creates an efficient and concise test set with fewer test cases.

3. Creates a test set that has an even distribution of all pair-wise combinations

What Can OATS Provide? (cont’d)

• OATS provides a means to select a test set (cont’d):

4. Exercises some of the complex combinations of all variables.

5. Is simpler to generate and less error prone than test sets created by hand.

Definition of Pairwise Testing

• Recall Orthogonal Array Testing Strategy is a systematical, statistical way of testing pairwise interactions by deriving a suitable small set of test cases from a large number of scenarios.

• Pairwise testing is a type of combination testing method, where each pair of input parameters to a system (usually software algorithms), tests all possible combinations of the parameters.

Pairwise Facts

• Pairwise testing is also known as Orthogonal Array Testing.

• The goal of using orthogonal arrays in testing is: – to get the biggest bang for the testing buck– performing the smallest number of tests likely to

expose defects

• To keep the test scenario numbers low, the desired combinations are pairs of variables in each of their possible states, rather than combinations of three or more variables. This makes pair-wise testing a kind of subset of orthogonal array testing.

Where to get All Pairs?

• All Pairs software will test for all possible discrete combinations of the parameters identified for testing.

• All Pairs can be downloaded from http://www.satisfice.com/tools.shtml

Summary

• Orthogonal Array Testing Strategy is a systematical, statistical way of testing pair-wise interactions by deriving suitable small set of test cases from a large number of scenarios.

• OAT should be used to reduce the number of combinations and provide maximum coverage with a minimum number of test cases.

• OAT guarantees testing the pair-wise combinations of all selected variables.

Summary

• Mistakes to Avoid.1. Focusing the testing effort on the wrong area of the

application.

2. Using OATS for minimal testing efforts.

3. Using OATS for high-risk applications.

4. Picking the wrong parameters to combine.

• Pairwise testing is a type of combination testing method, where each pair of input parameters to a system (usually software algorithms), tests all possible combinations of the parameters.

References

• Bolton, M. (2004). Pairwise testing. Retrieved on December 1, 2009, from http://www.developsense.com/testing/PairwiseTesting.html

• Dustin, E. (2001). Orthogonally Speaking. Retrieved on November 17, 2009, from http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=MAGAZINE&ObjectId=5031&tth=DYN&tt=siteemail&iDyn=2#table3go#table3go

References (cont’d)

• Harrell, J. (2001). Orthogonal Array Testing Strategy (OATS) Technique. Retrieved on November 29, from http://upiia.uab.es/teach/a21291/apunts/provaOO/OATS.pdf

• Learn Software Development. (2007). Orthogonal Array Testing Strategy (OATS). Retrieved on November 21, 2009, from http://learnsoftwareprocesses.com/2007/08/21/orthogonal-array-testing-strategy-oats/

References (cont’d)

• StickyMinds. (2001). Orthogonally Speaking. Retrieved on November 15, 2009, from http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=MAGAZINE&ObjectId=5031&tth=DYN&tt=siteemail&iDyn=2#table3go#table3go

• VTB. (2008). Orthogonal Array Testing Strategy (OATS). Retrieved on November 10, 2009, from http://www.vietnamesetestingboard.org/zbxe/?mid=download&category=12674&listStyle=list&document_srl=55352

Resourceful Websites

• http://www.combinatorialtesting.com/clear-introductions

• http://www.isixsigma.com/library/content/c030106a.asp

• http://upiia.uab.es/teach/a21291/apunts/provaOO/OATS.pdf

• http://www.vietnamesetestingboard.org/zbxe/?mid=download&category=12674&listStyle=list&document_srl=55352

• http://www.developsense.com/testing/PairwiseTesting.html