Upload
techwellpresentations
View
297
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Jamie Mitchell explores an underused and often forgotten test technique—white-box testing. Also known as structural testing, this technique requires some programming expertise and access to the code. Using only black-box testing, you could easily ship a system having tested only 50 percent or less of the code base. Are you comfortable with that? For mission-critical systems, such low test code coverage is clearly insufficient. Although you might believe that the developers have performed sufficient unit and integration testing, how do you know that they have achieved the level of coverage that your project requires? Jamie describes the levels of code coverage that the business and your customers may need—from statement coverage to modified condition/decision coverage. He explains when you should strive to achieve different code coverage target levels and leads you through examples of pseudocode. Even if you have no personal programming experience, understanding structural testing will make you a better tester. So, join Jamie in this code-diving session.
Citation preview
Structural TestingWhen Quality Really Matters
Jamie L Mitchell
Jamie Mitchell Consulting Inc.
Copyright (c) Jamie Mitchell Consulting Inc.
2
When Software Really Needs to Work
Copyright (c) Jamie Mitchell Consulting Inc.
3
Why Structure-based Testing?Experience-based techniques are not thorough
enough:◦ Error-guessing◦ Exploratory testing◦ Defect- and taxonomy-based
Black-box testing depends on how good your test basis model is:◦ Requirements◦ Use cases◦ Functional specifications
How well have you tested?
Is testing less than half of your code sufficient?
Copyright (c) Jamie Mitchell Consulting Inc.
4
White-Box Test BasisDesign documents and models
◦Architecture◦Functional design◦Low-level design
Object models Class interface documents
Code
Copyright (c) Jamie Mitchell Consulting Inc.
5
Today’s Discussion TargetsControl-flow analysis: selecting
certain paths through the software to test, based on desired coverage
Data-flow analysis: a control-flow technique that scrutinizes the life-cycle of data variables
Copyright (c) Jamie Mitchell Consulting Inc.
6
Control-flow TestingSelect certain paths through the software ◦Based on desired level of coverage Based on statements Based on decisions Based on loops Based on paths
Copyright (c) Jamie Mitchell Consulting Inc.
7
Statement CoverageExecute every line of codeTools available for measurement
◦Run tool while executing tests◦Evaluate coverage◦Create more tests to reach areas
missedAlso called instruction or code
coverage
Copyright (c) Jamie Mitchell Consulting Inc.
8
Decision (Branch)CoverageStatement coverage may miss bugs
◦ One test needed for statement coverage (a = 5, b = 4) Passes
z = 0;if (a > b) then
z = 12;x = 72 / z
Select inputs to force each decision to execute both possible ways (T/F)◦ Now two test cases are needed for coverage
(a = 5, b = 4) Passes (a = 4, b = 5) Finds bug
Copyright (c) Jamie Mitchell Consulting Inc.
9
Condition CoverageDecisions may be arbitrarily complex
if (a > b) then… if (A || B) && (C == D) || (!E) && (F) then…
Condition testing requires that each atomic condition must be evaluated both true and false
Note that condition coverage – by itself – may not be as strong as decision coverage
if (a AND b) then
(a = F, b = T F) (a = T, b = F F)
Copyright (c) Jamie Mitchell Consulting Inc.
10
Decision / Condition CoverageA combination of Condition and
Decision1. Each atomic condition must be evaluated
both ways AND2. Decision coverage must be satisfied
if (a AND b) then
(a = F, b = T F) (a = T, b = F F) (a = T, b = T T)
Is that enough coverage?
Copyright (c) Jamie Mitchell Consulting Inc.
11
Modified Condition / DecisionUsually called MC/DC
◦For each condition in the predicate, there is at least one test where the outcome is TRUE only because of that condition being TRUE
◦For each condition in the predicate, there is at least one test where the outcome is FALSE only because of that condition being FALSE
Usually results in N+1 test cases where N is number of conditions in predicate
Required by some safety standards to ensure that enough testing has been done
Ex: FAA DO/178B Level A (catastrophic) software
Copyright (c) Jamie Mitchell Consulting Inc.
12
MC/DC Coverage (2)
if ((a || b) && c) then …
Decision/Condition can be achieved with
(a = T, b = T, c = T) T (a = F, b = F, c = F) F
Note that (b) never independently affects the outcome of the condition
Copyright (c) Jamie Mitchell Consulting Inc.
13
MC/DC Coverage (3) if ((a || b) && c) then …
Following test set will achieve MC/DC coverage
1. (a = T, b = F, c = T) T 2. (a = F, b = F, c = T) F3. (a = F, b = T, c = T) T4. (a = T, b = F, c = F) F
Copyright (c) Jamie Mitchell Consulting Inc.
14
Coverage Based on LoopsLoops make testing …
interesting…The number of possible paths
may approach infinite depending on how many loops are executed
Basic coverage criteria◦Test when no loops are made◦Test when a single loop is made◦Test when n loops are made (where n
is either the expected max or a large number)
Copyright (c) Jamie Mitchell Consulting Inc.
15
Loops (2)This loop can calculate a factorial (n!)
for (i = 1; i <= n; i++) f *= i;
Use (n = 0) for zero loopsUse (n = 1) for one time throughDetermining the max requires
thought◦In this case, (f) gets very large◦Use (n = 12); 13 would cause an overflow
Copyright (c) Jamie Mitchell Consulting Inc.
16
Nested LoopsNested loops are really hard to testBoris Beizer* had some advice
1. Start at the innermost loop, setting all outer loops to their minimum iteration setting
2. Test boundaries for innermost loop while keeping outer loops at their minimum
3. If you have done outermost loop goto 5. Else move one loop outward and test as in step 2 with all inner loops set to their typical values
4. Continue outward until all loops done5. Test the boundaries for all loops simultaneously
* Software Testing Techniques, 2nd Ed. by Boris Beizer
Copyright (c) Jamie Mitchell Consulting Inc.
17
Path TestingFull path testing in a non-trivial system is
impossible◦The number of paths approaches infinite◦This is due to loops as seen previously
We can identify independent, non-looping paths which cover all the edges and nodes of a control-flow graph◦These are called basis paths◦Testing the basis paths can guarantee decision
(and statement) coverage◦McCabe cyclomatic complexity uses this method◦Tools are available for calculating this value
Copyright (c) Jamie Mitchell Consulting Inc.
18
Cyclomatic Complexity ExampleComplexity = 3Tests needed
◦ 1-2-6-7-8◦ 1-3-4-7-8◦ 1-3-5-8
Example from “A Complexity Measure” by Thomas McCabe, IEEE Transactions on SW Engineering, Vol. SE-2, No 4, December 1976
Copyright (c) Jamie Mitchell Consulting Inc.
19
Data-flow Analysis (1)Programs create, set, read, evaluate and
destroy dataThe life cycle of data can be analyzed to find
defects that control-flow techniques missSome defect examples
◦ Incorrect assignment of a value◦ Incorrect input statement ◦ Failure to define a variable before use◦ Incorrect path taken due to incorrect value in a
control predicate ◦ Use of a variable after it is destroyed or out of scope◦ Redefinition of a variable before it can be used.
Look for consecutive touches to each variable
Copyright (c) Jamie Mitchell Consulting Inc.
20
Data-flow Analysis (2)Dependent on the language usedWe look for anomalies in data life-
cycle ◦Not all anomalies are defects◦While this is nominally a static technique,
it will not find all data defects (often data is manipulated or assigned dynamically)
Sometimes called DUK testing◦D Define◦U Use◦K Kill
Copyright (c) Jamie Mitchell Consulting Inc.
21
Data-flow Testing Combinations (1)
Anomaly Explanation1. ~d first
defineAllowed
2. du define-use
Allowed, normal case
3. dk define-kill Potential bug; data was never used
4. ~u first use Potential bug; data was used without definition. It may be a global variable, set outside the routine
5. ud use-define
Allowed: data used and then re-defined
6. uk use-kill Allowed
7. ~k first kill Potential bug; data is killed before definition
8. ku kill-use Serious defect; data is used after being killed
Copyright (c) Jamie Mitchell Consulting Inc.
22
Data-flow Testing Combinations (2)
Anomaly Explanation9. kd kill-define Usually allowed. Data is killed and
then re-defined. Some theorists believe this should be disallowed
10.
dd define-define
Potential bug; double definition
11.
uu use-use Allowed; normal case. Some do not bother testing this pair since no redefinition occurred
12.
kk kill-kill Potential bug
13.
d~ define last Potential bug; dead variable? May be a global variable which is used in other context
14.
u~ use last Allowed. Variable was used in this routine but not killed off
15.
k~ kill last Allowed; normal case
Copyright (c) Jamie Mitchell Consulting Inc.
23
Function to Test1. public static double calculateBill (int Usage) {
2. double Bill = 0;
3. if(Usage > 0) {
4. Bill = 40;
5.
6. if(Usage > 100) {
7. if(Usage <= 200) {
8. Bill = Bill + (Usage - 100) * 0.5;
9. }
10. else {
11. Bill = Bill + 50 + (Usage - 200) * 0.1;
12. if(Bill >= 100) {
13. Bill = Bill * 0.9;
14. }
15. }
16. }
17. }
18. return Bill;
19. }
Copyright (c) Jamie Mitchell Consulting Inc.
24
Variable DUK Patterns
Pattern Explanation
1.
~d (1) normal case
2.
du (1-3) normal case
3.
uu (3-6)(6-7)(7-8)(7-11)
normal case
4.
uk (6-19)(8-19)(11-19)
normal case
5.
k~ (19) normal case
Pattern Explanation
1.
~d (2) normal case
2.
dd (2-4) suspicious
3.
du (2-18)(4-8)(4-11)(11-12)
normal case
4.
ud (8-8)(11-11)(13-13)
acceptable
5.
uu (12-13)(12-18) normal case
6.
uk (18-19) normal case
7.
k~ (19) normal case
Usage data-flow information
Bill data-flow information
Copyright (c) Jamie Mitchell Consulting Inc.
25
SummaryMany testers simply use black-
box and experience-based techniques
Some systems require far deeper testing
This presentation barely touches the many advanced techniques that can be used in testing
Learning these techniques can give you an advantage in your career