View
214
Download
1
Embed Size (px)
Citation preview
1
Software Testing and Quality Assurance
Lecture 9 - Software Testing Techniques
2
Lecture Outline Mutation Analysis
Theory of Mutation analysis and concept of mutant.
Mutation Operator Equivalent mutants
Comparing coverage criteria.
3
ExampleVoid squeeze (char s[],int c){
int i,j;for (i = j = 0; s[i] != ‘\0’; i++)
if (s[i] != c} {s[ j++ ] = s[i];
}}s[i] = ‘\0’;
}
4
ExampleVoid squeeze (char s[],int c){
int i,j;for (i = j = 0; s[i] != ‘\0’; i++)
if (s[i] != c} {s[ j ] = s[i];
}}s[i] = ‘\0’;
}
Now, function places all references to ‘c’ at position 0 in the array. (FAULT)
5
Test Cases for the Example Test cases (Test suite) is executed on
the faulty version; two possibilities An error is encountered, and the
programmer can debug the program to find the fault and repair it.
An error is not encountered, and all the test cases pass.
6
Coverage Based Testing Techniques To date, we have discussed methods
that aim to find faults by achieving coverage. Little idea as to how good the resulting test
suite is.
7
Mutation Analysis Method for measuring the effectiveness
of test suites; and As a result, we produce new test cases
to be added to that test suite.
A mutant of a program is a copy of the program, but with one slight syntactic
change.
8
Fault is a mutant. In this case, the slight syntactic change is the removal of ++ operator in the array index.
Mutant ExampleVoid squeeze (char s[],int c){
int i,j;for (i = j = 0; s[i] != ‘\0’; i++)
if (s[i] != c} {s[ j ] = s[i];
}}s[i] = ‘\0’;
}
9
Mutation Analysis Given a program, A test input for that
program, and a mutant, We say that test case kills the mutant if and only if The output of the program and the mutant differ
for the test inputs. A mutant that has been killed is said to be dead.
A mutant that has not been killed is said to be alive.
10
Coupling Effect - Theory behind Mutation Analysis A test case that distinguishes a small
fault in a program by identifying unexpected behavior is so sensitive that it will distinguish more complex faults. More complex faults are coupled with
simple faults.
This is our general observation as well. Many of the faults produced by test suites are a result of one
minor fault.
11
Coupling Effect, Mutant & Testing Any test suite that does not uncover the
mutant is inadequate. For example,we test the original squeeze
function with a test suite,and all test cases pass. Now, we deliberately insert the fault (to evaluate
the quality of test suites). If fault is uncovered, mutant is killed. Otherwise, test suites are inadequate.
Mutation Analysis Process
12
Mutation Analysis Used to guide test input generation. For
example, Find a test case that kills every mutant.
It is a way of assessing test suite quality. Test suites that kill more mutants is of
higher quality.
13
Mutation Analysis via Mutant Operator Mutant operator is a transformation rule that,
given a program, generates a mutant for that program.
For example, For ‘if (x < y)’ statement, possible mutants are If (x ≤ y) If (x > y) If (x ≥ y) If (x = y) If (x ≠ y) …..
14
Mutation Analysis via Mutant Operator Mutants are syntactic changes to a
program, so mutant operators are depend on the syntax of programming languages. The mutant operators of one programming
language may not necessarily apply to other languages.
15
Equivalent Mutants Given a program and a mutant of that
program, The mutant is said to be equivalent mutant
if, for every input, the program and the mutant produce the same output.
A major problem with mutation analysis
16
Equivalent Mutants - example Squeeze function
S [ j ++] = s [ i ];
S [ j ++] = s [abs[ i ]];
Absolute value function
Variable i only ever takes on value between 0 and the size of the array. Hence, abs[i] is irrelevant.
This mutant is equivalent can not be killed.In industry, testers aim to kill 95% of the mutants.
17
Comparing Coverage Criteria Which criteria gives the best coverage? Frankl et al. [An applicable Family of Data-Flow Testing Criteria, IEEE Transactions
on Software Engineering, 14(10), 1988] compared paths that each criteria selects.
18
Comparing Coverage CriteriaAll-Paths
All-Du-Paths Multiple condition
All-C-Use/Some-P-Use
All Uses
All-P-Uses/Some C-Uses
All-Defs
All-P-Uses
All-Edges
All-Nodes
Decision/condition
Condition
19
Comparing Coverage Criteria
If (y==1) {
x = 2;
}
If (z==1) {
a = f(x);
}
Multiple condition coverage enforces that y == 1 and z == 1 are both executed for the true and false.
To achieve any data-flow criteria, at leastOne test must execute a definition path
From the statement x = 2 to a = f(x).
20
Key points Mutation analysis is a method of
measuring effectiveness of test suites. And as a result, we produce test cases to
improve program testing. The set of paths that satisfy ‘data-flow’
and ‘coverage based testing’ are not necessarily unique.