16-Syntax

ECE453/SE465/CS447/ECE653/CS647: Syntax-based Testing (Mutation Testing)

Lin Tan

February 12, 2014

16-Syntax - February 12, 2014

Mutation Testing

• We will see two types of mutation testing:

• Input-space grammars: to create inputs (both valid and invalid)

• Program-based grammars: to modify programs (valid)

��2


Regular Expression!

• Consider this Perl Regular expression: !

• ^(deposit|debit) [0-9]{3} \$[0-9]+\.[0-9]{2}$

!!

• What are some valid strings?

��3


Grammar

• What is the limitation of regular expressions?

!

• Grammar

• action = dep | deb

• dep = "deposit" account amount

• deb = "debit" account amount

• account = digit{3}

• amount = "$"digit+"."digit{2}

• digit = ["0"-"9"]

��4


Using Grammar!

• Two ways to use input grammars for software testing and maintenance:

• Recognizer: can include them in a program to validate inputs;

• Generator: can create program inputs for testing.

��5


Grammar Terminology

• Grammar

• actions = action*





• amount = “$”digit+"."digit{2}

• digit = ["0"-"9"]

��6

Start symbol

Non-terminals

Production rule

Terminals


Coverage Criteria

• Terminal Symbol Coverage (TSC). TR contains each terminal of grammar G.

• Production Coverage (PDC). TR contains each production of grammar G.

• Derivation Coverage (DC). TR contains every possible string derivable from G.

!• PDC subsumes TSC.

• DC often generates infinite test sets, and even if you limit to fixed-length strings, you still have huge numbers of inputs.

��7


Grammar Terminology

• Grammar







• digit = ["0"-"9"]

��8

Start symbol

Non-terminals

Production rule

Terminals

How many terminals? How many productions? How many derivable strings?


Terminology

• Ground String: A (valid) string belonging to the language of the grammar.

• Mutation Operator: A rule that specifies syntactic variations of strings generated from a grammar.

• Mutant: The result of one application of a mutation operator to a ground string.

!• Mutants may be generated either by modifying existing

strings or by changing a string while it is being generated.

��9


Mutation Testing




��10


Testing Invalid Inputs

• Your program accepts strings described by a regex or a grammar.

• Was it implemented correctly?

!

• Reasons to test invalid inputs:

• Often implementers overlook invalid inputs;

• Undefined behaviour not acceptable (e.g. buffer overruns).

!

• Solution: Mutate grammars and generate test strings from the mutated grammars.

��11


Some Grammar Mutation Operators

• Nonterminal Replacement

• dep = "deposit" account amount =>

• dep = "deposit" amount amount

• Use your judgement to replace nonterminals with similar nonterminals.

!

• Terminal Replacement

• amount = “$”digit+"."digit{2} =>

• amount = “$”digit+"$"digit{2}

��12

deposit $9.22 $12.35

ground string: deposit 123 $12.35

deposit 123 $12$35


More Grammar Mutation Operators

• Terminal and Nonterminal Deletion; e.g.


• dep = "deposit" amount

!

• Terminal and Nonterminal Duplication; e.g.


• dep = "deposit" account account amount

��13

deposit $12.35

deposit 123 123 $12.35


Killing Mutants

!

• Test case t kills m if running t on m produces different output than running t on m0,

• where m is a mutant for an original ground string program m0.

��14


Recall the Terminology

• Ground String: A (valid) string belonging to the language of the grammar.

• Mutation Operator: A rule that specifies syntactic variations of strings generated from a grammar.

• Mutant: The result of one application of a mutation operator to a ground string.

��15


Recall Production

• Grammar







• digit = ["0"-"9"]

��16

Production rule


Coverage Criteria• Mutation Coverage (MC). For each mutant m, TR contains a

requirement to “kill m".

• Mutation score is the percentage of mutants killed.

!• Mutation Operator Coverage (MOC). For each mutation

operator op, TR contains requirement to create a mutated string m derived using op.

!• Mutation Production Coverage (MPC). For each mutation

operator op and each production p that op can be applied to, TR contains requirement to create a mutated string from p.

��17


Mutation Testing




��18


Mutant Testing: Program-based Grammars

• Generating mutants by modifying programs according to the language grammar, using mutation operators. !

• Mutants are valid programs (not tests) which ought to behave differently from the ground string. !

• The goal of mutation testing is to create tests which distinguish mutants from originals.

��19


Example

��20

int foo(int x, int y) { // original! if (x > 5) !

return x + y;! else !

return x;!}

int foo(int x, int y) { // mutant! if (x > 5) !return x - y;! else !return x;!}

• What test case can kill the mutant?

!

• Once we find a test case that kills a mutant, we can forget the mutant and keep the test case. The mutant is then dead.


Uninteresting Mutants

• Three kinds of mutants are uninteresting:

• Stillborn: such mutants cannot compile (or immediately crash);

• Trivial : killed by almost any test case;

• Equivalent: indistinguishable from original program.

!

• The usual application of program-based mutation is to individual statements in unit testing.

��21


Example Mutants

!22

Original Method !int Min (int A, int B) { int minVal; minVal = A; if (B < A) { minVal = B; } return (minVal); } // end Min

Mutant !int Min (int A, int B) { int minVal; minVal = A; ∆ 1 minVal = B; if (B < A) ∆ 2 if (B > A) ∆ 3 if (B < minVal) { minVal = B; ∆ 4 Crash (); ∆ 5 minVal = A; ∆ 6 minVal = failOnZero (B); } return (minVal); } // end Min

6 mutants

Each represents a separate program

Replace one variable with another

Changes operator

Immediate runtime failure … if reached

Immediate runtime failure if B==0 else does nothing


Goals of Mutation Testing

• 1. Mimic (and hence test for) typical mistakes !

• 2. Encode knowledge about specific kinds of effective tests in practice

• Statement coverage (∆4)

• Checking for 0 values (∆6)

��23


Process of Using Mutation Testing

• Goal

• Kill mutants

!

• Desired side effect

• Good tests which kill the mutants.

!

• The tests will help find faults.

• We find these tests by intuition and analysis.

��24


Strong and Weak Mutation

• Strong mutation: A fault must be reachable, infect the state, and propagate to output.

• Weak mutation: A fault which kills a mutant need only be reachable and infect the state. !

• The book claims that experiments show that weak and strong mutation require almost the same number of tests to satisfy them.

��25


Strong Mutation Coverage (SMC)

• Strongly Killing Mutants: Given a mutant m for a program P and a test t, t is said to strongly kill m iff the output of t on P is different from the output of t on m.

!

• Strong Mutation Coverage (SMC). For each mutant m ∈ M, TR contains a test which strongly kills m.

��26


Weak Mutation Coverage (WMC)

• Weakly Killing Mutants: Given a mutant m that modifies a source location L in program P and a test t, t is said to weakly kill m iff the state of the execution of P on t is different from the state of the execution of m on t, immediately after some execution of L.

• Weak Mutation Coverage (WMC). For each mutant m ∈ M, TR contains a test which weakly kills m.

��27


Condition Analysis

!28

Original Method int Min (int A, int B) { int minVal; minVal = A; if (B < A) { minVal = B; } return (minVal); } // end Min

Mutant int Min (int A, int B) { int minVal; ∆ 1 minVal = B; if (B < A) { minVal = B; } return (minVal); } // end Min

• Reachability:

• Infection:

• Propagation:

Unavoidable;

Need B ≠ A;

Need B > A;Wrong minVal needs to return to the caller; that is, we can't execute the body of the if statement.


Conditions

• Reachability: unavoidable;

• Infection: need B ≠ A;

• Propagation: need B > A.

Wrong minVal needs to return to the caller; that is, we can't execute the body of the if statement.

!• Condition for weakly killing mutant: B ≠ A

• Condition for strongly killing mutant: B > A

��29


WMC vs SMC

!30

Original Method int Min (int A, int B) { int minVal; minVal = A; if (B < A) { minVal = B; } return (minVal); } // end Min

With Embedded Mutants int Min (int A, int B) { int minVal; ∆ 1 minVal = B; if (B < A) { minVal = B; } return (minVal); } // end Min

• Condition for strongly killing mutant B > A

• A test case is A = 5, B = 7 (return value =7 , expected = 5).

• Condition for weakly killing mutant: B ≠ A

• A test case is A = 8, B = 2 (return value = 2, expected = 2).


Another Example

!31

Original Method !int Min (int A, int B) { int minVal; minVal = A; if (B < A) { minVal = B; } return (minVal); } // end Min

With Embedded Mutants !int Min (int A, int B) { int minVal; minVal = A; ∆ 3 if (B < minVal) { minVal = B; } return (minVal); } // end Min

• This mutant is an equivalent mutant, since A = minVal. (The infection condition boils down to “false”.)

• Equivalence testing is, in its full generality, undecidable, but we can always estimate.


Workflow of Mutation Testing

��32


Mutation Operators

��33

int mutationTest ( int a , b) { int x = 3 * a , y ; if (a > b) {

y = -x ; } else if ( ! ( a > -b ) ) {

x = a * b ; } return x ;

}

What (intra-procedural) mutation operators can you come up with?

• Absolute Value Insertion

• Operator Replacement

• Arithmetic, Relational, ...

• Scalar Variable Replacement

• Crash Statement replacement


Interface/Integration Mutation

• Change calling method by changing actual parameter values;

• Change calling method by changing callee; or

• Change callee by modifying the return statement.

��34

class M { int f, g; void c(int x) { foo (x, g); bar (3, x); } int foo(int a, int b) { return a + b * f; } int bar(int a, int b) { return a * b; } }


Summary of Syntax-based Testing

• Program-based testing has notion of strong and weak mutants.

• Sometimes we mutate the grammar, not strings, and get tests from the mutated grammar.

��35


PIT Mutation Testing Tool•Conditionals Boundary Mutator

•Negate Conditionals Mutator

•Remove Conditionals Mutator

•Math Mutator

•Increments Mutator

•Invert Negatives Mutator

•Inline Constant Mutator

•Return Values Mutator

•Void Method Calls Mutator

•Non Void Method Calls Mutator

•Constructor Calls Mutator

•Experimental Inline Constant Mutator

•Experimental Member Variable Mutator

•Experimental Switch Mutator��36

• http://pitest.org/


Documents

16-Syntax