Mutation testing with Pitest

Agenda

1) Mutation testing - definition2) Mutators3) Setup guide of Pitest in Maven4) Run mutation tests5) Pitest Coverage Report 6) Conclusions7) References

Mutation testingDefinition1

Definition

Mutation testing was originally proposed by Richard Lipton as a student in 1971 and first developed and published by DeMillo, Lipton and Sayward. The first implementation of a mutation testing tool has been created by Timothy Budd as part of his PhD work (titled Mutation Analysis) in 1980 from Yale University. [6]

Definition

Mutation testing is used to design new software tests and evaluate the quality of existing software tests. Basically the idea is about seeding artificial defects (mutations) into a source code and checks whether your test suite finds them.

If you change the source code▪ your test fails then mutation is killed - It’s good :)▪ your test passes then mutation survived - It’s bad :(

Mutators2

Mutators

Run by default by Pitest :

▪ INCREMENTS_MUTATOR▪ CONDITIONALS_BOUNDARY_MUTATOR (e.g.)▪ RETURN_VALS_MUTATOR▪ VOID_METHOD_CALL_MUTATOR (e.g.)▪ INVERT_NEGS_MUTATOR▪ MATH_MUTATOR▪ NEGATE_CONDITIONALS_MUTATOR (e.g.)

Conditionals Boundary Mutator

if (a < b) {//some code

}

mutated into

if (a <= b) {//some code

}

Original Mutated

< <=

<= <

> >=

>= >

Negate Conditionals Mutator

if (a == b) {//some code

}

mutated into

if (a != b) {//some code

}

Original Mutated

== !=

!= ==

<= >

>= <

< >=

> <=

Void method call mutator

public void myMethod(int a) {//some code

}public int rootMethod(int x) {

myMethod(x);return x;

}

mutated into

public void myMethod(int a) {//some code

}public int rootMethod(int x) {

return x;}

Mutators

Other mutators disabled by default :

▪ CONSTRUCTOR_CALLS - replaces constructor call with null▪ INLINE_CONSTS - assigns value to non-final variable▪ NON_VOID_METHOD_CALLS - removes calls to non void methods▪ REMOVE_CONDITIONALS - removes all conditionals statements▪ EXPERIMENTAL_MEMBER_VARIABLE - removes assignments to

member variables▪ EXPERIMENTAL_SWITCH - changes switch statement by replacing

the default label (wherever it is used) with first label. All the other labels are replaced by the default one

Setup guide of Pitest in Maven3

Run mutation tests4

Pitest Coverage Report6

Conclusions7

Conclusions

Mutation testing seems powerful and research indicates that mutation score is a better predictor of real fault detection rate than code coverage [2]. However, it has not yet received widespread popularity.

Creating mutations and executing tests against those mutations is not a lightweight process and can take quite a lot of time.

In addition to finding bugs, mutation testing is a great way to find redundant code thus making your code cleaner!

Conclusions

Should all mutants be killed ? No, but it depends.try { connection.close();} catch(SQLException e){ doNext("Call DevOps & create JIRA task for Dev :)");}

To get 100% line coverage here would mean adding a test scenario that throws an error on connection.close() and verify the call to doNext(..). If you haven’t got it covered, PIT will remove the call to ‘doNext’ and complain in turn that you didn’t kill that mutant. By all means do, if your time and budget are limitless.

Remember! Mutant is only an instance in which an artificial change to your code was not detected by your tests.

References8

References1) www.pitest.org2) Are Mutants a Valid Substitute for Real Faults in Software

Testing? https://people.cs.umass.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf

3) https://blog.codecentric.de/en/2016/02/sensible-mutation-testing-dont-go-killing-spree-2/

4) http://www.hascode.com/2015/05/mutation-testing-with-pitest-and-maven/

5) https://gofore.com/find-bugs-mutation-testing/6) https://en.wikipedia.org/wiki/Mutation_testing

THANKS!

Any questions ?

Thank you http://www.slidescarnival.com for layout