Improve your tests with Mutation Testing
Nicolas Fränkel
@nicolas_frankel 2
Me, Myself and I
Developer & ArchitectAs Consultant
Teacher/trainerBloggerSpeakerBook Author
@nicolas_frankel 3
Shameless self-promotion
@nicolas_frankel 4
My job
Many kinds of testing
Unit TestingIntegration TestingEnd-to-end TestingPerformance TestingPenetration TestingExploratory Testingetc.
@nicolas_frankel 5
Their only single goal
Ensure the Quality of the production code
@nicolas_frankel 6
The problem
How to check the Quality of the testing code?
@nicolas_frankel 7
Code coverage
“Code coverage is a measure used to describe the degree to which the source code of a program is tested”
--Wikipediahttp://en.wikipedia.org/wiki/
Code_coverage
@nicolas_frankel 8
Measuring Code Coverage
Check whether a source code line is executed during a testOr Branch Coverage
@nicolas_frankel 9
@nicolas_frankel 10
Computing Code Coverage
CC: Code Coverage(in percent)
Lexecuted: Number of executed lines of code
Ltotal: Number of total lines of code
Java Tools for Code Coverage
JaCoCoCloverCoberturaetc.
@nicolas_frankel 11
100% Code Coverage?
“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”
Robert Martin (Uncle Bob)https://twitter.com/unclebobmartin/status/
55966620509667328
@nicolas_frankel 12
@nicolas_frankel 13
Assert-less testing
@Testpublic void add_should_add() { new Math().add(1, 1);}But, where is the assert?As long as the Code Coverage
is OK…
Code coverage as a measure of test quality
Any metric can be gamed!Code coverage is a metric…
⇒ Code coverage can be gamedOn purposeOr by accident
@nicolas_frankel 14
Code coverage as a measure of test quality
Code Coverage lulls you into a false sense of security…
@nicolas_frankel 15
The problem still stands
Code coverage cannot ensure test qualityIs there another way?
Mutation Testing to the rescue!
@nicolas_frankel 16
@nicolas_frankel 17
The Cast
William StrykerOriginal Source Code
Jason StrykerModified Source Code
a.k.a “The Mutant”
@nicolas_frankel 18
public class Math { public int add(int i1, int i2) { return i1 + i2; }}
public class Math { public int add(int i1, int i2) { return i1 - i2; }}
@nicolas_frankel 19
Standard testing
✔Execute Test
@nicolas_frankel 20
Mutation testing
?Execute SAME Test
MUTATION
@nicolas_frankel 21
Mutation testing
✗
✔Execute SAME Test
Execute SAME Test
Mutant Killed
Mutant Survived
Killed or Surviving?
Surviving means changing the source code did not change the test resultIt’s bad!
Killed means changing the source code changed the test resultIt’s good
@nicolas_frankel 22
@nicolas_frankel 23
Test the code
public class Math { public int add(int i1, int i2) { return i1 + i2; }}
@Testpublic void add_should_add() { new Math().add(1, 1);}
✔Execute Test
@nicolas_frankel 24
Surviving mutant
public class Math { public int add(int i1, int i2) { return i1 - i2; }}
@Testpublic void add_should_add() { new Math().add(1, 1);}
✔Execute SAME Test
@nicolas_frankel 25
Test the code
public class Math { public int add(int i1, int i2) { return i1 + i2; }}
@Testpublic void add_should_add() { int sum = new Math().add(1, 1); Assert.assertEquals(sum, 2);}
✔Execute Test
@nicolas_frankel 26
Killed mutant
public class Math { public int add(int i1, int i2) { return i1 - i2; }}
@Testpublic void add_should_add() { int sum = new Math().add(1, 1); Assert.assertEquals(sum, 2);}
✗Execute SAME Test
Mutation Testing in Java
PIT is a tool for Mutation testingAvailable as
Command-line toolAnt targetMaven plugin
@nicolas_frankel 27
Mutators
Mutators are patterns applied to source code to produce mutations
@nicolas_frankel 28
@nicolas_frankel 29
PIT mutators sample
Name Example source ResultConditionals Boundary > >=Negate Conditionals == !=Remove Conditionals foo == bar trueMath + -Increments foo++ foo--Invert Negatives -foo fooInline Constant static final FOO= 42 static final FOO =
43Return Values return true return falseVoid Method Call System.out.println("foo")Non Void Method Call long t =
System.currentTimeMillis()long t = 0
Constructor Call Date d = new Date() Date d = null;
Important mutators
Conditionals BoundaryProbably a potential serious bug smell
if (foo > bar)
@nicolas_frankel 30
Important mutators
Void Method Call
Assert.checkNotNull()connection.close()
@nicolas_frankel 31
Remember
It’s not because the IDE generates code safely that it will never change
equals()hashCode()
@nicolas_frankel 32
False positives
Mutation Testing is not 100% bulletproof
Might return false positivesBe cautious!
@nicolas_frankel 33
@nicolas_frankel 34
Enough talk…
Time for DEMO
Drawbacks
SlowSluggishCrawlingSulkyLethargicetc.
@nicolas_frankel 39
Metrics (kind of)
On joda-moneymvn clean test-compilemvn surefire:test
Total time: 2.181 s
mvn pit-test...Total time: 48.634 s
@nicolas_frankel 40
Why so slow?
Analyze test codeFor each class under test
For each mutatorCreate mutation
For each mutationRun testAnalyze resultAggregate results
@nicolas_frankel 41
Workarounds
This is not acceptable in a normal test run
But there are workarounds
@nicolas_frankel 42
@nicolas_frankel 43
Set mutators
<configuration> <mutators> <mutator> CONSTRUCTOR_CALLS </mutator> <mutator> NON_VOID_METHOD_CALLS </mutator> </mutators></configuration>
@nicolas_frankel 44
Set target classes
<configuration> <targetClasses> <param>ch.frankel.pit*</param> </targetClasses></configuration>
@nicolas_frankel 45
Set target tests
<configuration> <targetTests> <param>ch.frankel.pit*</param> </targetTests></configuration>
@nicolas_frankel 46
Dependency distance
1 2
@nicolas_frankel 47
Limit dependency distance
<configuration> <maxDependencyDistance> 4 </maxDependencyDistance></configuration>
@nicolas_frankel 48
Limit number of mutations
<configuration> <maxMutationsPerClass> 10 </maxMutationsPerClass></configuration>
Use MutationFilter
@nicolas_frankel 49
PIT extension points
Must be packaged in JARHave Implementation-Vendor and
Implementation-Title in MANIFEST.MF that match PIT’s
Set on the classpathUse Java’s Service Provider feature
@nicolas_frankel 50
Service Provider
Inversion of controlSince Java 1.3!
Text file located in META-INF/services
InterfaceName of the file
Implementation classContent of the file
@nicolas_frankel 51
@nicolas_frankel 52
Sample structure
JARch.frankel.lts.Sample
@nicolas_frankel 53
Service Provider API
@nicolas_frankel 54
Service Provider sample
ServiceLoader<ISample> loaders = ServiceLoader.load(ISample.class);
for (ISample sample: loaders) {// Use the sample
}
Output formats
Out-of-the-boxHTMLXMLCSV
@nicolas_frankel 55
@nicolas_frankel 56
Output formats
<configuration> <outputFormats> <outputFormat>XML</outputFormat> <outputFormat>HTML</outputFormat> </outputFormats></configuration>
@nicolas_frankel 57
Mutation Filter
Remove mutations from the list of available mutations for a class
@nicolas_frankel 58
@nicolas_frankel 59
@nicolas_frankel 60
Time for DEMO
@nicolas_frankel 61
Don’t bind to test phase!
<plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <executions> <execution> <goals> <goal>mutationCoverage</goal> </goals> <phase>test</phase> </execution> </executions></plugin>
@nicolas_frankel 62
Use scmMutationCoverage
mvn \org.pitest:pitest-maven:scmMutationCoverage \
-DtimestampedReports=false
maven-scm-
plugin
must be configured!
@nicolas_frankel 63
Do use on Continuous Integration servers
mvn \org.pitest:pitest-maven:mutationCoverage \-DtimestampedReports=false
Is Mutation Testing the Silver Bullet?
Sorry, no!It only
Checks the relevance of your unit testsPoints out potential bugs
@nicolas_frankel 64
What it doesn’t do
Validate the assembled applicationIntegration Testing
Check the performancePerformance Testing
Look out for display bugsEnd-to-end testing
Etc.
@nicolas_frankel 65
Testing is about ROI
Don’t test to achieve 100% coverageTest because it saves money in the
long runPrioritize:
Business-critical codeComplex code
@nicolas_frankel 66
@nicolas_frankel 67
Q&A
@nicolas_frankelhttp://blog.frankel.ch/https://leanpub.com/
integrationtest/