Javantura v3 - Mutation Testing for everyone – Nicolas Fränkel

IMPROVE YOUR TEST QUALITY WITH MUTATION TESTINGNICOLAS FRANKEL@nicolas_frankel

@nicolas_frankel

ME, MYSELF AND IDeveloper & Architect•As Consultant

Teacher/trainerBook AuthorBlogger

@nicolas_frankel #mutationtesting 3

SHAMELESS SELF-PROMOTION

MANY KINDS OF TESTINGUnit TestingIntegration TestingEnd-to-end TestingPerformance TestingPenetration TestingExploratory Testingetc.

@nicolas_frankel #mutationtesting 4

THEIR ONLY SINGLE GOALEnsure the Quality of the production code

@nicolas_frankel #mutationtesting 5

THE PROBLEMHow to check the Quality of the testing code?

@nicolas_frankel #mutationtesting 6

CODE COVERAGE“Code coverage is a measure used to describe the degree to which the source code of a program is tested”

--Wikipediahttp://en.wikipedia.org/wiki/

Code_coverage

@nicolas_frankel #mutationtesting 7

MEASURING CODE COVERAGECheck whether a source code line is executed during a test•Or Branch Coverage

@nicolas_frankel #mutationtesting 8

@nicolas_frankel #mutationtesting 9

COMPUTING CODE COVERAGECC: Code Coverage(in percent)Lexecuted: Number of executed lines of codeLtotal: Number of total lines of code

JAVA TOOLS FOR CODE COVERAGEJaCoCoCloverCoberturaetc.

@nicolas_frankel #mutationtesting 10

100% CODE COVERAGE?“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”

Robert Martin (Uncle Bob)https://twitter.com/unclebobmarti

n/status/55966620509667328

@nicolas_frankel #mutationtesting 11

@nicolas_frankel #mutationtesting 12

ASSERT-LESS TESTING@Testpublic void add_should_add() {new Math().add(1, 1);

}

But, where is the assert?

As long as the Code Coverage is OK…

CODE COVERAGE AS A MEASURE OF TEST QUALITYAny metric can be gamed!Code coverage is a metric…

⇒ Code coverage can be gamed• On purpose• Or by accident

@nicolas_frankel #mutationtesting 13

CODE COVERAGE AS A MEASURE OF TEST QUALITYCode Coverage lulls you into a false sense of security…

@nicolas_frankel #mutationtesting 14

THE PROBLEM STILL STANDSCode coverage cannot ensure test quality• Is there another way?

Mutation Testing to the rescue!

@nicolas_frankel #mutationtesting 15

@nicolas_frankel #mutationtesting 16

THE CAST

William StrykerOriginal Source Code

Jason StrykerModified Source Code

a.k.a “The Mutant”

@nicolas_frankel #mutationtesting 17

public class Math { public int add(int i1, int i2) { return i1 + i2; }}

public class Math { public int add(int i1, int i2) { return i1 - i2; }}

@nicolas_frankel #mutationtesting 18

STANDARD TESTING

✔Execute Test

@nicolas_frankel #mutationtesting 19

MUTATION TESTING

?Execute SAME Test

MUTATION

@nicolas_frankel #mutationtesting 20

MUTATION TESTING

✗

✔Execute SAME Test

Execute SAME Test

Mutant Killed

Mutant Survived

KILLED OR SURVIVING?Surviving means changing the source code did not change the test result• It’s bad!

Killed means changing the source code changed the test result• It’s good

@nicolas_frankel #mutationtesting 21

@nicolas_frankel #mutationtesting 22

TEST THE CODEpublic class Math {public int add(int i1, int i2) { return i1 + i2;}

}

@Testpublic void add_should_add() {new Math().add(1, 1);

}

✔Execute Test

@nicolas_frankel #mutationtesting 23

SURVIVING MUTANTpublic class Math {public int add(int i1, int i2) { return i1 - i2;}

}

@Testpublic void add_should_add() {new Math().add(1, 1);

}

✔Execute SAME Test

@nicolas_frankel #mutationtesting 24

TEST THE CODEpublic class Math {public int add(int i1, int i2) { return i1 + i2;}

}

@Testpublic void add_should_add() {int sum = new Math().add(1, 1);Assert.assertEquals(sum, 2);

}

✔Execute Test

@nicolas_frankel #mutationtesting 25

KILLED MUTANTpublic class Math {public int add(int i1, int i2) { return i1 - i2;}

}

@Testpublic void add_should_add() {int sum = new Math().add(1, 1);Assert.assertEquals(sum, 2);

}

✗Execute SAME Test

MUTATION TESTING IN JAVAPIT is a tool for Mutation testingAvailable as• Command-line tool•Ant target•Maven plugin

@nicolas_frankel #mutationtesting 26

MUTATORSMutators are patterns applied to source code to produce mutations

@nicolas_frankel #mutationtesting 27

@nicolas_frankel #mutationtesting 28

PIT MUTATORS SAMPLEName Example source ResultConditionals Boundary > >=Negate Conditionals == !=Remove Conditionals foo == bar trueMath + -Increments foo++ foo--Invert Negatives -foo fooInline Constant static final FOO= 42 static final FOO = 43Return Values return true return falseVoid Method Call System.out.println("foo")Non Void Method Call long t = System.currentTimeMillis() long t = 0Constructor Call Date d = new Date() Date d = null;

IMPORTANT MUTATORSConditionals Boundary• Probably a potential serious

bug smell if (foo > bar)

@nicolas_frankel #mutationtesting 29

IMPORTANT MUTATORSVoid Method Call Assert.checkNotNull()connection.close()

@nicolas_frankel #mutationtesting 30

REMEMBER It’s not because the IDE

generates code safely that it will never change• equals()• hashCode()

@nicolas_frankel #mutationtesting 31

FALSE POSITIVESMutation Testing is not 100% bulletproofMight return false positivesBe cautious!

@nicolas_frankel #mutationtesting 32

@nicolas_frankel #mutationtesting 33

ENOUGH TALK…

Time for DEMO

DRAWBACKSSlowSluggishCrawlingSulkyLethargicetc.

@nicolas_frankel #mutationtesting 38

METRICS (KIND OF)On joda-moneymvn clean test-compilemvn surefire:test• Total time: 2.181 s

mvn pit-test...• Total time: 48.634 s

@nicolas_frankel #mutationtesting 39

WHY SO SLOW?Analyze test codeFor each class under test• For each mutator

• Create mutation• For each mutation

• Run test• Analyze result• Aggregate results

@nicolas_frankel #mutationtesting 40

WORKAROUNDSThis is not acceptable in a normal test runBut there are workarounds

@nicolas_frankel #mutationtesting 41

@nicolas_frankel #mutationtesting 42

SET MUTATORS<configuration> <mutators> <mutator> CONSTRUCTOR_CALLS </mutator> <mutator> NON_VOID_METHOD_CALLS </mutator> </mutators></configuration>

@nicolas_frankel #mutationtesting 43

SET TARGET CLASSES<configuration> <targetClasses> <param>ch.frankel.pit*</param> </targetClasses></configuration>

@nicolas_frankel #mutationtesting 44

SET TARGET TESTS<configuration> <targetTests> <param>ch.frankel.pit*</param> </targetTests></configuration>

@nicolas_frankel #mutationtesting 45

DEPENDENCY DISTANCE

1 2

@nicolas_frankel #mutationtesting 46

LIMIT DEPENDENCY DISTANCE<configuration> <maxDependencyDistance> 4 </maxDependencyDistance></configuration>

@nicolas_frankel #mutationtesting 47

LIMIT NUMBER OF MUTATIONS<configuration> <maxMutationsPerClass> 10 </maxMutationsPerClass></configuration>

USE MUTATIONFILTERFrom extension points

@nicolas_frankel #mutationtesting 48

PIT EXTENSION POINTSMust be packaged in JARHave Implementation-Vendor and Implementation-Title in MANIFEST.MF that match PIT’sSet on the classpathUse Java’s Service Provider feature

@nicolas_frankel #mutationtesting 49

SERVICE PROVIDERInversion of control• Since Java 1.3!

Text file located in META-INF/servicesInterface• Name of the file

Implementation class• Content of the file

@nicolas_frankel #mutationtesting 50

@nicolas_frankel #mutationtesting 51

SAMPLE STRUCTURE

JARch.frankel.lts.Sample

@nicolas_frankel #mutationtesting 52

SERVICE PROVIDER API

@nicolas_frankel #mutationtesting 53

SERVICE PROVIDER SAMPLEServiceLoader<ISample> loaders = ServiceLoader.load(ISample.class);for (ISample sample: loaders) {

// Use the sample}

OUTPUT FORMATSOut-of-the-box•HTML• XML• CSV

@nicolas_frankel #mutationtesting 54

@nicolas_frankel #mutationtesting 55

OUTPUT FORMATS<configuration> <outputFormats> <outputFormat>XML</outputFormat> <outputFormat>HTML</outputFormat> </outputFormats></configuration>

@nicolas_frankel #mutationtesting 56

MUTATION FILTERRemove mutations from the list of available mutations for a class

@nicolas_frankel #mutationtesting 57

@nicolas_frankel #mutationtesting 58

@nicolas_frankel #mutationtesting 59

Time for DEMO

@nicolas_frankel #mutationtesting 60

DON’T BIND TO TEST PHASE!<plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <executions> <execution> <goals> <goal>mutationCoverage</goal> </goals> <phase>test</phase> </execution> </executions></plugin>

@nicolas_frankel #mutationtesting 61

USE SCMMUTATIONCOVERAGEmvn \org.pitest:pitest-maven:scmMutationCoverage \-DtimestampedReports=false

maven-scm-

plugin

must be configured!

@nicolas_frankel #mutationtesting 62

DO USE ON CONTINUOUS INTEGRATION SERVERSmvn \org.pitest:pitest-maven:mutationCoverage \-DtimestampedReports=false

IS MUTATION TESTING THE SILVER BULLET?Sorry, no!It only• Checks the relevance of your

unit tests• Points out potential bugs

@nicolas_frankel #mutationtesting 63

WHAT IT DOESN’T DOValidate the assembled application• Integration Testing

Check the performance• Performance Testing

Look out for display bugs• End-to-end testing

Etc.

@nicolas_frankel #mutationtesting 64

TESTING IS ABOUT ROIDon’t test to achieve 100% coverageTest because it saves money in the long runPrioritize:• Business-critical code• Complex code

@nicolas_frankel #mutationtesting 65

@nicolas_frankel

Q&A@nicolas_frankelhttp://blog.frankel.ch/https://leanpub.com/integrationtest/