How good are your tests?Noam Shaish
How do we test?
• Unit testing
• Continues Integration
• Test-drive development (TDD)
• Code coverage
Unit testing
• Prove code works
• Instant regression tests
• Improve code design
Continous Integration
• Automated testing
• Maintain single source repository
• Collect statistics
All tests passedSo f*cking what?
Test that always pass
Code coverage
• Measures line/branches executed during testing
• Common way to estimate testing
We are all covered
• Are we?
Test that gives full coverage
So how good are your tests?
Mutation testing
• Proposed by Richard J. Lipton in 1971
• A way to measure the quality of your tests
Mutation
• A small change in your code
Mutant
• A mutated version of your class
Mutation testing
• Create mutants
• Run tests
• Check outcomes
Outcomes: killed• Mutant killed: Mutant is killed if a test fails, which
proves the mutated code is properly tested
Outcomes: lives
• Mutant lives: Mutant didn't fail any test…
Outcomes: timed out• Mutant timed out: Mutant caused program to loop or
stuck
Outcomes: other
• There there possible outcome but less relevant
Mutation testing with PIT
• http://pitest.org
• Configurable ‘Mutators’
• ASM (byte-code manipulation) is used for manipulation
• No mutation storage
• Reports
Mutators: Condition Boundary
> into >=
< into <=
>= into >
<= into <
Mutators: Negate Conditions
== into !=
!= into ==
>= into <
<= into >
< into >=
> into <=
Mutators: Remove Condition
if ( a == b) { attack(); }
into
if ( true ) { attack(); }
Mutators: Math
+ into -
- into +
* into /
/ into *
% into *
& into |
>> into <<
<< into >>
>>> into <<<
a++ into a—
a— into a++
Mutators:Others
• Replace return values (return a; into return 0;)
• Remove void calls (attack(); is removed)
• Replace value by null
• Some mutators enabled by default and some aren't
Mutation testing speed
• Mutation testing is CPU intensive
• Multiple test runs
• PIT has a lot of speed optimizations
Which test to run?
• PIT uses code coverage to decide which test to run
• A mutation is created only on lines covered by 3 test
Summary
• Mutation testing automatically tests your tests
• Mutation testing can find bugs in your tests
• Code coverage is half the story, and gives a false sense of security
• Mutation testing with PIT is easy
Show me some code
Recommended