How good are your tests?

How good are your tests?Noam Shaish

How do we test?

• Unit testing

• Continues Integration

• Test-drive development (TDD)

• Code coverage

Unit testing

• Prove code works

• Instant regression tests

• Improve code design

Continous Integration

• Automated testing

• Maintain single source repository

• Collect statistics

All tests passedSo f*cking what?

Test that always pass

Code coverage

• Measures line/branches executed during testing

• Common way to estimate testing

We are all covered

• Are we?

Test that gives full coverage

So how good are your tests?

Mutation testing

• Proposed by Richard J. Lipton in 1971

• A way to measure the quality of your tests

Mutation

• A small change in your code

Mutant

• A mutated version of your class

Mutation testing

• Create mutants

• Run tests

• Check outcomes

Outcomes: killed• Mutant killed: Mutant is killed if a test fails, which

proves the mutated code is properly tested

Outcomes: lives

• Mutant lives: Mutant didn't fail any test…

Outcomes: timed out• Mutant timed out: Mutant caused program to loop or

stuck

Outcomes: other

• There there possible outcome but less relevant

Mutation testing with PIT

• http://pitest.org

• Configurable ‘Mutators’

• ASM (byte-code manipulation) is used for manipulation

• No mutation storage

• Reports

Mutators: Condition Boundary

> into >=

< into <=

>= into >

<= into <

Mutators: Negate Conditions

== into !=

!= into ==

>= into <

<= into >

< into >=

> into <=

Mutators: Remove Condition

if ( a == b) { attack(); }

into

if ( true ) { attack(); }

Mutators: Math

+ into -

- into +

* into /

/ into *

% into *

& into |

>> into <<

<< into >>

>>> into <<<

a++ into a—

a— into a++

Mutators:Others

• Replace return values (return a; into return 0;)

• Remove void calls (attack(); is removed)

• Replace value by null

• Some mutators enabled by default and some aren't

Mutation testing speed

• Mutation testing is CPU intensive

• Multiple test runs

• PIT has a lot of speed optimizations

Which test to run?

• PIT uses code coverage to decide which test to run

• A mutation is created only on lines covered by 3 test

Summary

• Mutation testing automatically tests your tests

• Mutation testing can find bugs in your tests

• Code coverage is half the story, and gives a false sense of security

• Mutation testing with PIT is easy

Show me some code