Mutation Testing with PIT (Booster 2014, 2014-MAR-13)

Mutation Testing with PIT

Filip van LaenenBooster 20142014-03-13

2 © Computas AS 11.03.14

Agenda

• Basics of mutation testing• Relation to other testing techniques• Example• Mutation testing techniques• Mutation testing tools• Personal experiences and recommendations

Basics ofMutation Testing

Mutation Testing in a Nutshell

Seeking The Summoner @ The Daily WTFhttp://thedailywtf.com/Articles/Seeking-The-Summoner.aspx

Mutation Testing in a Nutshell (cont'd)

• Unit tests guard the source code• But who guards the guardians?

• Do the unit tests cover all source code?• Lines?• Branches?• Paths?

• Do the unit tests test the right things?

Mutation testing tests the tests!

int max(int a, int b) { return (a < b) ? b : a;}

int max(int a, int b) { return (a <= b) ? b : a;}

Does Mutation Testing Work?

In practice, if the software contains a fault, there will usually be a set of

mutants that can only be killed by a test case that also detects that fault.

Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992

Does Mutation Testing Work? (cont'd)

Complex faults are coupled to simple faults in such a way that a test data set

that detects all simple faults in a program will detect most complex faults.

K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995

Does Mutation Testing Work? (cont'd)

• “Generated mutants are similar to real faults.”• Andrews, Briand, Labiche, ICSE 2005

• “Mutation testing is more powerful than statement or branch coverage.”• Walsh, Ph.D. Thesis, State University of New York at

Binghampton, 1985• “Mutation testing is superior to data flow

coverage criteria.”• Frankl, Weiss, Hu, Journal of Systems and Software,

Relation to OtherTesting Techniques

Relation to Other Testing Techniques

• Unit tests• Test-Driven Development (TDD)• Test coverage• Static code analysis• Fuzz testing

Relation to Other Testing Techniques

Is Mutation Testing New?

• R. Lipton, “Fault Diagnosis of Computer Programs,” 1971

• R. Lipton et. al., “Hints on Test Data Selection: Help for the Practicing Programmer,” 1978

• Historical obstacles:• No unit testing• No TDD• Inefficient implementation

• Hence time-consuming• No integration with IDEs

Practical Exampleof Mutation Testing

Code Along…

https://github.com/computas-fvl/booster2014

git clone \https://github.com/computas-fvl/booster2014.git

mvn clean site:site \ org.pitest:pitest-maven:mutationCoverage

Mutation TestingTechniques

Mutation Testing Techniques

• Three aspects:• Mutation injection• Mutation types• Unit test selection per mutant

• Key properties:• Efficiency• Performance

Mutation Injection

• Source code mutation• Binary code mutation• Caveats:

• De-mutation• Compilation errors• Invalid binary code

Mutation Types

• Some mutations never change behaviour• Constants reused by unit tests• Log messages

• Some mutations can change behaviour• Switching between < and ≠• Switching between < and ≤

• Some mutations always change behaviour• Switching between < and ≥

Mutation Types (cont'd)

int max(int a, int b) { return (a < b) ? b : a;}

int max(int a, int b) { return (a <= b) ? b : a;}

int max(int a, int b) { return (a >= b) ? b : a;}

Mutation Types (cont'd)

for (int i = 0; i < 10; i++) …

for (int i = 0; i != 10; i++) …

for (int i = 0; i >= 10; i++) …

Mutation Types Guaranteed to Change Behaviour

• Negation of the comparison• Switching between = and ≠• Switching between < and ≥• Switching between > and ≤

• Negation of boolean conditions• Adding a ¬, ! or ~

• Shortcutting boolean conditions• Replacement with True or False

Unit Test Selection

• Goal: find the unit test that “kills” the mutant• Selection aids:

• Hints• Name/package matching• Code coverage tools• Automatic learning• Other heuristics

Unit Test Selection (cont'd)

• System:• 50 classes• 20 unit tests per class• 1 ms per unit test• Unit testing time: 50 × 20 × 1ms = 1s

• 10 mutants per class:• Brute-force: 10 × 50 × 1s = 6m 20s• Educated: 10 × 50 × 20 × 1ms = 10s

Unit Test Selection (cont'd)

• System:• 500 classes• 20 unit tests per class• 1 ms per unit test• Unit testing time: 500 × 20 × 1ms = 10s

• 10 mutants per class:• Brute-force: 10 × 500 × 10s = 13h 53m 20s• Educated: 10 × 500 × 20 × 1ms = 1m 40s

Complexity

• f: Number of function points• φ: Number of function points per class, ≥ 1• τ: Number of unit tests per function point, ≥ 1• μ: Number of mutants per function point, ≥ 1

• Brute-force: (f × τ) × (f × μ) = τ × μ × f²• Educated force: τ × μ × φ × f• Ideal: τ × μ × f

Complexity (cont'd)

• c: Number of classes• φ: Number of function points per class, ≥ 1• τ: Number of unit tests per function point, ≥ 1• μ: Number of mutants per function point, ≥ 1

• Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c²• Educated force: τ × μ × φ² × c• Ideal: τ × μ × φ × c

for (int i = 0; i < 10; i++) …

for (int i = 0; i < 10; i--) …

Infinite Loops

• Terminate mutants that take too long to run• What's “too long”?

• Ruins the performance• Can be hard to predict

Mutation Testing with PIT (Booster 2014, 2014-MAR-13)

Technology

Gene Mutations. Mutation Vocabulary Gene Mutation Point Mutation – Substitution – Silent – Missense – Nonsense Frameshift – Insertion – Deletion Chromosomal

2015 ENDOCRINOLOGY - myCMEmedia.mycme.com/documents/198/2016_endocrinology_49400.pdf · • Apoplexy • Genetic diseases - pit-1 mutation PANCE/PANRE Review Course Etiology of Hypopituitarism

Mutation Testing: A comparison of mutation selection methods

Z21 Booster Adapter L’adaptateur-booster Z21L’adaptateur-booster Z21® L’adaptateur-booster Z21® permet d’utiliser jusqu’à 15 boosters/amplificateurs grâce à une interface

Stickler syndrome, Gene Mutation: COL11A1 Stickler syndrome, Gene Mutation: COL11A2 Stickler syndrome, Gene Mutation: COL2A1

Mutation Detection from DNA Sequence Traces with Mutation ... Mutation Detection... · Mutation Detection from DNA Sequence Traces with Mutation Surveyor and Mutation Explorer Software

Mutation Detection from DNA Sequence Traces with … Mutation Detection... · Mutation Detection from DNA Sequence Traces with Mutation Surveyor and Mutation Explorer ... The green

SERVIÇOS DE ABASTECIMENTO DE ÁGUA DA SEDE MUNICIPAL … · Bom Pastor, Booster Bom Retiro, Booster Cojan, Booster Santa Clara e Booster Sol Nascente. O Booster Bom Retiro ainda

Mutation sociétale mutation psychosociétale

Series 6800 booster Series 6900 booster - phoenixpumps.com

MUTANTS KILLER - PIT: state of the art of mutation testing system

a+ --> a- mutation (forward mutation)

DNA repair. spontaneous vs. induced mutation gametic vs. somatic mutation lethal or conditional mutation Classification of the mutation

The Option Pit Method Option Pit Option Pit Boot Camp The Option Pit Method For trading options

Plus / Delta - Lean Construction Institute€¦ · 02/05/2018 · The New Model FA PIT 1 FA PIT 4 FA PIT 3 FA PIT 5 FA PIT 2 PLS PIT Offsite Fabrication PIT Logistics PIT LPS PIT

DNA Mutation and Methylation Quantification from … Mutation and Methylation Quantification from . Sanger Sequencing Traces with Mutation Surveyor Software. Introduction. Mutation,

A+ --> a- mutation (forward mutation) a- --> a+ reverse mutation (reversion)

Advanced Java Testing · Mutation Testing • Using PIT and ... advanced-java-testing-1.1 Created Date: 2/4/2018 1:39:47 PM

Pipeline/Transportation Pumps - Pumps! That's what Vertical in-line • Submersible motor • Multistage vertical wet pit • Single and multistage vertical canned tank farm booster

Genetics. Mutations Point mutation –Missense mutation –Nonsense mutation Frameshift mutation Trinucleotide repeat mutation