Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test

Mutation Testing ToolsSoftware Analysis and TestingCu Nguyen Duy (cunduy at fbk dot eu)Alessandro Marchetto (marchetto at fbk dot eu)Paolo Tonella (tonella at fbk dot eu)Mariano Ceccato (ceccato at fbk dot eu)

Academic Year 2011-2012

Friday, November 25, 11

mailto:[email protected]








Introduction

• Mutation testing is expensive

• Compilation, execution, comparison cost

• Not many supported tools

• MuClipse/mujava, Jumblen

• Improving mutant score is time consuming


Mutation testing with MuClipse


MuClipse!An Eclipse plugin for MuJava (http://

cs.gmu.edu/~offutt/mujava/)!Highlight

! Traditional operators! OO operators! Java

!Functionality!Generate mutants!Run tests!Report mutant score


http://cs.gmu.edu/~offutt/mujava/




MuClipse: generate mutants!Eclipse: Run > Run Configurations








MuClipse: run test!Eclipse: Run > Run Configurations








MuClipse: Mutant score output==== Generating original results ========================================--- Result for testAdd : true--- Result for testAdd_2 : true--- Result for testAdd_3 : true--- Result for testAdd_4 : true--- Result for testAdd_5 : true--- Result for testAdd_6 : true--- Result for testAdd_7 : true



***********| Executing Mutants |*********

----------------------------------------------AOIS_11 of 15----------------------------------------------- - testAdd = junit.framework.AssertionFailedError : expected:<5> but was:<6> ------------0 live, 1 killed.






----------------------------------------------AOIS_33 of 15----------------------------------------------- - testAdd = true - testAdd_2 = true - testAdd_3 = true - testAdd_4 = true - testAdd_5 = true - testAdd_6 = true - testAdd_7 = true ------------1 live, 2 killed.



----------------------------------------------AOIS_44 of 15----------------------------------------------- - testAdd = true - testAdd_2 = true - testAdd_3 = true - testAdd_4 = true - testAdd_5 = true - testAdd_6 = true - testAdd_7 = true ------------2 live, 2 killed.






-------------------------------Results for class math.Add-------------------------------Live mutants: 12Killed mutants: 31Mutation Score: 72.0Writing to file...


Exercise 1

!Install MuClipse!http://selab.fbk.eu/swat/projects/muclipse.tgz

!Add java library to Exercise-3-MathOperators!http://selab.fbk.eu/swat/projects/mu-lib.tgz

!Generate mutants! Identify 10 changes in the generated mutants!Hints:

! Use eclipse comparison feature


http://selab.fbk.eu/swat/projects/muclipse.tgz




Exercise 2

!Continue exercise 1!Report the mutant score of the test case:

TestCase1_add_Junit3_a of the project! use only traditional mutation operators

!Improve the mutant score of the test case


Exercise 3

!Compare the coverage of the test case: TestCase1_add_Junit3_a before and after being improved using mutant score

!Hints:!Enable Clover!Execute only the test case and report the

coverage


Mutation testing with Jumble


Installation

๏ Download Jumble from http://jumble.sourceforge.net

๏ Unzip the downloaded file to a folder, namely jumble_1_1_0

๏ Inside the folder eclipseplugin > plugin-export, Copy or Move the folder jumble to your Eclipse plugins folder: Eclipse > Plugins, or Eclipse > dropins > eclipse > plugins

๏ Restart Eclipse

2

Tuesday, November 9, 2010

http://jumble.sourceforge.net

Cu D. Nguyen

Cu D. Nguyen




Run default mutation testing

๏ Condition: Class-To-Be-Mutated and test class have the same package, and test class name = <Class-To-Be-Mutated>Test.java

๏ Steps: 1. Right click on <YourClass>2. Choose Jumble > Jumble Class 3. Look at the Console view to see the mutant

score

3


Run custom mutation testing

๏ Applied when <test class > # <class>Test.java, or in different packages

๏ Follow the step 1 and 2 of the default run (see previous section)

๏ Customize the execution arguments of Jumble to specify your test class. In Eclipse, go to Run > Run Configurations

4


Run custom mutation testing Select Run Jumble Configuration In the Arguments tab, put your <TestClass> after

<YourClass> Run

5


Mutation testing report Mutation points = number of mutants “M FAIL”: mutant was not killed by the current test

class, example: “M FAIL: math.Add:5: += -> -=” : line 5 of class math.Add,

“+=” is mutated to be “-=” Mutant score = (mutant points - “m-failed”) / mutant

points

6


Exercise 1

Apply Mutation testing with Exercise-3-MathOperators

Report the mutant score of the test case: TestCase1_add_Junit3_a of the project

Improve the mutant score of the test case

7


Cu D. Nguyen

Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not

be killed, for example:

!Write more test cases to try to kill other mutants

public class Add { public static int sum (int a, int b){ return a++ +b; } public static double sum (double a, double b){ return a+b; } public static long sum (long a, long b){ return a+b; }}


Mutation analysis!Use traditional mutation operators!Generate mutants of one of your classes

(your choice)!Execute unit test cases that test the chosen

class!Report the mutant score of the test set w.r.t

the chosen class!Improve the mutant score of the test set to

at least 90%


Documents

Mutation Testing Tools - FBK · Mutation testing: improving mutant score!Analyze live mutants!Discard the equivalent mutants which can not be killed, for example:!Write more test