Mutant Subsumption GraphsMutation 2014

March 31, 2014

Bob Kurtz, Paul Ammann, Marcio Delamaro, Jeff Offutt, Lin Deng

2

Introduction• In this talk, we will–Define true subsumption, dynamic

subsumption, and static subsumption to model the redundancy between mutants–Develop a graph model to display the

subsumption relationship– Examine via an example how subsumption

relationships behave and evolve

3

Motivation• What exactly is subsumption, anyway?– Lots of prior work – fault hierarchies, subsuming

HOMs, etc.– Can we specify some rules and produce a useful

representation?• What can we do with it once we have it?– Can we select a minimal set of mutants to reduce

testing complexity?– Can we use subsumption as a fitness function for

tasks like evaluating automated program repair candidates?

4

“True” Subsumption• Given a set of mutants M on artifact A, mutant

mi subsumes mutant mj (mi → mj) iff:– Some test kills mi

– All tests that kill mi also kill mj

• “True” subsumption represents the actual relationship between mutants

• We’d like to get this relationship, but in general it is undecidable, so we must approximate it

5

Dynamic Subsumption• Dynamic subsumption approximates true

subsumption using a finite test set T• Given a set of mutants M on artifact A and a

test set T, mutant mi dynamically subsumes mutant mj iff:– Some test in T kills mi

– All tests in T that kill mi also kill mj

• If T contains all possible tests– dynamic subsumption = true subsumption

6

Static Subsumption• Static subsumption approximates true

subsumption using static analysis of mutants rather than test execution

• Given a set of mutants M on artifact A, mutant mi statically subsumes mutant mj iff:– Analysis shows that some test could kill mi

– Analysis shows that all tests that could kill mi also would kill mj

• If we had omniscient analysis, then static subsumption = true subsumption

7

An Informal View

All tests

Tests that kill mj

Tests that kill mi

Tests that kill mk

mi → mj

Graph Model• In the Mutant Subsumption

Graph (MSG) graph model–Nodes represent a maximal

set of indistinguished mutants– Edges represent the

subsumption relationship– Thus, m1 → m2 → m3 is

represented as:88

9

Dynamic Subsumption Graph (DMSG)

m1 m2 m3 m4 m5

t1 t2 t3 t4

T = { t1, t2, t3, t4 }“Indistinguished” mutants

Minimal Mutants• Minimal mutants are

not subsumed by any other mutants

• If we execute a test set that kills all the minimal mutants, then we will kill all the mutants– All other mutants are

redundant!

10

Minimal

Minimal

10

11

DMSG Growthm1 m2 m3 m4

t1 t2 t3 t4

• We can observe the growth of the DMSG as we add tests– Dashed nodes indicate live mutants

T = { t1 } T = { t1, t2 } T = { t1, t2, t3 } T = { t1, t2, t3, t4 }

m1 m2 m3 m4

t1 t2 t3 t4

m1 m2 m3 m4

t1 t2 t3 t4

m1 m2 m3 m4

t1 t2 t3 t4

m1 m2 m3 m4

t1 t2 t3 t4

Subsumption State Model• Mutants change state (with

respect to subsumption relationships) as tests are added.– Live or killed– Distinguished or

indistinguished– Minimal or subsumed

• Only if killed

• These 3 attributes combine to create 8 possible states, but since subsumption is not defined for live mutants, we only care about 6 states

1212

• To explore mutant subsumption graphs in more detail, we selected a small example program

• cal() is a simple Java program of < 20 SLOC– cal() calculates the number of days between

two dates in the same year• Chosen for its well-defined finite input space

– See Ammann and Offutt, Introduction to Software Testing

• We used muJava to generate 173 mutants

The cal() Example

13

14

The cal() Example• Dynamic subsumption requires a test set• We used the Advanced Combinatorial Testing

System (ACTS) to generate a test set– Pairwise combinations of months and year types

(divisible-by-400, divisible-by-100, divisible-by-4, other) generated 90 test cases

– Test set killed 145 mutants, and the remaining 28 were analyzed by hand and determined to be equivalent

The cal() Example• 31 nodes of

indistinguished mutants

• 7 nodes of minimal mutants– Even though muJava

generated 145 non-equivalent mutants, we need to kill only 7 (one from each of these nodes) to ensure that we kill all 145

15

• We can observe the growth of the DMSG as we individually add the 90 “pairwise” tests in random order– Graph shows the number of minimal mutant

nodes (red) and the total number of graph nodes (red + blue)

DMSG Growth for cal()

16

17

DMSG Growth for cal()

cal() DMSG for Different Test Sets90-test “pairwise”

test set

31 nodes7 minimal nodes

6-test “minimal” test set

17 nodes6 minimal nodes

312-test “combinatorial”test set

33 nodes9 minimal nodes

18

19

cal() in C• We implemented the cal() program in C, then used

Proteum to generate mutants– Proteum’s mutation operators are not based on the selective

set of operators, so it generated many more mutants – 891– The same 90 tests killed all but 71 mutants, and those 71

were determined to be equivalent

128 nodesOnly 18 minimal

nodes

20

• May group mutants together where a distinguishing test is missing• May add unsound edges where a contradicting test is missing

Dynamic Approximation

TMSG DMSG

Not Distinguished

Unsound Edge

21

• May group mutants together where unable to solve constraints• If analysis is sound, should never add unsound edges

Static Approximation

SMSGNot Distinguished

Not Distinguished

TMSG

Static Refinement of the DMSG• Can the dynamic

results be refined by static analysis?

• We performed a manual analysis of a small portion of the graph

22

Static Refinement of the DMSG• COI_1 is killed by all tests• AORB_4 is killed whenever

(month2=month1) • AOUI_7 is killed whenever

(month2≠month1) or whenever ((month2=month1)^(day2≠day1))

Tests that kill COI_1(all tests)

Tests that kill AORB_4

Tests that kill AOUI_7

23

Static Refinement of the DMSG• COI_1 is killed by all tests• AORB_4 is killed whenever

(month2=month1) • AORB_2 is killed whenever

((month2=month1)^ ((day2≠day1)≠(day2-day1)))

Tests that kill COI_1(all tests)

Tests that kill AORB_4

24

Tests that kill AORB_2

Static Refinement of the DMSG• AORB_2 is killed whenever

(month2=month1)^((day2-day1)≠(day2/day1))

• AORB_3 is killed whenever(month2=month1)^((day2-day1)≠(day2%day1))

• What is the relationship between these mutants?

All tests / tests that kill COI_1

Tests that kill AORB_2

Tests that kill AORB_3?

25

Static Refinement of the DMSG• Combinations of day1 and day2 that

kill:– both AORB_2 and AORB_3 are GREEN– neither are BLUE– AORB_2 but not AORB_3 are RED– AORB_3 but not AORB_2 are YELLOW

• This one test case breaks AORB_3 → AORB_2

26

Static Refinement of the DMSG• Static analysis removes the unsound edge between

AORB_3 and AORB_2

Refines to

Unsound Edge

27

“Stubborn” Mutants• Yao, Harman, and Jia define

“stubborn” mutants as those non-equivalent mutants which are not killed by a branch-adequate test set– “A Study of Equivalent and

Stubborn Mutation Operators Using Human Analysis of Equivalence”, ICSE 2014

• What’s the relationship between “stubborn” mutants and minimal mutants?

82% kill

63% kill

28

29

Summary• We have developed a succinct definition of

mutant subsumption, as well as two practical approximations, dynamic and static subsumption

• We have developed a graphical notation for subsumption

• We have investigated some properties of subsumption, including growth patterns of the DMSG and a state machine

30

Open Questions• Why are the Java/muJava and C/Proteum

subsumption graphs so different?• Can we analyze static subsumption using Java

Pathfinder and differential symbolic execution (or some other tools/techniques)?

• How do we merge dynamic and static MSGs to get closer to the “true” MSG?

• What is the relationship between minimal and “stubborn” mutants?

• Establishing Theoretical Minimal Sets of Mutants– Paul Ammann, Marcio Delamaro, and Jeff Offutt– Tuesday, 11:30-1:00 in the Burlington Room

Related Information

31

Questions?

rkurtz2@gmu.edu

33

• AORB_13• ROR_16, ROR_20• ROR_17, AORB_12,

AORB_11, AORB_10, AOIS_20, AOIS_22, AOIS_21, AORB_9, AOIS_33, AOIS_34, AOIS_19, LOI_6, LOI_9, ROR_21, ROR_24, ROR_28

• ROR_14, ROR_10• AORB_19• AORB_3• AOIS_46, AOIS_8

Minimal Mutant Operators

34

Minimal Mutant OperatorsOperator #Minimal #Total %Mimimal

AOIS 8 70 11.4%

AOIU 0 7 0%

AORB 7 32 21.9%

COI 0 7 0%

COR 0 4 0%

LOI 2 19 10.5%

ROR 8 34 25.5%