Upload
annibale-panichella
View
90
Download
0
Embed Size (px)
Citation preview
A Guided Genetic Algorithm for Automated Crash Reproduction
MozhanSoltani AnnibalePanichella Arie vanDeursen
Software systems fail!
2
When Debugging, Crash Reproducible Tests are Required!
3
!
4
STAR
Automated Crash Reproduction
JCHARMING MuCrash
Nayrolles etal.,J.Softw.Evol.andProc.,2016
Chen,andKim,TSE2015 Xuanetal.,ESEC/FSE2015
5
STARJCHARMING MuCrash
Automated Crash Reproduction
ASampleStackTrace
6
Automated Crash Reproduction via JCHARMING
JCHARMINGApproach:1) Deriveapartialmodelofthesystem2) Identifythestatementsthattriggerthecrash
Limitation:Requiresexpensivecomputation!
Nayrolles etal.,J.Softw.Evol.andProc.,2016
7
Automated Crash Reproduction via STAR
Approach:1) Computethecrashpreconditions,usingBSE2) Generateatestthatsatisfiesthepreconditions
Limitation:Maynotscaledueto:• Pathexplosion• LimitationsofSMTsolvers
STAR
Chen,andKim,TSE2015
8
Automated Crash Reproduction via MuCrash
Approach:Mutateexistingtestsuntilonereproducesthecrash
Limitation:Lackofeffectivemutationoperators
MuCrash
Xuanetal.,ESEC/FSE2015
9
Crash Reproduction via Search-Based Software Testing!
10
ASampleStackTraceSearchproblem:Findingacrashreproducible unittest!
Metaheuristicoptimization:AGuided GeneticAlgorithm
LeveragingEVSUITE
Crash Reproduction via EvoCrash!
Guided Genetic Algorithm - GGA
11
EvaluateFitness
Selection GuidedCrossover
InitializePopulation
F=0|| T
GuidedMutation
CreatetheNextGeneration
Re-insertion
Guided Genetic Algorithm - GGA
12
EvaluateFitness
Selection GuidedCrossover
InitializePopulation
F=0|| T
GuidedMutation
CreatetheNextGeneration
Re-insertion
Howisthisaguided geneticalgorithm?
Guided Initialization
13
Eachgeneratedtestshallincludeatleastonecalltothetargetfailingmethod!
ASampleStackTrace
Thetargetfailingmethod
Anidealtestcaseleadsto:1. Theexecutionofline1902. TheIllegalArgumentException beingthrown3. Similarstacktracebeinggenerated
14
Fitness Evaluation
ASampleStackTrace
Targetfailingmethod
Guided Genetic Algorithm - GGA
15
EvaluateFitness
Selection Crossover
InitializePopulation
F=0|| T
Mutation
CreatetheNextGeneration
Whatifthereisnocrashreproducibletestinthepopulation?
16
Single-Point Crossover
Parent#1
Parent#2
Offspring#1
Offspring#2
Thetargetcallislost!
17
Parent#1
Parent#2
Offspring#1
Offspring#2
Thetargetcallislost!
What if the target call is lost?
Callincluded?GuidedCrossover
Taketheparents
Takethechildren
18
Mutation
BeforeMutation
Assignanewobject/value
Removeastatement
Addastatement
19
BeforeMutation
Assignanewstatement(constructor)
Removeastatement
Addastatement
What if the target call is lost?
Callincluded?GuidedMutation
MutationAccepted
Is the test useful for debugging?
20
UsefulnessCriterion:
1
34
2
Ausefultestreveals thebug!
5
TheTargetCrashTraceAUsefulCrashReproducibleTest
Chen,N.,&Kim,S.(2015).Star:Stacktracebasedautomaticcrashreproductionviasymbolicexecution.IEEETransactionsonSoftwareEngineering.
Guided Genetic Algorithm - GGA
21
EvaluateFitness
Selection Crossover
InitializePopulation
F=0|| T
Mutation
CreatetheNextGeneration
HowmanycrashescanEvoCrash successfully reproduce?
HowdoesEvoCrash performcomparedtothestate-of-the-artapproachestocrashreproduction?
Empirical Study
Project NumberofCrashes
ApacheCommonsCollections 12
ApacheAnt 20
ApacheLog4j 18
22
ExceptionTypes
NullPointerException
IllegalArgumentException
ArrayIndexOutOfBoundsException
StringIndexOutOfBoundsException
UnsupportedOperationException
IllegalStateException
ConcurrentModificationException
UsefulReproduction
Reproduced
How many crashes can EvoCrashsuccessfully reproduce?
230
5
10
15
20
ACC ANT LOG
• 82%werereproduced,and• ~80%ofthemwereuseful(inaverage)
Results:
20%
28%
34%
90% 71%
88%
24
An Example: ACC-104
Reproductionrate:4/50Challenge: settingboundedFifoBuffer0intherightstate
ThetestbyEvoCrash forACC-104
ThecrashstacktraceforACC-104
How does EvoCrash perform compared to the state-of-the-art approaches?
250
10
20
30
40
50
Total,JCHARMING,EvoCrash Total,STAR,EvoCrash Total,MuCrash,EvoCrash
EvoCrash out-performedthestate-of-the-art!
75% 100%8
50
58%
82%
1266% 83%
ReproductionrateforothertoolsReproductionrateforEvoCrash
Totalnumberofcases
26
EvoCrash Addresses the Limitations!
BackwardSymbolicExecution:• Pathexplosion• LimitationsofSMT solvers
DirectedModelChecking:Expensivecomputations
TestCaseMutation:Limitedmutationoperators
EvoCrash is Available!
27
www.evocrash.org