Upload
cs-ncstate
View
572
Download
3
Embed Size (px)
DESCRIPTION
Tim Menzies, Abdel Salam Sayyad, Hany Ammar
Citation preview
On#the#Value#of#User#Preferences#in#Search4Based#So7ware#Engineering:##
A#Case#Study#in#So7ware#Product#Lines###
Tim#Menzies,##Abdel#Salam#Sayyad,#
Hany#Ammar##
WVU,#USA#Nov’12##
#
Sound#bites#
• The#new#age#of#the#app##
• Stop#Nnkering#with#small#stuff##
• Enough#with#the#usual#suspects:##– NSGA4II,#SPEA2,#etc#
#
• If#preferences#maSer#– Then#the##best#opNmizer#understands#preferences#the#best#
2#
Roadmap#
① Feature(based,SE,② Algorithms,③ IBEA,④ Tree,muta<on,⑤ Conclusion,#
Roadmap#
① Feature(based,SE,② Algorithms,③ IBEA,④ Tree,muta<on,⑤ Conclusion,#
WELCOME,TO,,THE,NEW,WORLDE,,,
5#
The#Nmes,#they#are#a#changing#Olde,worlde:,,
product(based,SE,e.g.#Microso7#office#
New,worlde:,,app(based,SE,,
• E.g.#Apple#app#store#
6#
The#Nmes,#they#are#a#changing#Olde,worlde:,,
product(based,SE,
• Vendors#tried#to#retain#their#user#base#via#some#complete#ecologies##
• One#so7ware#soluNon#for#all#user#needs###(e.g.#Microso7#Office).###
• Large,#complex,#so7ware#pla\orms,##– very#slow#to#change.##
New,worlde:,,app(based,SE,,
• Smart#phones#and#tablet4based#so7ware##
• Users#choosing#many#numbers#of#small#apps#from#different#vendors,##– each#performing#a#specific#
small#task.##
• Vendors#must#quickly#and#conNnually#reconfigure#apps##– To#retain#and#extend#their#
customer#base.#7#
Feature–oriented#domain#analysis#• Feature#maps#=#a#
lightweight#method#for#defining#a#space#of##opNons#
• Product4line#configuraNons#• Defacto#standard#for#
modeling#variability##
8#
Original#FODA#paper#:#2700+#citaNons#
9#
Half#since#2007#
hSp://www.splot4research.org/#
10#200+#models,##plus#an#instance#generator#
LINUX#kernel#=#6000+#features#
11#
86%#declare#constraints#of#some#sort,##Most#features#refer#to#244#other#features.###
Need#for#beSer#automaNon#• Such#complexity#needs#automated#support#
– especially##feature#models##combined#with#• #user#preferences#and#prioriNes,#e.g.#cost#and#reliability.##
12#
Bad##scalability#
Search#for#valid#products:##9#state#of#the#art#theorem#provers##[Pohl,#ASE’11]##
And#these#were#“simple”#models#
Diving##deeper#• Much#prior#work#explored##
Nny#objecNve#spaces##– Two#or#three#objecNves#– Or,#higher#(but#only#for#small#models)#
##
#• So7ware#engineering#=#navigaNng#compeNng#concerns#
1. That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#100%)#2. That#offers#most#features#3. Build#“stuff”#In#least#Nme#4. That#we#have#used#most#before#5. Using#features#with#least#known#defects# 13#
The#usual#
suspects#
Roadmap#
① Feature(based,SE,② Algorithms,③ IBEA,④ Tree,muta<on,⑤ Conclusion,#
MOEA=#MulN4objecNve###evoluNonary#algorithms##############################
• Repeat#Nll#happy#or#exhausted#– SelecNon#(cull#the#herd)#– Cross4over#(the#rude#bit)#– MutaNon#(stochasNc#jiggle)#
15#
Some#MOEA#ApplicaNon#Domain Application Types
Control gas pipeline, pole balancing, missile evasion, pursuit
Design semiconductor layout, aircraft design, keyboard configuration, communication networks
Scheduling manufacturing, facility scheduling, resource allocation
Robotics trajectory planning
Machine Learning designing neural networks, improving classification algorithms, classifier systems
Signal Processing filter design
Game Playing poker, checkers, prisoner’s dilemma
Combinatorial Optimization
set covering, travelling salesman, routing, bin packing, graph colouring and partitioning
MOEA#for#Search4based#SE#
TransformaNon #Cooper,#Ryan,#Schielke,#Subramanian,#FaNregun,#Williams#Requirements## #Bagnall,#Mansouri,#Zhang#Effort#predicNon# #Aguilar4Ruiz,#Burgess,#Dolado,#Lefley,#Shepperd##Management # #Alba,#Antoniol,#Chicano,#Di#Pentam#Greer,#Ruhe#Heap#allocaNon #Cohen,#Kooi,#Srisa4an##Regression#test #Li,#Yoo,#Elbaum,#Rothermel,#WalcoS,#Soffa,#Kampxamer##SOA## # # #Canfora,#Di#Penta,#Esposito,#Villani##Refactoring # #Antoniol,#Briand,#Cinneide,#O’Keeffe,#Merlo,#Seng,#TraS#Test#GeneraNon #Alba,#Binkley,#BoSaci,#Briand,#Chicano,#Clark,#Cohen,#Gutjahr,##
# # # # #Harrold,#Holcombe,#Jones,#Korel,#Pargass,#Reformat,#Roper,#McMinn,## # # # #Michael,#Sthamer,#Tracy,#Tonella,Xanthakis,#Xiao,#Wegener,#Wilkins#
Maintenance # #Antoniol,#Lutz,#Di#Penta,#Madhavi,#Mancoridis,#Mitchell,#Swi7#Model#checking #Alba,#Chicano,#Godefroid#Probing # # #Cohen,#Elbaum##UIOs# # # #Derderian,#Guo,#Hierons#Comprehension #Gold,#Li,#Mahdavi#Protocols# # #Alba,#Clark,#Jacob,#Troya#Component#sel #Baker,#SkalioNs,#Steinhofel,#Yoo#Agent#Oriented #Haas,#Peysakhov,#Sinclair,#Shami,#Mancoridis#
17#
So#study#FODA4#to#learn#how#to#improve#these#
tasks.#
Much#increased#interest##in#Search4based#SE#
18#
The#Pareto#FronNer#• Mutants#=#<D,O>#=#<decisions,#objecNves>#
– E.g.#car#• Decisions:#color#of#car,#number#of#cylinders,#number#of#wheels#• ObjecNves:#miles#per#hour,#cost##(objecNves#may#complete)#
– E.g.#learning#formula#• Decisions:#what#variables#and#constants#to#use#• ObjecNves:#model#simplicity#vs#effecNveness#(objecNves#may#complete)#
• Pareto#fronNer:#select#the#non4dominated#mutants#– X#dominates#Y##
• if#for#all#objecNves,##X#is#never#any#worse##than#Y#• If#for#one#objecNve,#X#beSer#than#Y#
19#
20#
Issues:#1) Spread#2) Hypervolume#3) ComputaNonal#cost:##“g”##generaNons,#M#mutants,#O(gM2)#
Once#you#know#fronNer#•#Select#from#here#•#Ignore#here#
The#usual#suspects:###=#NSGA4II####=#SPEA2#
21#
Combines#N#objecNves##to#one##with#some##weighNng#scheme#
In#this#case,#NSGA4II#gets#more#spread#
Some#details#on#the#usual#suspects#
NSGA(II,,• Is#a#geneNc#algorithm#
#• Non4dominated#sort#
– HeurisNc#way#to#fast#group#mutants#into#bands##
• Crowd#pruning##via#approximate#hypercube#around#each#mutant:#O(Onlogn)#
SPEA2,,• Is#a#geneNc#algorithm#
#• Changes#the#definiNon#of#
“dominaNon”#– SPEA#(version#1)#scored#mutants#by#
how#many#others#they#dominated#– Got#confused#by#overlaps#in#the#
dominaNon#sets##
• SPEA#(version#2):#– Adds#a#“local#density#factor”#to#the#
dominaNon#weight#– Mutants#in#dense#areas#valued##
more###
• SPEA2#beSer#than#SPEA1#
22#
1#2#
3#
2#
Any#number#of#opNmizaNons##to#tradiNonal#GAs#
• The#history#of#MOEAs#in#the#last#15#years#is#– OpNmize#via#hybrid#GA#+#other#
search#method##
• Local#search:##– before#select,#do#a#liSle#simulated#
annealing#on#X%#of#the#populaNon##
• ScaSer#search#(Glover’s#next#generaNon#tabu#search)#– Includes#a#liSle#local#search#
#• ParNcle#swam#opNmizaNon#
– May#do#as#well#as#scaSer#search##(Yin#and#Glover#2007)#
• DifferenNal#evoluNon#(Storn#1996)#– Mutate#by#interpolaNons#between#
exisNng#mutants#– For#x#in#mutants##
####y#=#any1#+#extrapolate(any3#–#any2)#####if#y#dominates#x#then#x#=#y#
#
• Cellular#automata,#• #Ant#colony#opNmizaNon,####• Bayesian#staNsNcs#to#bias#the#
mutaNon,##• Etc##• etc#
23#
Roadmap#
① Feature(based,SE,② Algorithms,③ IBEA,④ Tree,muta<on,⑤ Conclusion,#
Three#groups#of#Algorithms#
25#
Spea2#
Nsga4II#
DE# ScaSer##search#
PSO#
SA#
mocell#Z3#
IBEA#
Indicator4based##evoluNonary#algorithms#
SMT#solvers#
DominaNon#Is#a#binary#concept#
DominaNon#Is#a#conNnuous#concept#
Aggressive#exploraNon#of#preference##space#
IBEA#
26#
• Bo#smarts#anywhere#except#in#the#exploraNon#of#preferences#• I(x1,x2):#
– Least#adjust#objecNve#scores#such#that#x1#dominates#x2#• Repeat#Nll#just#a#few#le7#
– Score#each#instance#x1##buy#summing#its#“I”#to#everyone#else#######
– Sort#all#instances#by#F#– Delete#worst#
• Then,#standard#GA#(cross4over,#mutaNon)#on#the#survivors###
K=#0.05#
Case#studies#
27#
Cross4tree#constraints#
Data#from#hSp://www.splot4research.org/#Algorithms#from#jMetal:#hSp://jmetal.sourceforge.net/##
4#studies:#Bi,#tri,#quad,#five4#objecNves#
28#
So7ware#engineering#=#navigaNng#compeNng#concerns#1. That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#
100%)#2. That#offers#most#features#3. Build#“stuff”#In#least#Nme#4. That#we#have#used#most#before#5. Using#features#with#least##
known#defects###Binary#objecNves#=#1,2#Tri4objecNve#########=#1,2,3#Quad4objecNve####=#1,2,3,4#Five4objecNve######=#1,2,3,4,5#
#HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#
29#
#HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#
30#Comment,1:,all#about#the#same#for#the#24objecNve#problem#
#HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#
31#Comment,2:,E4shop#is#a#nasty#problem:#needs#50M#evals#
#HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#
32#Comment,3:,IBEA#has#no#spread#operators,#but#gets#best#spread#
#HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#
33#Comment,4:,IBEA#has#no#HV#operators,#but#usually#gets#best#HV#
#HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#
34#Comment,5:,All#the#non4IBEA#algorithms#are#very#similar#
#HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#
35#Comment,6:,IBEA#does#much,#much##beSer#on#constraints#
Why#is#this#interesNng?#
Other,MOEAs,• The#usual#suspects#are#widely,#
uncriNcally#used#in#many#MOEA#applicaNons#– E.g.#especially#NSGA4II#and#SPEA2#
• Focused#on#internal#algorithmic#tricks#– Techniques#for#
• #improving#spread##• Improving#HV#• Avoid#overlaps#in#cross4over#of#
dominated#space#• etc#
• And#the#net#effect#of#all#those#differences?#– Not#much#
IBEA,• Rather#stupid#on#those#
internal#tricks#– Just#does#a#ye#olde#crossover#
mutate#GA#– Plus:#aggressive#exploraNon#
of#the#preference#space#• And#the#net#effect#of#all#
those#differences#– BeSer#spreads#– BeSer#HV#– Fewer#constraint#violaNons#
36#
Conclusion:#preference#is#power#
Roadmap#
① Feature(based,SE,② Algorithms,③ IBEA,④ Tree,muta<on,⑤ Conclusion,#
What#about#non4MOEA#soluNons?#
38#
Spea2#
Nsga4II#
DE# ScaSer##search#
PSO#
SA#
mocell#Z3#
IBEA#
Indicator4based##evoluNonary#algorithms#
SMT#solvers#
DominaNon#Is#a#binary#concept#
DominaNon#Is#a#conNnuous#concept#
Aggressive#exploraNon#of#preference##space#
Ethan’s#complaint#
• So7ware#engineer##designs#are#o7en##nested#hierarchical##constraints#
• Ethan#Jackson,#Microso7,##advocate#for#the#Z3#SMT#solver:#– Why#mutate#at#random,#then#check#for#constraint?#
– BeSer#to#drive#the#mutaNons#by#the#constraints?#
39#
Dump#MOEAs?##Move#to#more#logical#forms?#
Pro:,move,to,,say,,SMT,solvers,
• Next#generaNon#of#algorithm#– The#next#big#thing#
• BeSer#to#drive#the#mutaNons#by#the#constraints?#
Con:,stay,with,MOEA,
• ImplementaNon#complexity#less#
• More#tools#• Easier#modificaNon#and#
experimentaNon#• Models#can#be#expressed#
anyway#you#like#
40#
Tree#mutaNon#• Work#in#progress#• Simple#adaptaNon#of#current#MOEAs#for#systems#
of#hierarchical#constraints##
if#rand(0,1)<#mutaNon_probability:############Don't#mutate#if#you're#violaNng#one#of#the#rules:#
# #1)#if#deselecNng#root#feature## #2)#if#selecNng#feature#whose##
##########################parent#is#not#selected## #3)#if#deselecNng#feature#that#
#########################another#selected#feature#requires## #4)#if#group#cardinality#violaNon##else:## #flip#this#bit# # # ###### # #if#selecNng#(turning#on)#a#feature#then## # # #turn#on#children#### # #else#if#deselecNng#(turning#off)#
#######################################feature#then:## # ######### #########turn#off#all#children# 41#
IBEA#stabilizes#70#Nmes#faster#
Tree#mutaNon#preserves##domain#constraints#
42#
So#what##case#for#SMT?#
Roadmap#
① Feature(based,SE,② Algorithms,③ IBEA,④ Tree,muta<on,⑤ Conclusion,#
Sound#bites#• The#new#age#of#the#app#
– In,this,new,worlde:,,use,FODA,(feature(oriented,domain,analysis),
• Stop#Nnkering#with#small#stuff##– Many,MOEAs,have,strikingly,,
similar,performance,
• Enough#with#the#usual#suspects:##– NSGA4II,#SPEA2,#etc#– Too,much,uncri<cal,applica<on,of,these,algorithms,
• If#preferences#maSer#– Then#the##best#opNmizer#understands##preferences#the#best#– IBEA:,aggressive,preference,explora<on,– Tree,muta<on:,respect,your,domain,
44#