Upload
aria-shiers
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Menzies/Hihn - 1
STARSeeking New Frontiers in Cost Modeling
Tim Menzies (WVU)
Jairus Hihn (JPL)
Oussama Elrawas (WVU)
Karen Lum (JPL)
Dan Baker (WVU)
22nd International Forum on COCOMO and Systems/Software Cost Modeling (2007)
Menzies/Hihn - 2
STAR
STAR has three key advancements over traditional methods and even 2cee
– Provides an integrated set of COCOMO models• COCOMO II
• COQUALMO
• COCOMO II Risk (threats) Assessment Model
– Can be used to systematically analyze strategic and tactical policy decisions• Searches for optimal combination of inputs that jointly reduce effort, defect rates
and threats
• Uses constraints to restrict search
– Free, Floating, Fixed
– Can be tuned/calibrated with constraint sets instead of traditional historical data records
• Seek stable conclusions in space of all tunings
• Abduction: View it as an alternative to Bayesian methods based
STAR is an abductive inference engine that applies simulated annealing to a treatment learner (TAR)
Menzies/Hihn - 3
Note• This talk is an extension of
material presented in “The Business Case for Automated Software Engineering”
– IEEE ASE 2007
– Menzies, Elwaras, Hihn Feather, Madachy, Boehm
http://menzies.us/pdf/ 07casease.pdf
Menzies/Hihn - 4
Method
• Stagger across the space of known tunings and inputs (Monte Carlo)
• For N staggers, score N runs by an index we call energy:
– Ef = (effort - minEffort ) / (maxEffort - minEffort)
– De = (defects - minDefects ) / (maxDefects - minDefects)
– Th = (threats - minThreats) / (maxThreats - minThreats)
• Save the one with lowest energy index
normalization0 <= x <= 1
Menzies/Hihn - 5
How to Stagger• Simulated annealing (Von Neuman)
– Pick input ranges and internal values at random
– Do many runs • starting from “boiling hot”
(when you stagger around like a
drunk)
• to “cooler” (No staggering walk straight to your destination)
• Keep track of multiple solutions– Current– New– Best
Sample runs from STAR
(after 500 runs, little improvement)
Bad
Good Best 10%
Menzies/Hihn - 6
Staggering the Tunings Range of effort multipliers (COCOMO)
• COCOMO effort estimation
– Effort multipliers are straight (ish) lines
– when EM = 3 = nominal…• multiple effort by one (I.e. nothing)
– i.e. they pass through the point {3,1};
cplx, data, docupvol, rely, ruse,stor, time
Increase effort
acap, apex, ltex, pcap,pcon, plex,sced,
site,toool
decrease effort
Menzies/Hihn - 7
After staggering,select best things
• Sort all ranges by their “goodness”
– Try the first ranked range,
– Then the first and second,
– Then the first and second and third
– And so on
• Seek the “policy”
– The fewest ranges
– that most reduce • threats,
• effort,
• defects
Bad
Good
22 good ideas
38 not-so- good ideas
Menzies/Hihn - 8
Staggering the inputs : 5 different ways
1. COCOMO II: stagger over entire model input space
“Values” = fixed
“Ranges”= Loose (select within these ranges)
1
2 4
53
Menzies/Hihn - 9
All Flight Groundpmat = 5 pmat=3 ltex=4 acap = 5 ltex=4 plex=4 pcap = 5 plex=4 site=6 apex = 5 automatedanalysis=6 peer reviews=6 ltex = 5 site=6 automatedanalysis=6 prec = 4 apex=5 pcon=5 plex = 5 peer reviews=6 prec=5
site = 6 execution testing and tools = 6
execution testing and tools = 6
peer reviews = 6 pcon=5 apex=5 execution testing and tools = 6 prec=5 acap=5 automatedanalysis=6 acap=5 pcap=5 pcon=5 pcap=5 pmat=2 pcap=4 prec=4
Strategic
Making Strategic Decisions
Full range of modelConstrained by Jairus’ guess
at JPL environment
Menzies/Hihn - 10
Results : OSP
• One advantage of this output display
– If you can’t accept the full policy…
– … you can see what trade-offs arise with some partial policy
• But partial polices cannot include many choices. For example note the missing values:
– Peer reviews < 6
– Execution testing & tools < 6
– Automated analysis < 5
Menzies/Hihn - 11
Results: OSP2• OSP2 was a more constrained
environment as it was a follow-on from OSP and ‘inherited’ the – Team– Development Environment– Design– Etc.
• Again note the missing values:– Peer reviews < 6– Execution testing & tools < 6– Automated analysis < 6
Menzies/Hihn - 12
Results: all experiments• No point in half-hearted defect removal
– Never found in any policy• Peer reviews in 1,2,3,4
• Execution testing & tools in 1,2,3,4
• Automated analysis in 1,2,3,4
• Beware spurious generalities
– X= one of {cocomo or osp or osp2 or flight or ground}
– Y= one of {cocomo or osp or osp2 or flight or ground}
– Not(X = Y)
– X’s best policy is not Y’s best policy
– Exception …
• … Use more automated analysis (model checking, etc)
– Automated analysis = 5 or 6 always in best policy
Menzies/Hihn - 13
Calibrating/Tuning Models
• Traditional Approach Current cost models are tuned to local contexts– LC (Boehm, 1981)
• Tuned to local data using LC– Hard to tell when old data no longer locally relevant
• Suffers from the “large outlier problem”• Row pruning done heuristically
• Traditional approach Next Step– 2CEE (Menzies, Jalali, Baker, Hihn, Lum, 2007)
• Tuned to local data using LC • Tunes and validates every time it runs• Tames outliers primarily with column pruning• Uses Nearest Neighbor for row pruning
– Not all flight software is equal– Culls old data that is no longer relevant
– Both of these approaches require you get more data which may be hard to obtain
• STAR– Current research results suggest that we may be able to estimate almost as well without local
data and LC.• Use est vs actual instead of energy as evaluation metric
– Constrain parameter ranges based on • project being estimated• knowledge of what typically varies in your environment
– Assumes basic COCOMO tunings are ‘representative’• Seems reasonable
Menzies/Hihn - 14
Comparisons
Mre = abs(predicted - actual) /actual
Diff = ∑ mre(lc)/ ∑ mre(star)
“” same at 95% confidence (MWU)
“” same at 99% confidence (MWU)
Very little difference– Half the time: insignificantly different– Otherwise, median diffs = +/- 25%
Why so little difference?– Most influential inputs tightly constrained– Most of the variance comes from uncertainty
in the SLOC,• Not from noise of internal staggering
diff same
diff diff
same same
diff diff
same same
∑ mre(lc) / ∑ mre(star) strategic tactical
ground 66% 63%
all 91% 75%
OSP2 99% 125%
OSP 112% 111%
flight 101% 121%