Upload
helena-heath
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Using Bayesian Nets to Predict Software Defects in Arbitrary
Software Lifecycles
Martin NeilAgena Ltd
London, UK
Web: www.agenarisk.comEmail: [email protected]
Telephone: +44 (20) 7404 9722
Effective risk assessment in software projects
• See problems early and take mitigating actions
• Predict when project will be late and/or over-budget
• Decide early enough if project is infeasible
• Predict product quality and the impact of poor reliability
• Know when it makes sense to stop testing
• Support process improvement by determining the impact of good/poor processes
Impediments to effective software risk measurement
• Lack of data• Difficulty of communicating risks• Poor requirements management• Poor coordination with stakeholders and
customers• Difficulty in managing expectations• Difficulty of making progress visible• Lack of applicability of traditional project
management tools
New solutions for software project risk
• Risk maps for high-level project risk assessment
• Risk maps for managing and predicting defects during and after development
• Risk maps for project risk health checks
What the defect prediction risk maps provide
• Accurate predictions of defects at different phases
• Know when to stop testing• Identify where to allocate testing• Minimise the cost of rework• Minimise danger and cost to user• Highlight areas for improvement• Approach fully customisable• Models arbitrary project life-cycles
AgenaRisk modelling and validation
• AgenaRisk template risk maps designed using– Expert judgement– Published data– Company data– Published models
• Risk maps validated on projects at:– QinetiQ– Israel Aircraft Industries– Philips Netherlands and India
What’s special about the approach?
• Structured Method (for quantifying ALL risks)– Based on well proven Bayesian technology– Beyond current statistical & Monte Carlo techniques– Combines subjective judgements with historical data
• Visual Communication (of complex risk issues)– Manages risk through pictures– Useable by non-specialists as well as risk experts– Makes analysis of complex risks easy to communicate
• Robust (for large & complex risk applications)– Enables scalable, reusable & auditable risk models– Integrates easily with databases and spreadsheet systems– Enables developers to build end-user applications
Traditional metrics and statistical approaches
• Software metrics studies provide useful empirical baseline
• Regression-based models are good for predicting the past
• .. but are of little use for true risk assessment
Background: History of Reliability Modeling at Philips - Bangalore
Study of Reliability Models
AID
2000
MODIST
Agena Risk
2001 20032002 2004
Phase-level risk map for defect prediction
Schematic of (single) phase-level risk map for defect prediction
Newfunctionality implemented
Spec & docProcessquality
DevelopmentProcessquality
TestingProcessquality
ReworkProcessquality
DefectsPRE
Totaldefects in
DefectsPOST
Defects found
Defects fixed
Running the defects risk map
Newfunctionality implemented
Spec & docProcessquality
DevelopmentProcessquality
TestingProcessquality
ReworkProcessquality
DefectsPRE
Totaldefects in
DefectsPOST
Defects found
Defects fixed
250
0
Median value 257
High quality development
Newfunctionality implemented
Spec & docProcessquality
DevelopmentProcessquality
TestingProcessquality
ReworkProcessquality
DefectsPRE
Totaldefects in
DefectsPOST
Defects found
Defects fixed
150
0
Very highVery high
Median value 40
Median value 25
… but suppose more than expected defects found in testing
Newfunctionality implemented
Spec & docProcessquality
DevelopmentProcessquality
TestingProcessquality
ReworkProcessquality
DefectsPRE
Totaldefects in
DefectsPOST
Defects found
Defects fixed
150
0
Very highVery high
Median value 204
400
So process assumptions “overruled”by hard facts from QA/test results
How good must your testing and rework be to get ZERO residual defects?
Newfunctionality implemented
Spec & docProcessquality
DevelopmentProcessquality
TestingProcessquality
ReworkProcessquality
DefectsPRE
Totaldefects in
DefectsPOST
Defects found
Defects fixed
150
0
0
Linking phase level risk maps in AgenaRisk
Enables us to modelarbitrary projectsand process
Level of granularity can be chosen to suit the availabledata
AgenaRisk Validation Objectives
• To evaluate AgenaRisk (Method+Toolset) for its
defect prediction capabilities in software projects
• To help Philips decide whether it is a potential
candidate for deployment within software
development centers of Philips CE
• Validation was conducted during Q4/2004 -
Q1/2005
Distribution of selected projects
MM&C (9)28%
TV (16)50%
DVD (7)22%
(actual number of projects shown in brackets)
Results: Actual vs Predicted Defects (Round 1, 2 & 3)
0
500
1000
1500
2000
2500
3000
3500
4000
4500
# o
f de
fect
s
Actual 672 680 928 736 546 209 476 204 196 653 4223 688 261 1597 71 1906 148 228 616 253 53 1768 31 412 373 881 167 109 91 17 5 33
P redicted Rnd 1 666 722 695 759 181 350 551 274 164 467 2115 338 337 350 217 910 271 380 1264 447 146 447 231 430 349 721 246 237 235 200 260 200
P redicted Rnd 2 674 722 986 854 641 254 581 262 259 505 3190 517 407 1147 117 1307 278 485 1419 437 135 976 168 201 180 125
P redicted Rnd 3 2509 444 1514 51 1886 75 355 1272 501 48 1526 52 123 145 116 57 46 55
CEA2UIR
E
CE96DARE
CE11EU2
K
CEK6HDDL
CEC8UIF
T
CED6CHGR
CEL9CEMW
CEK9P SMW
CE39DARE
CEC9CSVC
CEI7FOC
E
CEE8CSVC
CEM4LC04
CEC6UI0
2
CEA540SA
CEH6UIA
2
CEN2DVP 9
CEH2LC3S
CEE3DRCS
CE44A02C
CED7963
S
CEE7UIL
S
CER7DIS
Y
CEO5P CP S
CEI6P I03
CEK2P CSW
CEM9
P SA
CEE7UILS-
CEK5P 2CL
CEQ1RC94
CEP3P EV3
CEN3NAKA
Actuals Vs Predictions
0
200
400
600
800
1000
1200
1400
1600
1800
2000
0 500 1000 1500 2000 2500
Actual defects
pre
dic
ted
def
ects
Validation Summary
• Approach perceived as innovative and promising
• Our evaluation showed excellent results for project sizes between 10 and 90 KLOC
• Initially, projects outside this range were not within the scope of the default model so predictions were less accurate, but accuracy was significantly improved after tailoring standard model
The Road Ahead
• Revised model tailored as Philips-specific for even more accurate results and for projects outside the scope of the default model– New structure already proposed– Replacing the underlying assumptions (where required)– Tailoring questionnaire interface for end-user (with explanations
using organization-specific terms)– Training potential “users” for “what-if-scenarios” and “risk
assessment”• Used AgenaRisk on live projects• Use AgenaRisk for post-release (Field Call Rate) defect
prediction• Incorporate other factors like effort, schedule, etc
Summary
• New approach to defect prediction and quality control based on Bayesian Networks, enables us to– Take account of process and people factors (with support for
process improvement)– Monitor and predict fault injection-detection and rework through
lifecycle– Avoid project failures and improve project communication – With minimal additional data collection requirements and costs
• Validation results show that the AgenaRisk tool and models can be used as is and that simple tailoring can enable even greater power and accuracy for Philips