CUSTOMIZATION SUPPORT FOR CBR-BASED DEFECT PREDICTION
Elham Paikari
Department of Electrical and Computer EngineeringUniversity of Calgary2500 University Drive, NWCalgary, AB, [email protected]
Bo Sun
Department of Computer ScienceUniversity of Calgary2500 University Drive, NWCalgary, AB, [email protected]
Guenther Ruhe
Department of Computer Science & Department of Electrical and Computer EngineeringUniversity of Calgary2500 University Drive, NWCalgary, AB, [email protected]
Emadoddin Livani
Department of Electrical and Computer EngineeringUniversity of Calgary2500 University Drive, NWCalgary, AB, [email protected]
2
Agenda
• Parameters of a CBR Model
• Parameters Instantiation
• Weighting Method SANN
• Frequency Analysis
• Dependency Network and the Customization Support
• Rules Transferability
• Conclusions and Future Work
3
CBR and the Parameters
Prediction Performance of
CBR model
Similarity Function
Number of Nearest
Neighbor Case
Weighting Technique used for
Attributes
Solution Algorithm
Each combination of the parameters :Instantiation of the general CBR-based prediction method
4
Instantiation Parameters of the CBR
Similarity (distance) function
Attribute weighting technique
Number of similar cases used for the prediction
Solution algorithm (how the new case is predicted from the known old ones)
Accuracy measurement of prediction technique
MLR
SANN
5
Sensitivity Analysis Based On NeuralNetwork (SANN)
A1 A2 … An
Mean(A1) Mean(A2) …. Mean(An)
Min(A1) Mean(A2) …. Mean(An)
Max(A1) Mean(A2) …. Mean(An)
Xmin(A1) NN OUTPUTmin(
A1)
Xmax (A1) NN OUTPUTmax
(A1)
∆1= |OUTPUTmin(A1) - OUTPUTmax(A1)|
Dataset
LOC CC ……………
6
AAE ARE AAE ARE AAE ARE AAE ARE AAE AREK=1 K=2 K=3 K=4 K=5
-0.05
0
0.05
0.1
0.15
0.2
Comparing SANN with MLR attribute selection (AAE and ARE)
CM1 KC3 PC1 PC3 PC4
AAE ARE AAE ARE AAE ARE AAE ARE AAE AREK=1 K=2 K=3 K=4 K=5
-0.02
0
0.02
0.04
0.06
0.08
Comparing SANN with un-weighted attribute weighting (AAE and ARE)
CM1 KC3 PC1 PC3 PC4
How different numbers of the nearest neighbors can affect the results?
What is the evaluation result in comparison with existing methods (un-weighted)
What is the evaluation result in comparison with existing methods (MLR)
7
Instantiation Parameters of the CBR
Similarity (distance) function
Attribute weighting technique
Number of similar cases used for the prediction
Solution algorithm (how the new case is predicted from the known old ones)
Accuracy measurement of prediction technique
MLR
SANN
8
Data Repository• PROMISE Repository
• 120 different CBR instantiations were created
and applied to 11 data sets from PROMISE repository
• Characterization of data setsData Set # Modules %Defective Code LOC
CM1 505 9.5 C 17KKC3 458 9.39 Java 8KMC1 9466 0.71 C++ 66KMW1 403 7.69 C 8KPC1 1107 6.87 C 26KPC2 5589 0.41 C 27KMC2 161 32.3 C++ 6KPC3 1563 10.24 C 36KPC4 1458 12.20 C 30KAR3 63 12.25 C 6KAR5 36 22.22 C 2K
9
Is One Instantiation Always the Best?
0 0.2 0.4 0.6 0.8 1 1.20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
MMRE
Pre
d0.1 0.120.140.160.18 0.2 0.220.240.26
0.65
0.7
0.75
0.8
0.85
0.9
MMRE
Pre
d
0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.180.78
0.8
0.82
0.84
0.86
0.88
0.9
0.92
0.94
MMRE
Pre
d
0 0.05 0.1 0.15 0.20.75
0.8
0.85
0.9
0.95
1
MMRE
Pre
d
MW1
CM1
PC1 MC2
KC3 AR5
0.1 0.2 0.3 0.4 0.5 0.6 0.70
0.10.20.30.40.50.60.70.80.9
MMREP
red
0.06
0.08 0.
10.
120.
140.
160.
18 0.2
0.22
0.24
0.7
0.75
0.8
0.85
0.9
MMRE
Pre
d
10
Experimental Design for Frequency Analysis
Dataset
Min(MMRE)
120 differentinstantiation
MMRE Pred(0.25)
1
2
3
120
11differentDatasets
Max(Pred(0.25))
MMRE
1
2
131
Pred(0.25)
1
2
159
Dataset
Min(MMRE)
120 differentinstantiation
MMRE Pred(0.25)
1
2
v
120
Max(Pred(0.25))
Dataset
Min(MMRE)
120 differentinstantiation
MMRE Pred(0.25)
1
2
v
120
Max(Pred(0.25))
11
Frequency Analysis• Frequency of the best performance in single attribute
analysis • Neural network based sensitivity analysis (as the weighting
technique)• Un-weighted average (as the solution algorithm)• Maximum number of nearest neighbors (as the number of
nearest neighbors)
12
Similarity Function
FunctionFrequency
MMRE Pred(0.25)Manhattan 68 79Euclidean 63 80
Total 131 159
Weighting Technique
TechniqueFrequency
MMRE Pred(0.25)Uniform 41 47
MLR 35 52SANN 55 60Total 131 159
Solution Algorithm
AlgorithmFrequency
MMRE Pred(0.25)Un-weighted Average 72 88
Rank-weighted Average 59 71Total 131 159
Number of Nearest Neighbors
FrequencyMMRE Pred(0.25)
1 5 162 3 63 6 84 12 125 13 15
10 18 1915 16 1720 19 1625 19 2230 20 28
Total 131 159
Rule Induction Customization Support
Customization Support Using DNA• Dataset
• Eight attributes defined as condition attributes• Four data set-related attributes:
(NumOfModule),(DefectRatio),(Language),(LOC)• Four CBR-related attributes:
(SimFunc),(WeightingTech),(NumOfNN),(SolutionAlgorithm)
• The decision attributes: Pred(0.25) and MMRE
13
Data set
DNA
Rule Set
CBR model instantiated by
(p1,p2,p3,p4)
New data(a1,a2,a3,a4)
Recommendation f (a1,a2,a3,a4)
(a1,a2,a3,a4)
(p1,p2,p3,p4)
14
Given:NumOfModule = HighDefectRatio = HighLOC = MediumLanguage = JAVA
Question: How to customize a CBR defect prediction model towards achieving high prediction accuracy measured in MMRE?
Justification: Based on the data set characteristics, assumptions of rules 3, 4, 5, 11 and 12 are fulfilled. By comparing the probability distributions of MMRE rule No. 11 is the best in terms of having the highest probability (69.2%) to achieve “Low” MMRE.
Recommendation: Customize CBR model by means of:WeightingTech = SANNNumOfNN ≥ 10SolutionAlgorithm = Rank-weighted Average
Application of DNA Results Generation of the Decision Trees
15
Transferability of Rules across Sets of DataData Set # Modules %Defective Code LOC
CM1 505 9.5 C 17KKC3 458 9.39 Java 8KMC1 9466 0.71 C++ 66KMW1 403 7.69 C 8KPC1 1107 6.87 C 26KPC2 5589 0.41 C 27KMC2 161 32.3 C++ 6KPC3 1563 10.24 C 36KPC4 1458 12.20 C 30KAR3 63 12.25 C 6KAR5 36 22.22 C 2K
9 differentDatasets
MMRE
1
2
131
Pred(0.25)
1
2
159
Dataset
Min(MMRE)
120 differentinstantiation
MMRE Pred(0.25)
1
2
v
120
Max(Pred(0.25))
Dataset
Min(MMRE)
120 differentinstantiation
MMRE Pred(0.25)
1
2
v
120
Max(Pred(0.25))
Rule Induction
Data set
DNA
Rule Set
Customization Support
CBR model instantiated by
(p1,p2,p3,p4)
New data
(a1,a2,a3,a4)Recommendation
f (a1,a2,a3,a4)
•Prediction on Pred(0.25) = 48.33%• Prediction on MMRE = 62.92% •The per-class (High, Medium and Low) are calculated based on the confusion matrix…
16
Validation and Limitations
• Tools used for attribute selection, and modeling tasks• Neural network, regression analysis, CBR, and dependency network
analysis
• Only four parameters of the CBR instantiation• The composition of the training and testing data sets• Another aspect of the analysis undertaken is the definition
of classification intervals for dependency networks, • Two discretization algorithms• Sensitivity analysis
17
Conclusions and Future Work• Starting with 11 data sets from the PROMISE repository
• Calculating the prediction performance of 120 instantiations of the CBR-based
defect prediction model based on the value of the MMRE and Pred(0.25)
• The frequency analysis on the top performances
• Generating the DNA to provide a customization support for a new data set
• The compatibility of rule sets extracted from different contexts
• Enhancement of the validity with inclusion of further data sets
• Comparing the performance against other measures
• Other methods for rule induction
18
References
• Brady, A. and Menzies, T. 2010. Case-based reasoning vs parametric models for software quality optimization. In Proceedings
of the 6thInternational Conference on Predictive Models in Software Engineering, pp. 3:1-3:10.
• Catal, C. and Diri, B. 2009. A systematic review of software fault prediction studies. Expert Systems with Applications, vol. 36
(4), pp. 7346-7354.
• El Emam, K., Benlarbi, S., Goel, N., and Rai, S. N. 2001. Comparing case-based reasoning classifiers for predicting high risk
software components. The Journal of Systems and Software, vol. 55, pp. 301-320.
• Foss, T., Stensrud, E., Kitchenham, B., and Myrtveit , I. 2003. A simulation study of the model evaluation criterion MMRE.
IEEE Transactions on Software Engineering, vol. 29 (11), pp. 985- 995.
• Ganesan, K., Khoshgoftaar, T. M., and Allen, E. B. 2000. Case-based software quality prediction. International Journal of
Software Engineering and Knowledge Engineering, vol. 10(2), pp. 139–152.
• Paikari, E., Richter, M. M., and Ruhe, G. 2010. A comparative study of attribute weighting techniques for software defect
prediction using case-based reasoning. In Proceeding of the 22nd International Conference on Software Engineering and
Knowledge Engineering, pp. 380-386.