Upload
isabelmargarido
View
163
Download
0
Tags:
Embed Size (px)
DESCRIPTION
A Method to Improve the Classification of Requirements Defects - Isabel Lopes Margarido (Faculty of Engineering, University of Porto)
Citation preview
Portugal
A Method to Improve the Classification of Requirements Defects
Isabel Margarido ([email protected])Ph.D. Student Researcher
Faculty of Engineering, University of Porto
06-07-2012, Coimbra
João Pascoal FariaFEUP/INESC
22/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
agenda
introduction
literature review
proposal
assessment
conclusion
33/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
introduction review proposal assessment conclusion
Explained Understood Designed Coded Sold
Installed Billed Supported NeededDocumented
44/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
v
introduction review proposal assessment conclusion
55/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
v
introduction review proposal assessment conclusion
66/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
v
introduction review proposal assessment conclusion
77/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
introduction review proposal assessment conclusion
88/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
introduction review proposal assessment conclusion
99/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)
introduction review proposal assessment conclusion
1010/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
introduction review proposal assessment conclusion
1111/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
Specific Practice (SP) 3.3 “Analyse Requirements”
introduction review proposal assessment conclusion
1212/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
Specific Practice (SP) 3.3 “Analyse Requirements”ensure that they are necessary and sufficient
introduction review proposal assessment conclusion
1313/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
Specific Practice (SP) 3.3 “Analyse Requirements”ensure that they are necessary and sufficient
Verification (VER)
introduction review proposal assessment conclusion
1414/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
Specific Practice (SP) 3.3 “Analyse Requirements”ensure that they are necessary and sufficient
Verification (VER) SP 1.3 “Establish Verification Procedures and Criteria”
introduction review proposal assessment conclusion
1515/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
Specific Practice (SP) 3.3 “Analyse Requirements”ensure that they are necessary and sufficient
Verification (VER) SP 1.3 “Establish Verification Procedures and Criteria” SP 2.1 “Prepare for Peer Reviews”
introduction review proposal assessment conclusion
1616/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
Specific Practice (SP) 3.3 “Analyse Requirements”ensure that they are necessary and sufficient
Verification (VER) SP 1.3 “Establish Verification Procedures and Criteria” SP 2.1 “Prepare for Peer Reviews” SP 2.2 “Conduct Peer Reviews”
introduction review proposal assessment conclusion
1717/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
Specific Practice (SP) 3.3 “Analyse Requirements”ensure that they are necessary and sufficient
Verification (VER) SP 1.3 “Establish Verification Procedures and Criteria” SP 2.1 “Prepare for Peer Reviews” SP 2.2 “Conduct Peer Reviews” SP 2.3 “Analyse Peer Review Data”
introduction review proposal assessment conclusion
1818/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
Specific Practice (SP) 3.3 “Analyse Requirements”ensure that they are necessary and sufficient
Verification (VER) SP 1.3 “Establish Verification Procedures and Criteria” SP 2.1 “Prepare for Peer Reviews” SP 2.2 “Conduct Peer Reviews” SP 2.3 “Analyse Peer Review Data”
maturity level 5
introduction review proposal assessment conclusion
1919/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
related CMMI practices
maturity level 3 (engineering process areas)Requirements Development (RD)
Specific Practice (SP) 3.3 “Analyse Requirements”ensure that they are necessary and sufficient
Verification (VER) SP 1.3 “Establish Verification Procedures and Criteria” SP 2.1 “Prepare for Peer Reviews” SP 2.2 “Conduct Peer Reviews” SP 2.3 “Analyse Peer Review Data”
maturity level 5Causal Analysis and Resolution (CAR)
introduction review proposal assessment conclusion
2020/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
Higher-severity Problem Factors
7 – continually changing requirements
2 – documentation obscure/untrustworthy
3 – changes not adequately documented
4 – lack of traceability
(Chen and Huang, 2009)
introduction review proposal assessment conclusion
10 – lack of consideration for sw quality requirements
motivation
2121/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
(Hamill and Goseva-Popstojanova, 2009)
Higher-severity Problem Factors
7 – continually changing requirements
2 – documentation obscure/untrustworthy
3 – changes not adequately documented
4 – lack of traceability
(Chen and Huang, 2009)
introduction review proposal assessment conclusion
10 – lack of consideration for sw quality requirements
motivation
2222/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
“classifying or grouping problems helps to indentify clusters in which systematic errors are likely to be found” (Card, 1998)
our goal is to define classification scheme for requirements defects that facilitates
identification of more frequent defects with higher impact analysis of root causes preparation of reviews checklists reduction of risks (bad communication, incomplete requirements,
final acceptance difficulties)
introduction review proposal assessment conclusion
motivation
2323/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
“classifying or grouping problems helps to indentify clusters in which systematic errors are likely to be found” (Card, 1998)
our goal is to define classification scheme for requirements defects that facilitates
identification of more frequent defects with higher impact analysis of root causes preparation of reviews checklists reduction of risks (bad communication, incomplete requirements,
final acceptance difficulties)
HP
ODC(Chillarege et al., 1992)
(Grady, 1976)
introduction review proposal assessment conclusion
motivation
2424/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
“classifying or grouping problems helps to indentify clusters in which systematic errors are likely to be found” (Card, 1998)
our goal is to define classification scheme for requirements defects that facilitates
identification of more frequent defects with higher impact analysis of root causes preparation of reviews checklists reduction of risks (bad communication, incomplete requirements,
final acceptance difficulties)
more adequate for other phases than the requirements phase
HP
ODC(Chillarege et al., 1992)
(Grady, 1976)
introduction review proposal assessment conclusion
motivation
2525/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
requirements review
introduction review proposal assessment conclusion
2626/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
quality requirements for classification schemes
introduction review proposal assessment conclusion
2727/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
quality requirements for classification schemes
clearly and meaningfully define attributes
introduction review proposal assessment conclusion
2828/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
quality requirements for classification schemes
complete: every defect is classifiable using the scheme
introduction review proposal assessment conclusion
2929/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
quality requirements for classification schemes
attributes values: • clear and meaningful definition• small number (5-9)• aggregate to reduce ambiguity(Freimut et al., 2005)• unambiguous
introduction review proposal assessment conclusion
3030/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
review literature to compile list of existent classifiers and remove
the ones that do not apply to the phase or document vague and generic classifiers overdetailed duplicates (classifiers with same meaning)
define each classifier and give examples, eliminate ambiguity through definition
phase 1 – assemble classifiers list
introduction review proposal assessment conclusion
3131/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
introduction review proposal assessment conclusion
3232/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
(Bell and Thayer, 1976)
introduction review proposal assessment conclusion
3333/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other
(Bell and Thayer, 1976)
(Basilli and Weiss, 1981)
introduction review proposal assessment conclusion
3434/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
(Bell and Thayer, 1976)
(Basilli and Weiss, 1981)
(Walia and Craver, 2007)
introduction review proposal assessment conclusion
3535/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
(Bell and Thayer, 1976)
(Basilli and Weiss, 1981)
(Walia and Craver, 2007)
(Ackerman et al., 1989)
introduction review proposal assessment conclusion
3636/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm
(Bell and Thayer, 1976)
(Basilli and Weiss, 1981)
(Walia and Craver, 2007)
(Ackerman et al., 1989)
(Chillarege et al., 1992)
introduction review proposal assessment conclusion
3737/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
(Bell and Thayer, 1976)
(Basilli and Weiss, 1981)
(Walia and Craver, 2007)
(Ackerman et al., 1989)
(Chillarege et al., 1992)
(Grady, 1992)
introduction review proposal assessment conclusion
3838/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency
(Bell and Thayer, 1976)
(Basilli and Weiss, 1981)
(Walia and Craver, 2007)
(Ackerman et al., 1989)
(Chillarege et al., 1992)
(Grady, 1992)
(Porter et al., 1992)
introduction review proposal assessment conclusion
3939/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Not Traceable• Unachievable• Intentional Deviation
(Bell and Thayer, 1976)
(Basilli and Weiss, 1981)
(Walia and Craver, 2007)
(Ackerman et al., 1989)
(Chillarege et al., 1992)
(Grady, 1992)
(Porter et al., 1992)
(Hayes et al., 2003/6)
introduction review proposal assessment conclusion
4040/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
type of defect
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Not Traceable• Unachievable• Intentional Deviation• Extraneous
Information(Bell and Thayer, 1976)
(Basilli and Weiss, 1981)
(Walia and Craver, 2007)
(Ackerman et al., 1989)
(Chillarege et al., 1992)
(Grady, 1992)
(Porter et al., 1992)
(Hayes et al., 2003/6)
(Kalinowski et al., 2010)
introduction review proposal assessment conclusion
4141/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Not Traceable• Unachievable• Intentional Deviation• Extraneous
Information
type of defect
introduction review proposal assessment conclusion
4242/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Not in current baseline• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible• New• Changed Requirement• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Not Traceable• Unachievable• Intentional Deviation• Extraneous
Information
change management
type of defect
introduction review proposal assessment conclusion
4343/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible • Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Unachievable• Intentional Deviation• Extraneous
Information
change management
type of defect
introduction review proposal assessment conclusion
4444/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete• Inadequate• Incorrect• Inconsistent• Incompatible • Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced• Other• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate• General
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Unachievable• Intentional Deviation• Extraneous
Information
vague
type of defect
introduction review proposal assessment conclusion
4545/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete • Incorrect• Inconsistent• Incompatible • Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced • Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Unachievable• Intentional Deviation• Extraneous
Information
vague
type of defect
introduction review proposal assessment conclusion
4646/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete • Incorrect• Inconsistent• Incompatible • Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced • Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Unachievable• Intentional Deviation• Extraneous
Information
subsumed
type of defect
introduction review proposal assessment conclusion
4747/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete • Incorrect• Inconsistent • Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced • Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Unachievable• Intentional Deviation• Extraneous
Information
subsumed
type of defect
introduction review proposal assessment conclusion
4848/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete • Incorrect• Inconsistent • Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced • Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Incorrect or Extra Functionality
• Data Type Consistency• Over-specification
• Unachievable• Intentional Deviation• Extraneous
Information
generic
type of defect
introduction review proposal assessment conclusion
4949/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete • Incorrect• Inconsistent • Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced • Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Data Type Consistency• Over-specification
• Unachievable• Intentional Deviation• Extraneous
Information
generic
type of defect
introduction review proposal assessment conclusion
5050/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete • Incorrect• Inconsistent • Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced • Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Missing Interface• Missing Function/Description
• Missing/Incorrect Checking• Missing/Incorrect
Assignment• Missing/Incorrect
Timing/Serialization• Missing/Incorrect
Build/Package/Merge• Missing/Incorrect
Documentation• Missing/Incorrect Algorithm• Missing
Functionality/Feature• Missing Software Interface• Missing Hardware Interface• Missing User Interface• Missing
Requirement/Specification
• Data Type Consistency• Over-specification
• Unachievable• Intentional Deviation• Extraneous
Information
over detailed
type of defect
introduction review proposal assessment conclusion
5151/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete • Incorrect• Inconsistent • Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced • Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Over-specification • Unachievable• Intentional Deviation• Extraneous
Information
over detailed
type of defect
introduction review proposal assessment conclusion
5252/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope• Missing/Omission• Incomplete• Incorrect• Inconsistent• Typos/Clerical• Unclear
• Ambiguity• Wrong Section/Misplaced
• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Over-specification• Unachievable• Intentional Deviation
• Extraneous Information
introduction review proposal assessment conclusion
5353/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope
• Missing/Omission
• Incomplete• Incorrect• Inconsistent• Typos/Clerical• Unclear• Ambiguity
• Wrong Section/Misplaced
• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Over-specification• Unachievable• Intentional Deviation
• Extraneous Information
introduction review proposal assessment conclusion
5454/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope
• Incorrect• Inconsistent• Typos/Clerical• Unclear• Ambiguity
• Wrong Section/Misplaced
• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Over-specification• Unachievable• Intentional Deviation
• Extraneous Information
• Missing or Incomplete
introduction review proposal assessment conclusion
5555/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Out of scope
• Incorrect• Inconsistent• Typos/Clerical• Unclear• Ambiguity
• Wrong Section/Misplaced
• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Over-specification
• Unachievable
• Intentional Deviation
• Extraneous Information
• Missing or Incomplete
introduction review proposal assessment conclusion
5656/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Incorrect• Inconsistent• Typos/Clerical
• Unclear• Ambiguity• Wrong Section/Misplaced
• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Unachievable
• Missing or Incomplete• Not Relevant or Extraneous
introduction review proposal assessment conclusion
5757/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Incorrect• Inconsistent• Typos/Clerical
• Unclear• Ambiguity• Wrong Section/Misplaced• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Unachievable
• Missing or Incomplete• Not Relevant or Extraneous
introduction review proposal assessment conclusion
5858/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Incorrect• Inconsistent• Typos/Clerical
• Wrong Section/Misplaced• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Unachievable
• Missing or Incomplete• Not Relevant or Extraneous• Ambiguous or Unclear
introduction review proposal assessment conclusion
5959/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Incorrect• Inconsistent• Typos/Clerical
• Wrong Section/Misplaced• Infeasible• Untestable/Non-verifiable• Redundant/Duplicate
• Unachievable
• Missing or Incomplete• Not Relevant or Extraneous• Ambiguous or Unclear
introduction review proposal assessment conclusion
6060/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Incorrect• Inconsistent• Typos/Clerical
• Wrong Section/Misplaced• Infeasible• Untestable/Non-
verifiable• Redundant/Duplicate
• Unachievable
• Missing or Incomplete• Not Relevant or Extraneous• Ambiguous or Unclear
introduction review proposal assessment conclusion
6161/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Incorrect• Inconsistent• Typos/Clerical
• Wrong Section/Misplaced• Infeasible• Untestable/Non-
verifiable• Redundant/Duplicate
• Unachievable
• Missing or Incomplete• Not Relevant or Extraneous• Ambiguous or Unclear
introduction review proposal assessment conclusion
6262/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
• Incorrect• Inconsistent• Typos/Clerical
• Wrong Section/Misplaced
• Redundant/Duplicate
• Missing or Incomplete• Not Relevant or Extraneous• Ambiguous or Unclear• Infeasible or Non-verifiable
introduction review proposal assessment conclusion
6363/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra Missing or Incomplete
Incorrect InformationInconsistentAmbiguous or UnclearMisplacedInfeasible or Non-verifiableRedundant or DuplicateTypo or FormattingNot Relevant or Extraneous
introduction review proposal assessment conclusion
6464/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
train pilots understand classifiers, distinguish them, can use them in
practice use examples (apply classification scheme)
conduct experiment same document, team A trained and using new
classifiers, team B not trained nor using new classifiers
analyse results number detected defects number similar classifications to same defect description classifiers systematically confounded
phase 2 – validate classification list
introduction review proposal assessment conclusion
6565/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
adjust definitions, examples, words
repeat phase 2
repeat phase 3 if necessary
phase 3 – readjust classifiers
introduction review proposal assessment conclusion
6666/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
experiments
group A:19 MSc students
group B:6 undergrad students
introduction review proposal assessment conclusion
6767/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
results
H0 - all subjects use the same value to classify the type of a defectH1 - not all subjects use the same value to classify the type of a defect
introduction review proposal assessment conclusion
6868/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
results
H0 - all subjects use the same value to classify the type of a defectH1 - not all subjects use the same value to classify the type of a defect
Fleiss Kappa: moderate (0,46 and 0,44)
introduction review proposal assessment conclusion
6969/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
results
H0 - all subjects use the same value to classify the type of a defectH1 - not all subjects use the same value to classify the type of a defect
Cochran: 0,60 and 0,63 p-value > 0,05
H0 cannot be rejected
Fleiss Kappa: moderate (0,46 and 0,44)
introduction review proposal assessment conclusion
7070/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
results
H0 - all subjects use the same value to classify the type of a defectH1 - not all subjects use the same value to classify the type of a defect
Cochran: 0,60 and 0,63 p-value > 0,05
H0 cannot be rejected
Fleiss Kappa: moderate (0,46 and 0,44)
McNemar: similar results
introduction review proposal assessment conclusion
7171/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
in pilot teams use the least in a full development cylce
control number of defects from requirements detected on
subsequent phases compare with number of defects before using classifiers
list
make adjustments to the classifiers list as necessary (as in phase 3) and repeat phase 4
phase 4 – pilot
introduction review proposal assessment conclusion
7272/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
update tools include tool tips so people can remember definitions include definitions and examples on help
test tools use them in practice to detect bugs and necessary
improvementstrain all teams (phase 2)deploy tools
control number of defects from requirements detected on
subsequent phases
phase 5 – deploy
introduction review proposal assessment conclusion
7373/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
people should be trained in the usage of the defects classification focusing on
distinction of classifiers clarification of definitions practical examples and exercises
avoid choosing a classifier based on its name only definition easily available e.g., as a tool tip in a tool
recommendations
introduction review proposal assessment conclusion
7474/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
summary
introduction review proposal assessment conclusion
7575/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelines
summary
introduction review proposal assessment conclusion
7676/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)
summary
introduction review proposal assessment conclusion
7777/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)organisations should follow proposed methodology when deciding on classification schemes
summary
introduction review proposal assessment conclusion
7878/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)organisations should follow proposed methodology when deciding on classification schemestype of defect should follow quality properties and support
summary
introduction review proposal assessment conclusion
7979/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)organisations should follow proposed methodology when deciding on classification schemestype of defect should follow quality properties and support
CAR
summary
introduction review proposal assessment conclusion
8080/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)organisations should follow proposed methodology when deciding on classification schemestype of defect should follow quality properties and support
CAR creation of checklists
summary
introduction review proposal assessment conclusion
8181/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)organisations should follow proposed methodology when deciding on classification schemestype of defect should follow quality properties and support
CAR creation of checklists prevent risks resulting from requirements defects
summary
introduction review proposal assessment conclusion
8282/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)organisations should follow proposed methodology when deciding on classification schemestype of defect should follow quality properties and support
CAR creation of checklists prevent risks resulting from requirements defects
different people may classify same defect in a different way
summary
introduction review proposal assessment conclusion
8383/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)organisations should follow proposed methodology when deciding on classification schemestype of defect should follow quality properties and support
CAR creation of checklists prevent risks resulting from requirements defects
different people may classify same defect in a different wayproblems in defects classification
summary
introduction review proposal assessment conclusion
8484/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)organisations should follow proposed methodology when deciding on classification schemestype of defect should follow quality properties and support
CAR creation of checklists prevent risks resulting from requirements defects
different people may classify same defect in a different wayproblems in defects classification
different interpretations
summary
introduction review proposal assessment conclusion
8585/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
proposed and assessed a requirements defects classification scheme based on a literature review and quality guidelinesdefect taxonomy needs to support specific organisation needs (Card)organisations should follow proposed methodology when deciding on classification schemestype of defect should follow quality properties and support
CAR creation of checklists prevent risks resulting from requirements defects
different people may classify same defect in a different wayproblems in defects classification
different interpretations misleading analysis
summary
introduction review proposal assessment conclusion
8686/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
questions
8787/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
http://softwareandme.files.wordpress.com/2009/10/3260585819-project_management.jpg – 21-04-2010http://www.screenhog.com/sketch/LightbulbIdea.jpg – 21-04-2010http://igraduatedwhatnow.files.wordpress.com/2009/11/thank_you_small.jpg – 02-05-2010http://cartoontester.blogspot.com/2010/01/big-bugs.html – 01-06-2011http://4.bp.blogspot.com/_eCShgHga-_g/TPA5KRac8_I/AAAAAAAAH3o/mgw0g75jOus/s400/disagreement.jpg – 13-06-2011http://chaospet.com/comics/2008-06-16-90.png – adapted, 13-06-2011http://3.bp.blogspot.com/-_Z2dYcXxMmA/TbLat4c6i_I/AAAAAAAAAnk/KlLdgG-dgtw/s1600/whereamigoing.jpg – adapted, 25-05-20111http://www.veryhappypig.com/blog/results.jpg -06-06-2011http://www.stampa.unibocconi.it/immagini/LA4_economiaq20100603145905.jpg - 06-06-2011
images
8888/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
J.-C. Chen and S.-J. Huang, "An empirical analysis of the impact of software development problem factors on software maintainability," Journal of Systems and Software, vol. 82, pp. 981-992 June 2009.
M. Hamill and G.-P. Katerina, "Common Trends in Software Fault and Failure Data," IEEE Trans. Softw. Eng., vol. 35, pp. 484-496, 2009.
R. Chillarege, et al., "Orthogonal Defect Classification - A Concept for In-Process Measurements," IEEE Transactions on Software Engineering, vol. 18, pp. 943-956, November 1992.
R. B. Grady, Practical software metrics for project management and process improvement: Prentice-Hall, Inc., 1992.
T. E. Bell and T. A. Thayer, "Software requirements: Are they really a problem?," presented at the Proceedings of the 2nd international conference on Software engineering, San Francisco, California, United States, 1976.
V. R. Basili and D. M. Weiss, "Evaluation of a software requirements document by analysis of change data," presented at the Proceedings of the 5th international conference on Software engineering, San Diego, California, United States, 1981.
references
8989/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
G. S. Walia and J. C. Carver, "Development of Requirement Error Taxonomy as a Quality Improvement Approach: A Systematic Literature Review," Department of Computer Science and Engineering, 2007.
A. F. Ackerman, et al., "Software Inspections: An Effective Verification Process," IEEE Software, vol. 6, pp. 31-36, May 1989.
A. A. Porter, et al., "Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment," IEEE Transactions on Software Engineering, vol. 21, pp. 563-575, June 1995.
J. H. Hayes, et al., "Case History of International Space Station Requirement Faults," presented at the Proceedings of the 11th IEEE International Conference on Engineering of Complex Computer Systems, Standford, California, 2006.
M. Kalinowski, et al., "Applying DPPI: A Defect Causal Analysis Approach Using Bayesian Networks," in Product-Focused Software Process Improvement. vol. 6156, M. Ali Babar, et al., Eds., ed: Springer Berlin / Heidelberg, 2010, pp. 92-106.
M. Kalinowski, et al., "Guidance for Efficiently Implementing Defect Causal Analysis," presented at the Brazilian Software Quality Symposium,VII SBSQ Florianópolis, Brazil, 2008.
references
9090/27
Isa
bel L
opes
Mar
garid
o,
6th o
f Ju
ne 2
012
, C
oim
bra
published on IEEE: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?
tp=&arnumber=5974237&contentType=Conference+Publications&queryText%3DLopes+Margarido
author for correspondence: Isabel Lopes Margarido, [email protected] http://paginas.fe.up.pt/~pro09003/
affiliation mui nobre
partially sponsored by:
interested in our research?