A Quality Certification Model for Grid Research Projects the ETICS feasibility Study

www.eu-etics.org

INFSOM-RI-026753

A Quality Certification Model for A Quality Certification Model for Grid Research ProjectsGrid Research Projects

the ETICS feasibility Studythe ETICS feasibility Study

Adriano RippaAdriano Rippa(([email protected]@eng.it)) - Engineering Ingegneria Informatica s.p.a.- Engineering Ingegneria Informatica s.p.a.On behalf of the ETICS ProjectOn behalf of the ETICS Project

ETICS 2nd Review - CERN 15/02/2008 2INFSOM-RI-026753

Summary

• Introduction to QA conceptsIntroduction to QA concepts

• The starting point of the studyThe starting point of the study

• The proposed The proposed Grid Quality Certification ModelGrid Quality Certification Model (GQCM) (GQCM)

• Comparing GQCM and other QA standardsComparing GQCM and other QA standards

• Timeline and pubblicationsTimeline and pubblications

• ConclusionsConclusions


Quality Assurance refers to several concepts

• Quality of the implementation Quality of the implementation process process – High level steps of the software High level steps of the software

production cycle suggesting what production cycle suggesting what the organization must do (not how) the organization must do (not how) to have effective development to have effective development processes that “processes that “may leadmay lead” to good ” to good software.software.

• Quality of the requirements Quality of the requirements management management – Correct collection/management of Correct collection/management of

requirements and relation with the requirements and relation with the customer and stakeholders, to customer and stakeholders, to reduce the percentage of failures reduce the percentage of failures due to misinterpreted requirements.due to misinterpreted requirements.

• Quality of the service Quality of the service – Performances and correctness of Performances and correctness of

the servicethe service

• Finally quality of the software…Finally quality of the software…

QUALITY

Quality of Product

Quality of Process

GQCMGQCMCMMCMM

ITILITILISO – 900xISO – 900x


Some QA Terminology

• According to ISO 9126 documentation we can define:According to ISO 9126 documentation we can define:

• Measure: Measure: the number or category assigned to an the number or category assigned to an attribute of an entity by making a measurement attribute of an entity by making a measurement (sometimes used as synonymous of metric)(sometimes used as synonymous of metric)

• MetricMetric: : The defined method to measure an attribute and The defined method to measure an attribute and the scalethe scale

• MeasurementMeasurement: : The use of a metric to assign a value The use of a metric to assign a value (which may be a number or category) from a scale to (which may be a number or category) from a scale to an attribute of an entity)an attribute of an entity)


Quality is a matter of measure!

• State-of-the-art provide hundreds of metricsState-of-the-art provide hundreds of metrics– The most commonThe most common

– Cyclomatic complexityCyclomatic complexity– Lines of CodeLines of Code– Function PointsFunction Points– Mean Time Between(to) failureMean Time Between(to) failure– Bugs densityBugs density– ……

– Other approaches (Goal Question Metric - GQM) promote user Other approaches (Goal Question Metric - GQM) promote user defined metricsdefined metrics

– Anomalies distributionAnomalies distribution– Effort used to solve anomaliesEffort used to solve anomalies– Cost of not founded anomaliesCost of not founded anomalies– ……


Quality Standards

• There are There are many standardsmany standards to asses the quality of the to asses the quality of the processes of an organization processes of an organization – CMMCMM– ISO familyISO family– ITILITIL– AQAPAQAP

• But QA means initial investments and managing QA But QA means initial investments and managing QA means devote resources to it!means devote resources to it!

• Several studies show that lot of companies (e.g. many Several studies show that lot of companies (e.g. many Small/Medium Enterprises) Small/Medium Enterprises) can’t afford the initial effortcan’t afford the initial effort and don’t recognise the promised and don’t recognise the promised increase of valueincrease of value..


e.g. CMM-I

• CMM-I levels:CMM-I levels:– 1 Initial: Processes are low managed and controlled. There is a first 1 Initial: Processes are low managed and controlled. There is a first

tentative to look at the qualitytentative to look at the quality– 2 Managed: Processes are specific, controlled, applied, replicable2 Managed: Processes are specific, controlled, applied, replicable– 3 Defined: There is homogeneity in the processes between all the 3 Defined: There is homogeneity in the processes between all the

projects. They are defined by the organizationprojects. They are defined by the organization– 4 Quantitatively Managed: Processes are measured and controlled4 Quantitatively Managed: Processes are measured and controlled– 5 Optimising: Focus on the continuous improvement process5 Optimising: Focus on the continuous improvement process

• Only ~70 companies in the world are certified at level 5Only ~70 companies in the world are certified at level 5– 50 of them are in India50 of them are in India

• Only 25% of the companies in the world are level 2 or aboveOnly 25% of the companies in the world are level 2 or above


QA in research project is missing!

• Current quality assurance standards are useful Current quality assurance standards are useful butbut

– They need lot of time to be applied. The organisation They need lot of time to be applied. The organisation need to be structured and certified. need to be structured and certified. What for the short-What for the short-live consortialive consortia??

– They provide onlyThey provide only theoretical guidelines theoretical guidelines which need to which need to be adapted and realised. be adapted and realised. What for the objectivityWhat for the objectivity??

– It’s It’s hard to systematically verify hard to systematically verify goodness of results: goodness of results: managing tools neededmanaging tools needed

– They They need resourcesneed resources to be devoted to to be devoted to

– People needPeople need training training and certification needs and certification needs inspections inspections and time and time to be achievedto be achieved


QA in grids: our proposal

• GQCM is aGQCM is a modelmodel forfor quality assurance that isquality assurance that is

– fully automatablefully automatable in measuring and verifying activities to in measuring and verifying activities to reduce investments and management effort,reduce investments and management effort,

– not subjectivenot subjective, to certify the object not the process nor the , to certify the object not the process nor the organization,organization,

– product orientedproduct oriented, not process oriented,, not process oriented,

• ……easily adoptable within (Grid) Research Projectseasily adoptable within (Grid) Research Projects


GQCM: Preliminary Remarks (1/2)

• GQCM is GQCM is independentindependent from the ETICS tools from the ETICS tools

• Any GQCM implementation needs justAny GQCM implementation needs just “automation” “automation”

• GQCMGQCM has been developed within a Grid project and to has been developed within a Grid project and to asses the quality of grid software research projects but asses the quality of grid software research projects but it it can be used for any software applicationscan be used for any software applications


GQCM: Preliminary Remarks (2/2)

• GQCM GQCM has been developed according to several has been developed according to several standards:standards:

– GQCMGQCM has been described has been described according to several ISO standardsaccording to several ISO standards (e.g. ISO/IEC (e.g. ISO/IEC 25000, 1459825000, 14598))

– GQCMGQCM has been restructured has been restructured according to ISO 25041according to ISO 25041

– Quality attributesQuality attributes has been named using the same terminology has been named using the same terminology as as ISO 9126ISO 9126


GQCM: Structure

• GQCM is structured in GQCM is structured in Evaluation Modules (EM)Evaluation Modules (EM)..

• The set of evaluation techniques are grouped in The set of evaluation techniques are grouped in families. Every family is an EMfamilies. Every family is an EM

• 5 EMs:5 EMs:

– EM: Static analysisEM: Static analysis– EM: Coding styleEM: Coding style– EM: Structural testingEM: Structural testing– EM: Functional testingEM: Functional testing– EM: Standards complianceEM: Standards compliance


Evaluation Modules 1/5

• EM Static analysisEM Static analysis– Quality characteristics:Quality characteristics:

– Reliability – maturityReliability – maturity– Maintainability – analysabilityMaintainability – analysability– Maintainability – changeabilityMaintainability – changeability– Maintainability – testabilityMaintainability – testability

– Technique:Technique: – Static analysis of classes. Statistics on measures are used as Static analysis of classes. Statistics on measures are used as

predictor of quality characteristics. predictor of quality characteristics.

– Applicability:Applicability:– Object oriented programming languages. Object oriented programming languages.

– Input:Input:– source code, each class of the code is analysed source code, each class of the code is analysed



• EM Coding styleEM Coding style– Quality characteristics:Quality characteristics:

– Maintainability – analysabilityMaintainability – analysability

– Technique: Technique: – Static analysis of the source code. Static analysis of the source code.

– Applicability:Applicability:– Most programming languageMost programming language

– Input:Input:– source codesource code



• EM Structural testingEM Structural testing– Quality characteristics:Quality characteristics:

– Functionality – accuracyFunctionality – accuracy– Reliability – maturityReliability – maturity

– Technique:Technique:– Structural testing. The intention of this EM is to test specific classes Structural testing. The intention of this EM is to test specific classes

that are identified by static measures as being statistically more that are identified by static measures as being statistically more likely to have many errors. likely to have many errors.

– Applicability:Applicability:– Object oriented programming languagesObject oriented programming languages

– Input:Input:– source codesource code



• EM Functional testingEM Functional testing– Quality characteristics:Quality characteristics:

– Functionality – accuracyFunctionality – accuracy– Functionality – interoperabilityFunctionality – interoperability– Reliability – maturityReliability – maturity– Portability – adaptabilityPortability – adaptability– Portability - installabilityPortability - installability

– Technique:Technique:– Functional testingFunctional testing

– The purpose of this EM is both to check platform compliance and The purpose of this EM is both to check platform compliance and to check to functional abilities of the software product.to check to functional abilities of the software product.

– Applicability:Applicability:– GeneralGeneral

– Input:Input:– compiled codecompiled code– user documentationuser documentation



• EM Standards complianceEM Standards compliance– Quality characteristics:Quality characteristics:

– Functionality – standards complianceFunctionality – standards compliance

– Technique:Technique:– Functional testing Functional testing

– This EM has a good potential for automating the test. This EM has a good potential for automating the test.

– Applicability:Applicability:– Standard specificStandard specific

– Input:Input:– compiled codecompiled code


GQCM: Final Score

• Final score should be provided according to the following Final score should be provided according to the following schema. The items which should be available for the users are:schema. The items which should be available for the users are:– A A table summarizing the resultstable summarizing the results– A A list of passed and non passed testslist of passed and non passed tests– All the All the important informationimportant information as: as:

– Software product (e.g. name, version, executable code, documentation..)Software product (e.g. name, version, executable code, documentation..)

– Platform (name, version, date)Platform (name, version, date)

– Quality characteristics (name, evaluation result, evaluation module Quality characteristics (name, evaluation result, evaluation module identification)identification)

– Standard compliance (for each standard: name, version, date)Standard compliance (for each standard: name, version, date)

– Identification of evaluation report (organization, report number, date)Identification of evaluation report (organization, report number, date)

– Identification of certification body (organization, contact information)Identification of certification body (organization, contact information)

– Certification data (dates, certification number)Certification data (dates, certification number)

– Electronic signature of certification record Electronic signature of certification record


GQCM: Summarizing Table example

Consolidated eval. result

EM Static

EM Coding

EM structural

EM Functional

EM Std compliance

EM eval. result M M G G G Y

Functionality G G G

Accuracy G G G

Interoperability G G

Compliance Y (Y)

Reliability M M G G

Maturity M M G G

Maintainability M M G

Analyzability M M G

Changeability M M

Testability M M

Portability G G

Adaptability G G

Installability G G

G = GoodM = MediumP = Poor


GQCM: FAQ

• I can’t add any overhead to my projectI can’t add any overhead to my project– This model (and the capability of automate) will reduce the effort This model (and the capability of automate) will reduce the effort

in performing continuous build and test activities (e.g. coverage in performing continuous build and test activities (e.g. coverage tests) on different releasestests) on different releases

• How much costs adopting it?How much costs adopting it?– Nothing, the model will be discussed publicly and the final Nothing, the model will be discussed publicly and the final

version will be released under open license (e.g. Creative version will be released under open license (e.g. Creative Commons)Commons)

– The ETICS framework is provided as a service running on a The ETICS framework is provided as a service running on a dedicated infrastructure, free of charge for Research Project dedicated infrastructure, free of charge for Research Project

• My organisation is certified ISO/CMMi so I…?My organisation is certified ISO/CMMi so I…?– See next slides…See next slides…


GQCM integration in an ISO 9126 certified organization

• Facts:Facts:– ISO 9126 define quality attributes ISO 9126 define quality attributes – GQCM support the measure and evaluation of quality metricsGQCM support the measure and evaluation of quality metrics

• Why are they compatible each other?Why are they compatible each other?– Many ISO9126 aspects can be evaluated by GQCM Many ISO9126 aspects can be evaluated by GQCM GQCM GQCM

apply what ISO 9126 asksapply what ISO 9126 asks– Any tool implementing GQCM and can be integrated as Any tool implementing GQCM and can be integrated as

supporting ISO9126 adoptionsupporting ISO9126 adoption

• SoSo– GQCM can be both an independent quality model and a “tool” to GQCM can be both an independent quality model and a “tool” to

be easily compliant with ISO 9126be easily compliant with ISO 9126


GQCM integration in a CMMi certified organization

• FactsFacts– CMM is process oriented (it gives guidelines).CMM is process oriented (it gives guidelines).– GQCM is product orientedGQCM is product oriented..

• Hypothesis:Hypothesis:• CMM2 CMM2

– asks to “control and value”asks to “control and value”• CMM3 CMM3

– asks to “use internal standards and have a reference model”asks to “use internal standards and have a reference model”• CMM4 CMM4

– asks to “have measured and controlled processes using quantitative and asks to “have measured and controlled processes using quantitative and statistical techniques”statistical techniques”

• CMM5 CMM5 – asks to have an “improvement process and quality based on asks to have an “improvement process and quality based on

measurement”measurement”..

• Our ThesisOur Thesis– GQCM is not in contrast with CMMI and can be integrated in organization GQCM is not in contrast with CMMI and can be integrated in organization

as a tool to support CMMI adoptionas a tool to support CMMI adoption


A first “implementation”

• Currently we implement partially two evaluation Currently we implement partially two evaluation modulesmodules

– EM: Static AnalysisEM: Static Analysis– WMC: looks at the complexity of classesWMC: looks at the complexity of classes

– EM: Coding StyleEM: Coding Style– Only some measurement but we can have a realistic value from it Only some measurement but we can have a realistic value from it

– The first implementation will aim at the programming languages The first implementation will aim at the programming languages Java and Python. […]Java and Python. […]


The timeline

OCTOBEROCTOBER DECEMBERDECEMBER

ECHOGRID/EUChinagridECHOGRID/EUChinagrid

Conference 24-25 AprilConference 24-25 April

Beijing (CHINA)Beijing (CHINA)

MAYMAYFEBRUARYFEBRUARY

OGF 20/EGEE UFOGF 20/EGEE UF

7-11 May7-11 May

Manchester (UK)Manchester (UK)

Belief ConferenceBelief Conference

25-28 June25-28 June

Rio de Janeiro (BRA)Rio de Janeiro (BRA)

OGF 21OGF 21

15-19 Oct15-19 Oct

Seattle (USA)Seattle (USA)

QUALIPSO QUALIPSO ConferenceConference

16-17 Jan 200816-17 Jan 2008

Rome (IT)Rome (IT)

NowNow

EGEE ’07EGEE ’07

1-5 Oct1-5 Oct

Budapest (Hun)Budapest (Hun)

EELA 3° Conference EELA 3° Conference 3-5 Dec 20073-5 Dec 2007

Catania (IT)Catania (IT)


GQCM: publications

• 2 papers submitted and accepted2 papers submitted and accepted

• When/WhereWhen/Where• EELA third conferenceEELA third conference

• 3-5 December 20073-5 December 2007

• Catania/ItalyCatania/Italy

• QUALIPSO first conferenceQUALIPSO first conference• 16-17 January 200816-17 January 2008

• Rome/ItalyRome/Italy

• A A quality oriented conferencequality oriented conference


Conclusions

• GQCM is a certification modelGQCM is a certification model– AutomatableAutomatable– FreeFree– Not in contrast with classical standardsNot in contrast with classical standards– Ready to interact with classical standards Ready to interact with classical standards – Not limited to the ETICS build and test toolNot limited to the ETICS build and test tool– Not limited to research projectsNot limited to research projects– Not limited to grid softwareNot limited to grid software

• GQCM require less human effort to be used because it GQCM require less human effort to be used because it is almost fully automatableis almost fully automatable

• ETICS tool is ready to implement GQCMETICS tool is ready to implement GQCM


Q&A

http://www.eu-etics.org

• Acknowledgements:Acknowledgements:

• Many thanks for his contribution to Many thanks for his contribution to • Andrea ManieriAndrea Manieri

Documents

A Quality Certification Model for Grid Research Projects the ETICS feasibility Study