Download ppt - Toward a Quality Certification Process for Grid Research Projects: an ETICS feasibility Study

www.eu-etics.org

INFSOM-RI-026753

Toward a Quality Certification Toward a Quality Certification Process for Grid Research Process for Grid Research Projects: an ETICS feasibility Projects: an ETICS feasibility StudyStudyAndrea ManieriAndrea ManieriAdriano RippaAdriano Rippa - Engineering Ingegneria Informatica s.p.a.- Engineering Ingegneria Informatica s.p.a.On behalf of ETICS ProjectOn behalf of ETICS Project

Open Grid Forum 20 - Manchester 07-11 May 2007 2INFSOM-RI-026753

Why bother?

• R&D project results are often interesting, but:R&D project results are often interesting, but:– They aim only at proving conceptsThey aim only at proving concepts– They suffer of deviation from plans due to their investigation They suffer of deviation from plans due to their investigation

naturenature– Developers (sometimes) don’t pay enough time to testing, Developers (sometimes) don’t pay enough time to testing,

documentation, etcdocumentation, etc– Developers are not so skilled in software engineeringDevelopers are not so skilled in software engineering

• After the end of a projects:After the end of a projects:– None (out of the consortium) is willing to use such swNone (out of the consortium) is willing to use such sw– The software itself is difficult to maintain and improveThe software itself is difficult to maintain and improve

• So, many times, effort and brilliant results are lost!So, many times, effort and brilliant results are lost!


That are only dreams?

• Software components fully tested and documentedSoftware components fully tested and documented

• Maintenance of the know-how for build, deploy and run Maintenance of the know-how for build, deploy and run the softwarethe software

• Software components that build on different platformSoftware components that build on different platform

• Software bundle with all the right dependencies and Software bundle with all the right dependencies and the right libraries to be used by other developersthe right libraries to be used by other developers


Quality assurance…

• QA is a matter of:QA is a matter of:– In which way sw has been created (procedural)In which way sw has been created (procedural)– How sw is written (structural)How sw is written (structural)– How sw perform (behavioural)How sw perform (behavioural)

• Currently we have:Currently we have:– QA standards (I.e. CMMi, ISO, ITIL, …)QA standards (I.e. CMMi, ISO, ITIL, …)– SW TestingSW Testing– So-called Quality of Service (at least partially) So-called Quality of Service (at least partially)


What’s missing, then?

• A systematic approach to A systematic approach to – Build software with all the correct dependenciesBuild software with all the correct dependencies– Extensively Test each componentsExtensively Test each components

• That allows gathering some measuresThat allows gathering some measures– Related to the build performanceRelated to the build performance– Reporting tests results and Reporting tests results and – Other characteristics of the codeOther characteristics of the code

• With which derive the overall quality of the software With which derive the overall quality of the software produced!produced!

• DISCLAIMER: this study will focus, initially, on DISCLAIMER: this study will focus, initially, on Research ProjectsResearch Projects


Yet another Quality Assurance model?

• Current models are useful but:Current models are useful but:– are process oriented are process oriented – provide theoretical guidelines on the organizationprovide theoretical guidelines on the organization

• TheThe problemproblem::– Guidelines can be applied but it’s hard to Guidelines can be applied but it’s hard to

systematically verify goodness of resultssystematically verify goodness of results


ETICS feasibility study

• To study and propose aTo study and propose a grid software grid software Certification model that canCertification model that can – Be automated andBe automated and– Be not subjectiveBe not subjective– Used within Research ProjectsUsed within Research Projects

• ETICS will analyse the feasibility of a ETICS will analyse the feasibility of a Certification for grid software:Certification for grid software:– To support the safe adoption of software generated by To support the safe adoption of software generated by

research projects research projects– To help researchers in produce good grid applicationsTo help researchers in produce good grid applications


The timeline

FIRST BETA FIRST BETA

VERSIONVERSION

FINAL BETA FINAL BETA

VERSIONVERSION

OCTOBEROCTOBER

FIRST OFFICIALFIRST OFFICIAL

VERSIONVERSION

DECEMBERDECEMBER

ECHOGRID/EUChinagridECHOGRID/EUChinagrid

Conf. Beijing 24-25 AprilConf. Beijing 24-25 April

MAYMAYFEBRUARYFEBRUARY

OGF 20/EGEE UFOGF 20/EGEE UF

7-11 May7-11 May

Belief ConferenceBelief Conference

RIO – BRASIL 25-28 JuneRIO – BRASIL 25-28 June

OGF 21OGF 21

15-19 Oct15-19 Oct

QUALIPSO ConferenceQUALIPSO Conference

Dec 2007Dec 2007

NowNow


ETICS: the project

• Started in January 2006Started in January 2006• Duration: 2 yearsDuration: 2 years• Funded by the European Commission: ~1.4 MFunded by the European Commission: ~1.4 M€€• 5 partners5 partners• 2 early adopter projects2 early adopter projects

– EGEE/gLiteEGEE/gLite– DILIGENT/gCubeDILIGENT/gCube

• Spin-off of the integration and test activities of EGEE-I Spin-off of the integration and test activities of EGEE-I for gLite middlewarefor gLite middleware

• ETICS Service: ETICS Service: – Improve the way Grid software is built and testedImprove the way Grid software is built and tested– Increase sharing in the Grid communitiesIncrease sharing in the Grid communities


The ETICS Partners

Build system, software

configuration, service infrastructure,

dissemination, EGEE, gLite, project coord.

Software configuration, service infrastructure, dissemination

Web portals and tools, quality process, dissemination, DILIGENT

Test methods and metrics, unit testing tools, EBIT

The Condor batch system, distributed testing tools, service infrastructure, NMI


ETICS is a framework…

• based on a based on a grid infrastructuregrid infrastructure where it’s where it’s possible to:possible to:– Run any scripts over the code (e.g ant )Run any scripts over the code (e.g ant )– Run the unit or component testingRun the unit or component testing– Run a script to deploy one or more services Run a script to deploy one or more services – Run a test to verify the conformity of some API or Run a test to verify the conformity of some API or

guidelines (e.g IPV6 compliance)guidelines (e.g IPV6 compliance)– ……– Then collect all these data to Then collect all these data to

– perform (trend) analysisperform (trend) analysis– gather measures and calculate metrics…gather measures and calculate metrics…


Where ETICS tools apply

DESIGN and CODE

WRITING

CODE

BUILD and

INTEGRATION

TESTINGincluding

COMPLIANCE

EXECUTION

ETICS context


A Land of nobody?

• WRONG!WRONG!

• Many projects on-going (i.e. QUALIPSO for the quality Many projects on-going (i.e. QUALIPSO for the quality in the open source environment..)in the open source environment..)

• Plenty of build tools (GUMP, Maven, *-forge, Savanna, Plenty of build tools (GUMP, Maven, *-forge, Savanna, etc)etc)– Often open source…Often open source…

• Several testing tools (Pulse, Visual Build, etc)Several testing tools (Pulse, Visual Build, etc)– Often commercial…Often commercial…


Terminology

• Measure: Measure: a value (number or category) assigned to an a value (number or category) assigned to an attribute of an entity (e.g. software component)attribute of an entity (e.g. software component)

• MeasurementMeasurement: : The act or process of assigning a The act or process of assigning a number or category to an entity to describe an attribute number or category to an entity to describe an attribute of that entity. of that entity.

• MetricMetric: : A function whose inputs are software measures A function whose inputs are software measures and whose output is a single numerical value that can and whose output is a single numerical value that can be interpreted as the degree to which software be interpreted as the degree to which software possesses a given attribute that affects its quality.possesses a given attribute that affects its quality.

• PerspectivePerspective: A set of homogenous software attributes : A set of homogenous software attributes that lead to some quality assumptionsthat lead to some quality assumptions


Grid Quality Assurance Certification Model (GQACM ): a 3D QA Model

• Let’s suppose A, B, C to be Let’s suppose A, B, C to be different perspectivesdifferent perspectives

• Common QA Standards are Common QA Standards are Waterfall-likeWaterfall-like

• GQACM uses 3 independent GQACM uses 3 independent perspectives to provide perspectives to provide client-friendly resultsclient-friendly results– Users can be interested in Users can be interested in

some attributes and less in some attributes and less in others..others..

– So-made perspectives can So-made perspectives can have sublevels…have sublevels…

AA

BB

CC

AA

BB

CC


GQACM: Preliminary remarks

• GQACM is GQACM is independentindependent from the ETICS tools from the ETICS tools

• However, any GQACM implementation However, any GQACM implementation must use must use automationautomation

– – the most useful characteristic of the model is that it the most useful characteristic of the model is that it can be automatically applied!can be automatically applied!

• GQACM has three perspectives (I.e. independent points GQACM has three perspectives (I.e. independent points of view) to derive the quality degree of a software :of view) to derive the quality degree of a software :– CodeCode– PlatformsPlatforms– StandardsStandards

CC


The GQACM: Perspectives

• Code Perspective Code Perspective – Usual metrics depending on programming paradigmUsual metrics depending on programming paradigm– … … to qualify the degree of goodness of the code to qualify the degree of goodness of the code

(static analysis)(static analysis)• Platform PerspectivePlatform Perspective

– The type and quantity of different platform (I.e. The type and quantity of different platform (I.e. family of platforms) where software built correctly family of platforms) where software built correctly and can be automatic deployedand can be automatic deployed

• Standard PerspectiveStandard Perspective– Software tested for compliance to applicable Software tested for compliance to applicable

standardsstandards (e.g. BES, JSDL, SRM, WS-I, IPV6)(e.g. BES, JSDL, SRM, WS-I, IPV6)

• These perspectives areThese perspectives are independent each independent each otherother


Perspectives composition

• Code PerspectiveCode Perspective– Coding ConventionsCoding Conventions– Code CommentingCode Commenting– Other QA metrics related to the corresponding Other QA metrics related to the corresponding

programming languageprogramming language

• Platform perspective Platform perspective – The set of platforms (declared, most used, all The set of platforms (declared, most used, all

available?)available?)

• Standard perspectiveStandard perspective– The set of (available) standards against the The set of (available) standards against the

compliance is proven.compliance is proven.


Test execution metrics

• C1 (method level): C1 (method level): – if C is the cyclomatic complexity (of the method) and T is the if C is the cyclomatic complexity (of the method) and T is the

number of executed tests then RM1 = C/Tnumber of executed tests then RM1 = C/T

• C2 (class level): C2 (class level): – if CK RFC is the the number of methods which can be invoked if CK RFC is the the number of methods which can be invoked

from an instance of the class target to answer a messagefrom an instance of the class target to answer a message ) ) and and T is the number of executed tests on it then RM1 = RFC/T.T is the number of executed tests on it then RM1 = RFC/T.

• C3 (class level): C3 (class level): – if E is the number of events which interact with the class and T is if E is the number of events which interact with the class and T is

the number of executed tests then RM4 = E/Tthe number of executed tests then RM4 = E/T

• C4 (class level): C4 (class level): – if CK NOC is the Number Of Children and T is the number of if CK NOC is the Number Of Children and T is the number of

executed tests on the class then RM8 = NOC/Texecuted tests on the class then RM8 = NOC/T


Code comment metrics

• C5: C5: – if CLOC are lines of comments, then RM5 = if CLOC are lines of comments, then RM5 =

CLOC/LOCCLOC/LOC

• C6: C6: – if F is the number of functions and C is the number of if F is the number of functions and C is the number of

comment paragraphs then RM6 = F/Ccomment paragraphs then RM6 = F/C

• C7:C7:– Automatic documentation programs use: e.g. doxygenAutomatic documentation programs use: e.g. doxygen


Coding convention metrics 1/2

• Only one class definition per header fileOnly one class definition per header file• Filename = ClassnameFilename = Classname• Filename in lowercaseFilename in lowercase• No cycles in “include” operationsNo cycles in “include” operations• No C instruction (in C++ programming)No C instruction (in C++ programming)• Coherent Programming ConventionCoherent Programming Convention

– Curly Brackets positioningCurly Brackets positioning– First character in variables namesFirst character in variables names– Lower case in macros namesLower case in macros names– Coherent use of new linesCoherent use of new lines– Semicolons only at the end of a lineSemicolons only at the end of a line


Coding convention metrics 2/2

– No Never used functionsNo Never used functions– No Not reachable code pathsNo Not reachable code paths– No Not defined prototypesNo Not defined prototypes– No Never used constantsNo Never used constants– No Never used macrosNo Never used macros– No Never used types definitionsNo Never used types definitions– No Never used variablesNo Never used variables– Too frequent “breaks” useToo frequent “breaks” use– No absence of variables de-alloactionNo absence of variables de-alloaction– No Null pointersNo Null pointers– Management of one instruction long “if” or one instruction long Management of one instruction long “if” or one instruction long

“cycle”“cycle”– Shared resources managementShared resources management


other metrics

• Errors correction abilityErrors correction ability– C7: errors trend between a test and another after C7: errors trend between a test and another after

correctionscorrections

• Other software related metricsOther software related metrics– (e.g. CK Metrics for OO software)(e.g. CK Metrics for OO software)


GQACM: Users benefits

• Users will: Users will: – Have a clear picture about the quality and … Have a clear picture about the quality and … – ……will be able to select components and software of will be able to select components and software of

certified quality levelcertified quality level

• Concretely for different roles:Concretely for different roles:– DeveloperDeveloper will be able to certify its software will be able to certify its software– IntegratorIntegrator will be safe in compose new software will be safe in compose new software

starting from existing certified components of the starting from existing certified components of the appropriate quality degreeappropriate quality degree

– Final UserFinal User will be sure to use certified software will be sure to use certified software


Yes, but!

• I’m not able to add any overhead to my projectI’m not able to add any overhead to my project– This model (and the capability of automate) will reduce the effort This model (and the capability of automate) will reduce the effort

in performing continuous build and tests (e.g. coverage tests) on in performing continuous build and tests (e.g. coverage tests) on different releasesdifferent releases

• How much costs adopting it?How much costs adopting it?– Nothing, the model will be discussed publicly and the final Nothing, the model will be discussed publicly and the final

version will be released under open license (e.g. Creative version will be released under open license (e.g. Creative Commons)Commons)

– The ETICS framework is provided as a service running on a The ETICS framework is provided as a service running on a dedicated infrastructure, free of charge for Research Project dedicated infrastructure, free of charge for Research Project (until dec 2007)(until dec 2007)

• My organisation is certified ISO/CMMi?My organisation is certified ISO/CMMi?– See next slides…See next slides…


ISO 9126

• is an international standard for the evaluation is an international standard for the evaluation of software. of software.

• is divided into four parts which addressis divided into four parts which address– quality modelquality model– external metricsexternal metrics– internal metricsinternal metrics– quality in use metrics. quality in use metrics.

• The quality model established in the first part The quality model established in the first part of the standard, ISO 9126-1, classifies software of the standard, ISO 9126-1, classifies software quality in a structured set of characteristics quality in a structured set of characteristics and sub-characteristics divided into attributes. and sub-characteristics divided into attributes.


GQACM integration in an ISO 9126 certified organization

• Facts:Facts:– ISO 9126 define quality attributes .ISO 9126 define quality attributes .– GQACM support the measure and evaluation of quality metrics.GQACM support the measure and evaluation of quality metrics.

• Hypothesis:Hypothesis:– ISO9126 Functionality: adequacy.ISO9126 Functionality: adequacy.

– ISO9126 asks for correct functions with specific goals.ISO9126 asks for correct functions with specific goals.– GQACM may check not dead functions and other coding GQACM may check not dead functions and other coding

convention.convention.

– ISO9126 Maintainability: all.ISO9126 Maintainability: all.– ISO9126 asks to develop a software that is easy to analyse, easy to ISO9126 asks to develop a software that is easy to analyse, easy to

modify, Stable…modify, Stable…– GQACM metrics (such as the CK for OO) helps to verify in concrete GQACM metrics (such as the CK for OO) helps to verify in concrete

these requirementsthese requirements


GQACM integration in an ISO 9126 certified organization

• ISO9126 Functionality: accuracy.ISO9126 Functionality: accuracy.– ISO9126 asks that a software have to give results according with ISO9126 asks that a software have to give results according with

requirements.requirements.– GQACM certification can run functional test (or plug-in external GQACM certification can run functional test (or plug-in external

test suits) of declared compliances.test suits) of declared compliances.

• ISO9126 Portability: adaptability, installabilityISO9126 Portability: adaptability, installability– ISO9126 asks OS adaptability and installability for software of ISO9126 asks OS adaptability and installability for software of

certified organizations.certified organizations.– GQACM may test deployment of SW on different platforms GQACM may test deployment of SW on different platforms

(HW+OS)(HW+OS)

• Our thesis:Our thesis:– ISO9126 quality metrics can be measured and evaluated ISO9126 quality metrics can be measured and evaluated

based on GQACMbased on GQACM– Any tool implementing it and can be integrated as Any tool implementing it and can be integrated as

supporting ISO9126 adoptionsupporting ISO9126 adoption


CMMI

• It provides organizations with guidelines for effective It provides organizations with guidelines for effective organisational processesorganisational processes

• It can guide organisational process improvement It can guide organisational process improvement across a project, a division, or an entire organizationacross a project, a division, or an entire organization

• It helps to integrate traditionally separate It helps to integrate traditionally separate organizational functionsorganizational functions

• It sets goals and prioritiesIt sets goals and priorities


GQACM integration in a CMMi certified organization

• FactsFacts– CMM is process oriented (focused on the organisation).CMM is process oriented (focused on the organisation).– GQACM is product oriented.GQACM is product oriented.

• Hypotesis:Hypotesis:– CMM2 CMM2

– asks to “control and value”asks to “control and value”– GQACM propose the measures and metrics to be collected, hence GQACM propose the measures and metrics to be collected, hence

controlling and evaluating specific sw related itemscontrolling and evaluating specific sw related items

– CMM3 CMM3 – asks to “use internal standards and have a reference model”asks to “use internal standards and have a reference model”– GQACM proposes code and implementation conventions (how to GQACM proposes code and implementation conventions (how to

organise and structure code, how to define tests, etc…)organise and structure code, how to define tests, etc…)


GQACM integration in a CMM certified organization

• CMM4 CMM4 – asks to “have measured and controlled processes using asks to “have measured and controlled processes using

quantitative and statistical techniques”quantitative and statistical techniques”– GQACM propose the continuous collection of quality data to GQACM propose the continuous collection of quality data to

perform trends analysis.perform trends analysis.

• CMM5 CMM5 – asks to have an “improvement process and quality based on asks to have an “improvement process and quality based on

measurement”measurement”– GQACM provide numerical reference to set the improvement GQACM provide numerical reference to set the improvement

processes!!processes!!

• Our ThesisOur Thesis– GQACM is not in contrast with CMMI and can be integrated in GQACM is not in contrast with CMMI and can be integrated in

organization as a tool to support CMMI adoptionorganization as a tool to support CMMI adoption


ETICS B&T tool and GQACM

• ETICS:ETICS:

– Can perform many measurements and Can perform many measurements and calculate metrics calculate metrics code perspective code perspective

– Can build and test software on several distinct Can build and test software on several distinct platforms platforms platform perspective platform perspective

– Can test the software compliance with Can test the software compliance with standards standards standard perspective standard perspective

– Can automate all the processes (build, Can automate all the processes (build, deployment, test)deployment, test)


Where to find more on ETICS?

• ETICS portal: ETICS portal: http://http://eueu--eticsetics.org.org


Where to find more on ETICS?

• ETICS wiki:ETICS wiki:https://https://uimonuimon..cerncern..chch//twikitwiki/bin/view/ETICS//bin/view/ETICS/WebHomeWebHome

• ETICS client ETICS client getting started getting started https://https://uimonuimon..cerncern..chch//twikitwiki/bin/view/ETICS//bin/view/ETICS/ClientHowToClientHowTo

• ETICS ETICS tutorialstutorialshttps://https://uimonuimon..cerncern..chch//twikitwiki/bin/view/ETICS//bin/view/ETICS/ETICSTutorialsETICSTutorials

UnderUnderdevelopmentdevelopment


Conclusions

• ETICS provides a ETICS provides a multi-platform environmentmulti-platform environment for for building, testing and checking the quality of distributed building, testing and checking the quality of distributed and grid applicationsand grid applications

• ETICS focus is on ETICS focus is on automation automation of builds, tests, of builds, tests, reporting, collection and analyse of quality measures reporting, collection and analyse of quality measures … to simplify life when managing complex software … to simplify life when managing complex software management tasksmanagement tasks

• We want it to be as We want it to be as simple and effective as possible to simple and effective as possible to useuse. For this . For this we need your feedbackwe need your feedback::

ETICS-ETICS-QAstudyQAstudy@@cerncern..chchETICS-discuss@[email protected]@[email protected]


Thanks

http://www.eu-etics.org

Thanks you for taking the time Thanks you for taking the time

to learn about ETICSto learn about ETICS

Have a good day Have a good day

Give us feedbackGive us feedback