View
4
Download
0
Category
Preview:
Citation preview
Spago4Q and the QEST nD Model:An Open Source Solution
for Software Performance MeasurementClaudio A. Ardagna, Ernesto Damiani, Fulvio Frati, Mauro Regoli
Dipartimento di Tecnologie dell’InformazioneUniversità degli Studi di Milano – Italy
Sergio Oltolina, Gabriele RuffattiEngineering Ingegneria Informatica - Italy
Outline
• QEST nD Model
• Spago4Q Open Source Platform
• Roadmap to build the Integrated Framework
• Case Study
2OSS 2010
Introduction
• The availability of detailed and updated information on development process is paramount– To maintain the competitiveness level – To operate in new and more challenging markets
• The scenario of integrated information is known as Business Intelligence, and enclose all the processes and tools for data acquisition and analysis
• A number of structured process models have been adopted depending on the domain and size– To collect specific knowledge about their development processes– To strengthen the know-how in terms of more efficiency and quality
3OSS 2010
Goals
• Deploy an environment that integrates models for software performance evaluation in a more complete and coherent framework– Non-invasive– Dynamic– Configurable– Scalable– Based on a formal mathematical model
• Provide a solution integrating– The QEST nD model for process performance evaluation– The Open Source Business Intelligence platform Spago4Q
4OSS 2010
The QEST nD Model - 1
• Several mono-dimensional performance models are available in SE context that integrates individual measurements into a single performance index– Models are oversimplified and unable to reflect different performance
aspects
• Traditional domains (e.g., Business Modeling) exploit multi-dimensional models – Data derived directly from accounting systems, implying multiple
analysis viewpoints
– Complex viewpoints analyze distinct aspects of the overall process performance
5OSS 2010
The QEST nD Model - 2
• The QEST nD model is aimed at measuring software project performances addressing multi-dimensionality and qualitative-quantitative assessment
• In the QEST model the quality can be viewed as the integration of at least three different viewpoints
– Economical
– Social
– Technical
• User-specific dimensions could be defined to evaluate ad-hoc analysis dimensions
L. Buglione, “Misurare il software. Quantità, Qualità, Standards eMiglioramento di processo nell'Information & Communication Technology,” 3rd Edition, Franco Angeli, 2008
OSS 2010 6
Mathematical Formalization
• QEST Performance value (P) given by the combination
– Quantitative measurements (RP-Rough Productivity)
– Qualitative measurements, calculated as a perception-based measurement of the overall product quality (QF-Quality Factor)
OSS 2010 7
Mathematical Formalization - 2
• The basic 3-D model QEST is represented as a geometrical regular tetrahedron– Corners E, S, T and the convergence
of the edges to the P vertex describe the performance level
– Points Qe, Qs, Qt represent the respective dimension values (Rough Productivity)
• Values calculated as the weighted sum of a list of n distinct measures
• Represented as the distance between points and corners
• If Quality factor is applied, points Q’e, Q’s, Q’t represent an upward or downward translation of Qe, Qs, Qt
OSS 2010 8
Mathematical Formalization - 3
• Performance P calculated as the distancebetween the center of gravity of the original tetrahedron and the center of the plane (Q’e, Q’s, Q’t) section along the tetrahedron height
• Through computational geometry, the multi-dimensional case is described with a simple formula for global performance value computation
OSS 2010 9
Spago4Q
• Spago4Q (SpagoBI for Quality) is an open source platform for the continuous monitoring of software quality
– Multi-process multi-project monitoring
– Collection of data executed in a fully-transparent way
– Equipped with extractors specific of most-common software process environments (IDE, workflow management, text editing, …)
• Spago4Q (www.spago4q.org) is a vertical adaptation of SpagoBI, the Open Source Business Intelligence suite (www.spagobi.org)
• Recently adopted by European Commission –Directorate General for Regional Policy
OSS 2010 10
Spago4Q Structure
OSS 2010 11
Spago4Q Metamodel
OSS 2010 12
An Integrated Environment
• The definition of a QEST nD model is a five-step procedure fully supported by Spago4Q
• The procedure is coherent with the PMAI (Plan-Measure-Assess-Improve) cycle:– PLAN, defining a set of KPIs, basing on the GQM approach, and
dimensions that characterize the analysis– MEASURE, which includes the collection of data, and the
computation of KPI values and global performance value– ASSESS, where results are presented in dashboards and
reports– IMPROVE, where every negative or low value is deeply
analyzed to find problems in the process
OSS 2010 13
1st Step: KPIs and Model Definition
• Declaration of a complete GQM, with the definition of – the analysis dimensions
– the concepts to measure
– the KPIs to apply to project work-products
OSS 2010 14
2nd Step: Weights and Thresholds Definition
• Couple each KPI with the respective weight– Indicates the importance that
such a concept plays in the dimension it belongs to
• Define the specific thresholds– Evaluates the value with
respect to organization policies
• Assign (if provided) the QF to each specific dimension
OSS 2010 15
3rd Step: Value Collection
• Measures are taken directly from Spago4Q data warehouse– The DB is filled by data automatically collected by
extractors accessing process work-products (code package, text documents, project information, …)
• KPIs are described in terms of – Default value
– Minimum and maximum values (for normalization)
– KPI computation algorithm
OSS 2010 16
4th Step: Global Performance Computation
• Global and dimension-wise performance indexes are computed as KPIs that take in input configuration data and results of the KPIs
• Dimension performance is calculated as the weighted sums of each specific KPI
OSS 2010 17
5th Step: Reports
• Sets of reports and dashboards could be defined and configured to satisfy reporting and managerial needs
• Spago4Q provides methods and interfaces to directly configure and create new reports using all the facilities provided by the open source reporting tool Eclipse BIRT
OSS 2010 18
Case Studies
• Case study based on the monitoring of the complex performance of three real projects with several measurements in a three-month time slot
• We use a QEST nD model called Business-Service Model and taking into consideration four specific analysis dimensions:1. QEST-EC: Economic performance indicator
2. QEST-RS: Resource performance indicator
3. QEST-TE: Technical performance indicator
4. QEST-CS: Customer Satisfaction performance indicator
OSS 2010 19
Case Studies: GQM
OSS 2010 20
Case Studies: KPIs
OSS 2010 21
Case Studies: Dashboards
• Spago4Q dashboard gives an immediate snapshot of the situation, highlighting problems and suggesting areas that need a process quality improvement
– Red squares indicate under-threshold performance values
OSS 2010 22
Spago4Q: Reports
• Reports indicate that all projects were concluded with an excellent global performance (close to one), showing some issues in the process that is worth analyzing– Project3 experienced bad performances in Technological and
Economic dimensions
OSS 2010 23
Conclusions
• We provide a complete multi-project multi-process performance evaluation environment combining– The mathematical formalization of the QEST nD model– The monitoring and reporting facilities offered by Spago4Q
• It gives the possibility to analyze the performance of the development process from different points of view integrating semantically different KPIs in a single indicator
• The described QEST nD model will be made available in the future version of Spago4Q (www.spago4q.org)
OSS 2010 24
Questions?
Thank you for the attention
OSS 2010 25
Recommended