62 Testing Experience – 22/2013

Can software quality be seen as practical and business ori-ented, and with indicators showing that quality can have the effect of improving products and business?

Can we go beyond the idea of measuring software quality through defects of the software product, and move on to also consider other parameters?

What does a software quality model look like in practice?

By analyzing an actual case, this paper shows how the ab-stract concepts of software quality were implemented, using applicable quality models and quality indicators, and how a path to future cycles of optimization could be achieved while pursuing a continuous cycle of business and software product improvement.

To do this, the intuitive process followed by the Organiza-tion is first described, and then a series of actions to further improve what has already been achieved are proposed and explained. These actions could be implemented by differ-ent organizations to improve the quality of their software products.

IntroductionSoftware development and evolution processes are considered essential to obtaining and maintaining quality in software products. For a long time, several models have been proposed to help understanding while increasingly improving how software products are built and maintained.

From the point of view of the software product itself, software quality models, perhaps less known and used in practice, help us to understand how quality requirements can be found, describe their quality charac-teristics, and then evaluate the quality of the software built, based on these characteristics.

This paper shows how an Organization that develops and maintains a large product, which is central to the company business, has moved towards the idea of product quality and has identified the need for a new organization in its Quality Assurance area and for a change to the established processes.

It also discusses how to define business indicators closely related to the software product, so that the indicators are actionable, taking into ac-count the observed behavior of users and customers.

An e-commerce software product and the evolution of its quality

The original product

Architecture

For several years the e-commerce software product was a closed mono-lithic platform. Both database and source code were centralized. The design of this architecture was doubtless intended to generate multiple competitive advantages, but high costs, loss of flexibility, and high risks were incurred when changes were required.

The impact of bugs

Under this monolithic, centralized, and closed scheme, defects introduced as a result of a mistake during development had a high probability of impacting on different elements of the application and, therefore, having direct and important consequences on the business.

The risk associated with a latent defect was very high and the unintended consequences on the business could be significant. As a result, for several years the focus was on correcting defects. The QA Team was committed to covering the widest possible set of code with tests building a very large safety net before any implementation.

The product development process

Consequently, processes designed under these assumptions have had very slow development and testing cycles, taken extensive time, been difficult to scale up and had a “time-to-market” far from usual in the industry today.

The introduction of a defect generated an impact, not only often unpre-dictable, but also difficult to solve due to the slow and costly processes involved. Development and release cycles were significantly long, weeks/months were devoted to testing, and the release rate was sometimes biweekly or monthly. Exceptionally, a release was implemented to solve critical failures, but only in cases of emergency and after deliberation and approval by a committee.

This all created organizational dynamics with a low tolerance for defects, and with incentives aligned to avoid and prevent any type and size of defect. Errors were frowned on and, therefore, the ability to produce and iterate tended to decline.

Consequently, the task of testing became critical, and motivations and challenges revolved around improving testing practices so they were scalable in effort and time, increasing the coverage of testing, and im-proving the design of test cases, defect reports, etc.

By Rodrigo Guzmán, Pilar Barrio la Iglesia & Raúl Martínez

Software Product Quality Beyond Defects

Testing Experience – 22/2013 63

Indicators and incentives

The concept of product quality was simple and was defined only in terms of detected errors. Metrics and goals were designed around these con-cepts and focused on measuring and controlling the number of failures in production. Incentives for team members were aligned with these goals.

This biased approach introduced important frictions into the Organiza-tion and also introduced business constraints. Assuming that success and product quality were dependent only on the software failures variable was like not seeing the wood for the trees.

UsablePerformable

Secure …Functionality OK

Figure 1. This stage can be visualized as the base of a pyramid. The effort is put into a flawless product and the rest of the features are assumed to have already been implemented. Indicators of success are related to a product with a low error rate. [15]

The new product

Architecture and its influence on quality and processes

A dramatic change happened when the e-commerce product became an open platform for e-commerce instead of a website.

This had strong implications for all the variables of the Organization and its business. The change meant redesigning architecture, open-ing the platform, and decentralizing both the database and the source code, allowing separate departments distribution of the e-commerce application.

Definition and design of the new architecture succeeded in eliminating friction as well as functional and technical dependencies which existed among the different modules in the monolithic model. Consequently, risk was distributed and this became the most important change factor in the Organization and the business.

Limiting the impact of errors and reducing business risk resulted in diminished tension regarding processes and considerable improvement in satisfying the Organization needs.

Additionally, responsibility for product quality control was distributed throughout the various areas.

Agile processes for developing and testing were established, enabling an increase in test frequency, the amount and parallelism of changes, improved deployment speed and defect correction, and increasing de-ployment frequency from one every fifteen days to, in some cases, more than eighty releases to production in one day.

Agile practices reduced cycles of design, development, testing, and im-plementation, and weekly sprints with constant delivery of software artifacts were achieved. Project duration was also significantly reduced.

Implementing a change in this context is now simpler, solving a problem is quicker than before, the defects life cycle has been significantly reduced, the impact of errors has diminished, and risk has therefore decreased.

Organizational changes

This new perspective has introduced a radical change into the orga-nizational dynamics, focusing and aligning teams on the search for continuous improvement and constant change rather than attempting to prevent all defects and on achieving a good enough quality product which does not affect the business indicators.

Development teams have increased their capacity to build, test, and quickly solve problems.

In this new scenario, the goals and competences of the original QA Team became obsolete, and a radically new paradigm was necessary.

The effect and impact of product defects is no longer the main cause of friction with the business. Other variables have been exposed that affect it, which formerly were not explicitly considered part of the definition of product internal quality. These variables had always existed but had never come to the fore.

The incentive of the new QA Team is no longer based on reducing errors in the production environment, but on being committed to continuous improvement, constant change, and identifying aspects of the product that generate friction, and directly affect the business and the comple-tion of transactions on the platform.

This is how the evolution of the concept of product quality has enabled the e-commerce company platform to support business needs.

New indicators and incentives

The evolution in the definition of product quality can encompass aspects beyond the platform operation, highlighting other issues such as overall product performance, usability, user’s understandability, usefulness, degree of transaction conversion, success, etc., which certainly affect the evolution and growth of the business.

It is no longer enough to avoid product errors and ensure platform per-formance and availability. It is necessary to give the users what they need and the way they need it. The new goals are to fulfill their expectations at the right time, and create an experience that differentiates the e-com-merce product from its competition, with chance of mistakes but with very streamlined and responsive processes for change and correction.

Indicators at several levels monitor the behavior of the new variables:

Level 1 – Scalability

Traffic light status indicators track platform availability; “health” of the infrastructure, networks and databases; downtime detection; opera-tional capacity; performance; etc.

Level 2 – Core Operation

Malfunction-oriented indicators on product operation, detecting inci-dents, security issues, integration issues, code coverage level in testing, etc.

64 Testing Experience – 22/2013

Level 3 – Usability

Indicators focusing on detection of transactional flow frictions and navigational friction within the platform, as well as understanding, comprehension, and user adoption, detection of “drop outs”, “bounce”, response times, etc.

Level 4 – Customer Satisfaction

Indicators focusing on the level of complaints, claims, and “noise” deter-mination arising from users’ problems on the platform, understanding, failures, adoption, etc.

Level 5 – Profitability

Indicators to measure degree of flows conversion within the platform, visits vs. transactions, and other business and money metrics.

Level 6 – Competitiveness

Indicators that analyze product positioning vs. competitive products, platform level of traffic, market share, new “features”, etc.

Successful

Useful

UsablePerformable

SecureInstallable …

Functionality OK

Figure 2. This stage can be visualized when completing the pyramid in Figure 1 with two other levels: a useful product for users and, once this is filled in, a successful product for the business. In the lower levels, success indicators relate to low rates of errors, whereas, as they move towards the vertex, they are defined in relation to KPIs of interest to the business, not forgetting client satisfaction.

Situation analysisWhile several decisions mentioned in the preceding paragraphs were possibly adopted in an intuitive way, they were undoubtedly intended to solve problems that arose in the operational context, and/or to improve it.

Some questions arising from the original product description are:

▪ What motivates the e-commerce Company …

▪ … to face a change in architecture?

▪ … to face an organizational change?

▪ What is the meaning of the new indicators generated?

Possible reasons behind the decisions and changes are analyzed below.

Stakeholders and their expectations

The Company and their expectations:

▪ An agile product, easy to expand

▪ With a competitive level of functionality

▪ With acceptable development and operation costs

▪ With indicators that show business performance through customer behavior in using the product.

The visitor and their expectations:

▪ A mature and reliable product

▪ A fully fledged product compared to others of its kind

▪ Simple to use

▪ Highly secure

▪ With adequate performance

Groups responsible for product development and their expectations:

▪ A product with adequate time-to-market

▪ Analyzable and modifiable with minimum impact

▪ Testable before releasing changes

▪ Auditable

Although expectations were initially fulfilled by the original product, they changed as new competition appeared in the market and the product evolved. Stakeholders began to feel disappointed, perceiving the product as difficult to maintain and test. The Organization was seen as rigid and with few indicators for tracking product/business performance.

These warnings forced the e-commerce Company to face radical changes, such as:

▪ A very important change in architecture

▪ A new product development process

▪ A new product development and support organization

▪ A new set of indicators

The architecture

As mentioned previously, the original architecture was monolithic, with technical problems and organizational accountability issues, and conflict situations were difficult to solve.

The architectural change was conceived to fulfill management expecta-tions and the product groups’ expectations.

Visitors/users still observed the same or even improved product behavior. But the cost of that behavior was very different when comparing the original architecture with the new one.

This was not visible to visitors, but was very significant to the business.

The product development process

The original development and product maintenance processes were pos-sibly based on the product structure, and included all the steps needed to develop and test a large monolithic product with the deliverables and controls required to perform a comprehensive test and subsequent implementation of a highly coupled product.

The consequences were mentioned earlier in this paper.

The processes were changed to fulfill management and product groups’ expectations.

Implementation of Agile practices seems appropriate for a decentralized architecture as described for the new product, assigning smaller teams complete responsibility for some aspect of it, thus providing improved clarity and definitions.

CaseMaker SaaS systematically

supports test case design by

covering the techniques taught in

the ISTQB® Certi� ed Tester program

and standardized within the British

Standard BS 7925. The implemented

techniques are: equivalence

partitioning, boundary check, error

guessing, decision tables, pairwise

testing, and risk-based testing.

CaseMaker SaaS is your perfect � t

between requirement management

and test management/test

automation.

Quit struggling

with the decision

and enjoy test

case design in

the cloud today!

Subscribe today

and enjoy a 14-day

trial period for free!

Your license starts at 75 €/month (+ VAT).

saas.casemaker.eu

66 Testing Experience – 22/2013

The organization

The original organization that developed and maintained the e-com-merce product evolved in parallel with the new processes.

The organizational changes were focused on fulfilling the expectations of management and of the groups responsible for the product relative to the “time-to-market”, responsibilities, and maintenance cost.

The development of new indicators showing the relationship between product behavior and business performance was also addressed.

The new organizational model dissolved the centralized QA Team and distributed its members and responsibilities in the new product teams, now responsible for planning and implementing new features or changes until these reach the production or live stage.

A new area called Business Assurance was created, with no similarities to the previous QA Team but in charge of operational control of the product and analyzing its impact on business through performance indicators.

The indicators

Due to the product’s original monolithic structure and organization, it was difficult to detect which item to act on in a conflict situation. Also, indicators related to the base of the pyramid in Figure 1 were not useful for the business.

Management’s expectations of monitoring evolved into actionable in-dicators, meaning that action can now be taken on any portion of the product or the business to correct or enhance a situation once it has been detected. Not all indicators are necessarily financial.

Therefore, new processes, architecture, indicators, and organization al-low a more direct measurement of the behavior of a given feature and enable it to be acted on it if necessary.

The new indicators defined by levels are intended to detect the behavior of the product in use by the customers (or other interested visitors) and define interventions, if needed, into parts of the product. See Figure 2.

While not simple, the indicators enable the e-commerce Company to establish whether or not business objectives related to the software product have been achieved. They also form the basis for evaluating incentives given to those responsible for the various aspects of its de-velopment and operation.

Possible next steps

Current and new stakeholders

Continue with stakeholder detection. In addition to site visitors and merchants, reach other possible stakeholders: external developers who extend the product, auditors, information security, etc. Understand their influence and propose communication channels with them to improve relationships.

Expectations discovery tools

For the detected interested communities, find out what their expecta-tions are in respect of the product characteristics. This can be done through instant surveys, online questions to visitors/buyers and mer-chants, or through meetings with other stakeholders, e.g. with external developers.

Formalization of quality model

The evolution of quality concepts in the Organization justifies a formal-ization of the quality model allowing transference to the Company’s own staff and third parties, such as developers, that generate extensions to the e-commerce platform through APIs.

This will permit understanding by all developers of the expected quality level and a possible assessment by the e-commerce Company, regarding the level of quality offered. User, visitors, or buyers will perceive a product with homogeneous quality.

Committed external developers

Explain to external developers the expectations of the e-commerce Com-pany regarding quality through a visible, accurate, and easily understood model. This will guide their development work and help to obtain their commitment to quality.

Business-related KPIs

These KPIs are derived from measures taken after a release of changes to the production environment.

In this way, external and internal developers will easily associate a busi-ness KPI with the portions of the product affecting or determining that KPI and take action on the product if necessary.

Product development KPIs

These are indicators derived from measures obtained prior to the release of a new feature of the product.

Develop and/or improve the KPIs that measure the quality of product development and testing, so that final quality can be predicted and their potential impact on the production environment measured.

Business and product KPIs and how to obtain them must be proposed, discussed and agreed with technical and business stakeholders.

These measures must form another component of the quality model mentioned early.

Continuous improvement

The steps described close the loop on an improvement cycle that should be repeated continuously in order to improve product quality and make the business more and more competitive and successful every single day.

ConclusionThe case has shown, from a practical point of view, how an organization has evolved in relating business success to the software product quality.

It also explained how the Organization changed its structure and work-ing models in the area of software to adapt to market demands on times and expectations.

After the description of the case, an analysis was made of the steps followed intuitively by the Organization, creating a parallel with the development of a software quality model.

Finally, a proposal was made to the Organization to further improve the quality of its product and, thus, the business results.

Testing Experience – 22/2013 67

The example, although just a case, can be used by other enterprises and organizations who want to achieve a stronger and better relationship between their software product and their business.

Bibliography and references[1] Factors in software quality; NTIS, 1977, J. McCall.

[2] Software Quality: The elusive target; IEEE, 1996, B. Kitchenham & S. L. Pfleeger.

[3] What does “Product Quality” really mean?; Sloan Management Review, Fall 1984, D. Garvin.

[4] A model for software product quality; Australian Software Quality Research Institute, October, 1994, G. Dromey.

[5] Relating Business Goals to Architecturally Significant Requirements for Software Systems; CMU/SEI-2010-TN-018, 2010, Bass, Clements.

[6] Quality Attribute Workshops (QAWs); Third Edition, CMU/SEI-2003-TR-016, 2003, Barbacci.

[7] Software Architecture in Practice; 2nd ed., 2003, Bass, Clements, Kazman.

[8] ISO/IEC 25000 Software engineering: Software product Quality Requirements and Evaluation (SQuaRE) – Quality model; 2005, ISO/IEC.

[9] ISO/IEC 25010 Software engineering: Software product Quality Re-quirements and Evaluation (SQuaRE) – Quality model; 2011, ISO/IEC.

[10] Cornering the chimera; IEEE SOFTWARE, 1996, G. Dromey.

[11] Competing on the Eight Dimensions of Quality; HBR, 1987, D. Garvin.

[12] Software Quality Models in Practice; Umfrage-Ergebnisse, 2010, QuaMoCo Group.

[13] In Application Projects, ‘Success’ Needs Many Definitions; 2011, Gartner.

[14] Application Quality Assurance for Nonfunctional Requirements; 2011, Gartner.

[15] Redefining-software-quality; www.gojko.net, 2012, Gojko Adzic.

[16] Norms and Standards in SAP’s Development Process Framework; 2010, SAP.

[17] Attractive quality and must-be quality; ASQC, 1996, N. Kano, N. Seraku, F. Takahashi, S. Tsuji. ◼

Rodrigo Guzmán is Quality Assurance Senior Man-ager at MercadoLibre.com a Latin American leading e-commerce technology company. He joined the com-pany in 2004 and is responsible for defining, imple-menting, and managing software quality policies that enable IT to ensure and control the operation of the website in the 12 countries of Latin America. He is also responsible for improving the IT processes as well as

project management.Rodrigo has a degree in Business Administration, a postgraduate degree in Quality & Management, and is a Certified Scrum Master. He has nineteen years’ experience in IT, mainly in processes, projects, and quality.Before joining MercadoLibre.com, he worked for 10 years in the IT area at Telecom Argentina, a telecommunications company.Among other things, in 2007 he implemented the quality management system for the QA department, currently certified under ISO-9001:2008 standards. He also implemented test automation, project management process improvement, and Agile practices in IT projects.

Pilar Barrio la Iglesia has specialized in Quality Man-agement, having been responsible for those areas at consulting firms. She has extensive practical experi-ence, as well as a solid theoretical background in soft-ware development and the technology-related aspects of business. She has collaborated in the development and implementation of tools to support IT process improvement, and participated in, coached, and man-

aged diverse projects in Argentina and abroad.Pilar is a founding partner of RMyA SRL, Argentina, a firm established 13 years ago which is dedicated to providing software product testing and process improvement services to IT organizations and to medium-to-large customers in the areas of project management and quality management.She is an instructor on quality-related courses in various areas related to computer technology and is a former Professor at Universidad de Buenos Aires in the Databases and Operative Systems areas.Pilar is a member of the Quality Commission – CESSI.

Raúl Martínez specializes in IT Project Management and software engineering.He has a theoretical background and practical experi-ence in software development in leading local and international companies and has worked with clients’ IT divisions to develop innovative, reliable, and cost-effective software products.Raúl is a founding partner of RMyA SRL, Argentina,

a firm established 13 years ago which is dedicated to providing software product testing and process improvement services to IT organizations and to medium-to-large customers in the areas of project management and quality management.He has been a Regular Associate Professor in the area of Software Engineer-ing at the Faculty of Engineering, Universidad de Buenos Aires for over 30 years, teaching Project Management courses for future System Engineers and Bachelors.Raúl is a member of the Curriculum Committee at the Faculty of Engineering (UBA), for the undergraduate degree in Systems Engineering. He is also a member of the Quality subcommittee in Information Technology and Project Management at IRAM and an IRAM appointed expert to ISO/IEC JTC1/SC7/WG6 on the subject of Product Quality ISO/IEC 25000 Standards (SQuaRE). He is a member of the Quality Commission – CESSI.

> about the authors