A Case Study Analyzing the Impact of Software Process Adoption on Software Quality

Reham Tufail Faculty of Information Technology

University of Central Punjab Lahore, Pakistan

reham_tufail@ucp.edu.pk

Ali Afzal Malik Faculty of Information Technology

University of Central Punjab Lahore, Pakistan drali@ucp.edu.pk

Abstract— A constant focus on improving the quality of software lies at the heart of better customer satisfaction, increased market share, and higher profit margins. This paper presents the results of a case study designed to analyze the impact on software quality of the adoption of Scrum – an agile software development process – by a Pakistani software house that was not using any particular process earlier. Pre- and post-process adoption data on the severity of errors and defects and the ratio of passed to failed test cases is gathered and analyzed for four web-based financial applications developed by the software house. The results of the quantitative analysis clearly indicate that the adoption of an agile process improved the quality of the software produced by the software house by reducing the percentage of serious errors and defects and by increasing the ratio of passed to failed test cases.

Keywords-agile processes; case study; defect severity; Scrum; software quality.

I. INTRODUCTION Like the quality of tangible products (for example cars, lawn

mowers, microwave ovens), the quality of a software product plays a major role in ensuring its success. A lack of focus on software quality can lead to diminished market share, disgruntled customers, and lower profit margins. Nowadays, increasing emphasis is being placed on producing high quality software products within budgetary constraints and time limitations [1]. Software organizations have spent a lot of money in research to develop methodologies for producing high quality and dependable software [2].

The quality of a software product is influenced by a number of factors [3] [4]. The quality of the personnel developing the software, for instance, is a key determinant of the quality of the delivered product. Other things being constant, the more qualified and experienced these personnel are, the better the software quality. Similarly, another factor that significantly influences software quality is the process used by the software house to develop the software.

This research paper presents a case study analyzing the impact of software process adoption on software quality. This case study looks at four web-based financial applications developed by a well-established Pakistani software house that switched to Scrum [5] – an agile software development process – to improve the quality of its software. It compares the pre- and post-process adoption data on the severity of errors and defects and the ratio of passed to failed test cases.

The results of the quantitative analysis clearly indicate that the adoption of Scrum improved the quality of the software produced by the software house by reducing the percentage of serious errors and defects and by increasing the ratio of passed to failed test cases. These results can act as supporting evidence to convince other software houses to adopt a systematic approach to develop software.

Rest of the paper is organized as follows. The next section presents the related previous work in this area and compares it with our work. Section III describes the background and empirical setting of our case study. Section IV presents the results of our data collection and includes a commentary on their salient aspects. Finally, Section V concludes this paper by summarizing the major findings of our case study and presenting ideas for future work in this area.

II. RELATED WORK A comparison of lightweight agile processes with the

heavyweight waterfall model is presented in [6] to show how agile processes can be used to produce good quality software. This comparison, however, is qualitative in nature. No statistics are presented to substantiate the comparison. Our work, on the other hand, presents a quantitative comparison showing the statistics of three quality indicators obtained by analyzing real projects.

Ashrafi [7] reports the results of a survey conducted to judge the impact of software process improvement on software quality factors such as correctness, maintainability, verifiability. Even though this survey is taken by members of four professional organizations, unlike our work, real project data is not used to derive the conclusions.

2012 10th International Conference on Frontiers of Information Technology

978-0-7695-4927-9/12 $26.00 © 2012 IEEE

DOI 10.1109/FIT.2012.52

254

A quantitative analysis is conducted in [8] to assess the impact of software process improvement on the severity of defects. Even though this longitudinal field study performs a quantitative analysis on real data, it focuses only on a single quality indicator that is the severity of defects. Our study, on the other hand, looks at three quality indicators that is severity of errors, severity of defects, and passed-to-failed ratio (P/F) of test cases.

III. EMPIRICAL SETTING This case study focuses on a well-established Pakistani

software house that specializes in developing applications for the prepaid card industry. This software house has a strength of over 400 employees. Applications developed by this software house enable its clients to handle financial transactions involving interaction with banks. The business-critical nature [9] of these applications necessitates a constant focus on their quality.

Fig. 1 depicts the timeline showing the progress in process maturity of the software organization used in our case study. Till the year 2008, no formal software development process was being used by this software house. In 2009, this software house began a transition towards adopting Scrum [5] – an agile software process. During this transition period, a hybrid approach was being used for software development that is the only a subset of Scrum practices had been adopted. In the year 2010, this software house completely switched to Scrum.

Data related to software quality is gathered only from years 2008 and 2010. These two years represent pre-formal process adoption and post-formal process adoption periods, respectively. We do not gather software quality data from the year 2009 since the year 2009 represents a transition period.

Fig 1. Formal process adoption timeline

In this case study, data is gathered from four major web-based financial projects done at the software house under consideration. To maintain confidentiality, these projects have been renamed to Projects A, B, C, and D. Table I contains the descriptions of these projects. Project A is related to the administrative site while project B belongs to the customer service site. Administrative and customer service sites are managed by the clients of the software house to facilitate their customers. Project C is related to the cardholder site while project D belongs to the online sales site. Cardholder and online sales sites are used by the customers themselves for their management and processing.

TABLE I. PROJECTS’ DESCRIPTION

Project Description

Project A Used by clients to manage their customer's settings for example roles, privileges, transactions and many more.

Project B

Used by client's customer service representatives (CSRs) to facilitate their customers by. Answering phone and email inquiries to block/reissue cards, change personal information.

Project C Used by the customers to access and manage their data that is the personal information, card transactions, block/re-issue requests.

Project D Used by customers to order delivery of their cards.

To assess a project’s quality we have used three indicators

that is severity of errors, severity of defects, and passed-to-failed ratio (P/F) of test cases. Using the definition in [10], errors are distinguished from defects using the time of occurrence. A problem encountered in the local environment before deployment is considered an error while a problem uncovered after deployment in the production environment is treated as a defect.

As shown in Fig. 2, the severity of both errors and defects is judged using a four-level scale. An error or defect is labeled a showstopper (highest severity level) if it crashes or halts the application. Critical errors/defects are related to problems in functional requirements. Incorrect results are an example. Significant errors/defects, on the other hand, are related to non-functional requirements. An actual response time that is longer than the minimum acceptable response time is an example. Minor problems related to usability, placement or alignment of controls on the GUI, spelling mistakes in labels are classified as cosmetic (lowest severity level). These problems neither disrupt the flow of the application nor impact its functionality.

Fig 2. Severity levels of errors and defects

IV. RESULTS AND DISCUSSION Tables II, III, and IV depict the results of this case study

obtained by compiling and consolidating data from all of the four projects. Each of these three tables presents a comparison between the two periods that is one in which no formal process was being used (No Formal Process) and one in which Scrum was being used (Agile). Each table presents this comparison for a different quality indicator.

255

TABLE II. COMPARISON OF ERROR SEVERITY

Process % Showstopper

% Critical

% Significant

% Cosmetic

No Formal Process 3.37 33.03 38.95 26.63

Agile Process 1.66 29.92 49.16 19.23

TABLE III. COMPARISON OF DEFECT SEVERITY

Process % Showstopper

% Critical

% Significant

% Cosmetic

No Formal Process 2.71 28.05 25.33 44.11

Agile Process 1.32 24.50 30.46 43.70

TABLE IV. P/F OF TEST CASES

Process P/F

No Formal Process 1.25

Agile Process 8.62

Table II presents this comparison with respect to the severity of errors. It shows the percentage share of each type of error for both stages. Going from the No Formal Process stage to the agile stage, the percentage of showstopper, critical, and cosmetic errors decreases while the percentage of significant errors increases. The percentage reduction is over 50% for showstopper errors, under 10% for critical errors, and over 25% for cosmetic errors. The percentage of significant errors increases by over 25%.

Table III compares defect severity of the two periods. Like the composition of errors, the composition of defects also changes due to the adoption of Scrum. The percentage of showstopper defects is reduced to almost half while the percentage of critical defects is reduced by about 13%. The percentage of cosmetic defects remains almost the same while the percentage of significant defects increases by almost 20%.

Table IV compares P/F for the two periods. As is evident from the numbers, P/F is almost seven times more in the agile stage.

The results shown above clearly indicate a couple of important points. Firstly, the adoption of Scrum decreases the percentage of high severity (that is showstopper and critical) errors. In other words, it reduces the impact of software failure. Secondly, moving from the No Formal Process stage to the agile stage increase the P/F ratio. Apart from implying better quality, it also implies a reduction in the amount of effort spent on testing the software. This is due to the fact that fewer errors need to be fixed and, therefore, fewer resources need to be devoted to regression testing.

V. CONCLUSIONS AND FUTURE WORK In this research paper we have tried to determine the impact

on software quality of adopting a formal process by using the case study of a Pakistani software house specializing in developing applications for the prepaid card industry. Three different quality indicators – error severity, defect severity, and

P/F – were used for this purpose. Data related to these indicators were obtained and analyzed for two different stages that is No Formal Process stage and agile stage. The results of the comparison of these two stages indicate that adopting a formal agile process – Scrum in our case study – contributes to improving software quality by not only reducing the percentage of high severity errors and defects but also by increasing the P/F ratio.

To make our analysis more complete and thorough, we plan to look at other quality indicators such as defect density, defect removal efficiency in the future. We also intend to use this extended set of quality indicators to compare the impact of the adoption of different agile process models such as Scrum [5], XP [11], Feature Driven Development [12], Crystal Clear [13] on software quality.

ACKNOWLEDGMENT I am very thankful to Mr. Saqib Bukhari for helping me in

acquiring the data needed for this case study.

REFERENCES [1] S. M.Tawfik, M. M. Abd-Elghany, S. Green, “A software cost

estimation model based on quality characteristics”, IWSM-Mensura, 2007.

[2] P. J. Denning, “What is software quality?” A Commentary from Communications of ACM, Jan., 1992.

[3] M. C. Paulk, "Factors affecting personal software quality", Institute for Software Research, Paper 4, 2006.

[4] F. J. Heemstra and R. J. Kusters, “Soft factors affecting software quality”, Software Quality Professional, 5(1), 20-29, 2002.

[5] K. Schwaber and M. Beedle, “Agile software development with Scrum”, 1st ed., Prentice Hall, 2001.

[6] M. Huo, J. Verner, M. A. Babar, L. Zhu, “How does agility ensure quality?”, Scotland 2nd Workshop on Software Quality, 2004.

[7] N. Ashrafi, “The impact of software process improvement on quality: In theory and practice”, Information and Management, Vol. 40, no. 7, pp. 677–690, August 2003.

[8] D. E. Harter, C. F. Kemerer, and S. A. Slaughter, “Does Software Process Improvement Reduce the Severity of Defects? A Longitudinal Field Study,” IEEE Transactions on Software Engineering, pp. 1-22, June 2011

[9] I. Sommerville, "Software Engineering," 9th ed., Addison-Wesley, 2010.

[10] R.S. Pressman, “Software Engineering: A practitioner’s approach”, 7th ed., McGraw-Hill, 2009.

[11] K. Beck, “Extreme Programming explained: Embrace change”, 1st ed., Addison-Wesley Professional, 1999.

[12] S. R. Palmer and J. M. Felsing, “A practical guide to Feature-Driven Development, 1st ed., Prentice Hall, 2002.

[13] A. Cockburn, “Crystal Clear: A human-powered methodology for small teams”, 1st ed., Addison-Wesley Professional, 2004.

256