Toward a cooperative software testing process

SOFTWARE TESTING, VERIFICATION AND RELIABILITYSoftw. Test. Verif. Reliab.8, 177–190 (1998)

Toward a Cooperative Software TestingProcess

MANOOCHEHR GHIASSI*

Department of Operations and Management Information Systems, Santa Clara University, 500 El CaminoReal, Santa Clara, CA 95053, U.S.A.

SUMMARY

This paper presents a case study examining the implementation of a change from ‘independentSoftware Quality Assurance testing’ to ‘cooperative testing’ and the resulting impact on thesoftware life cycle. The same test tools have been used, so that any change can be primarilyattributed to the process change. In the new process, bugs are caught and addressed earlierin the cycle. Additionally, early integration of the systematic test process coupled with a changein ownership, improved infrastructure and a more formal auditing of test plan execution hasproduced higher quality software with a more predictable release schedule. The new processhas resulted in a higher rate of bug detection and correction during testing, as well as fewerbug reports from the field. 1998 John Wiley & Sons, Ltd.

KEY WORDS Software testing; software process; SQA testing; cooperative testing

1. INTRODUCTIONThis research examines the software development process in a company producing softwaretools for the embedded systems market. Its product line includes compilers, assemblers,debuggers, linkers, monitors, and real-time operating systems. Most are cross-developmenttools that support various target microprocessors (i.e. Motorola 68000 and Power PCseries, Intel 80x86, i960, Sun SPARC, HP RISC, and a few others). The cross-developmenttools are hosted on PC, UNIX (Sun, HP, IBM), and VAX/VMS. The company is mediumin size (more than 200 employees) and has organized its software development organizationinto two tiers: core technology and toolkit product groups.

The primary function of the core technology group is to develop new technologies andto enhance existing ones. This group is staffed with seasoned software engineers (101years of experience), specializing in various fields of compiler, debugger and real-time operating systems technologies. The toolkit groups are organized around the targetmicroprocessors that they support. Each toolkit group includes a mix of seasoned and lessexperienced (51 years of experience) software engineers. Each group has its own managerand is composed of 6 to 8 engineers. These groups port the generic modifications and

* Correspondence to: Professor M. Ghiassi, Department of Operations and Management Information Systems,Santa Clara University, 500 El Camino Real, Santa Clara, CA 95053, U.S.A. Email: mghiassiKscu.edu

Received 28 September 1994CCC 0960–0833/98/040177–14$17.50 Revised 27 August 1997 1998 John Wiley & Sons, Ltd. Accepted 9 October 1998

178 M. GHIASSI

enhancements developed in the core technology group (as well as target specific modifi-cations and enhancements) to the various targets. For example, consider a component ofa toolkit, the C compiler. The core group will provide support and development for thedriver, the front end, and the optimizer. The toolkit group will provide support for thecode generator and the integration of these parts into the final compiler product.

This paper will examine the software development process at this company and howchanges in the development model have improved quality and product delivery (Figure 1).

The primary focus of this paper, however, is to study the effect of a change in thetesting phase, marked with points A and B in Figure 1. To demonstrate the results ofchanging from testing by an independent group to a more integrated and cooperativetesting process, a case study using three products is presented (Table I).

The next section introduces both the old independent model and the newly implementedcooperative one.

1.1. SQA process, the independent modelThe previoussoftware quality assurance(SQA) process used the independent approach

to software testing. In this model, SQA performed black box testing independently of thedevelopment organization. This was the primary model used for software development inthe earlier days of the software industry (which advocated an independent SQA group).The motivations were specialization, objectivity and independent test development (Daly,1977; Ghiassi, 1985). The availability of standard test suites (Plum, 1994), especially forproducts with industry standards such as compilers [ANSI C/C11], coupled with auto-mated test tools, has allowed non-testers to perform a large portion of the required testing.However, this model was not very effective in enforcing the ‘early-to-test’ principle.Literature examining this model is full of testimonies of ‘back loading’ of the test cycle.This puts SQA groups under constant pressure, and they are sometimes perceived as an‘obstacle to release’ rather than a contributor to product development. The efforts of thisindependent group were often considered to be ‘extraneous’ due to lack of integrationwith the development process (McGregor and Korson, 1994). Occasionally, the need tomeet contractual obligations or desire to avoid missing a marketing opportunity leads toabbreviated testing periods.

It is important to point out that ‘SQA independence’ refers to the organization andhow black box testing is performed. The independence and objectivity requirements oftest development are a separate concept and were maintained across both models.

The SQA staff in this model consists of a manager and nine software quality engineerswith experience levels ranging from three to more than ten years. The team memberswere specialized by products (three quality engineers for each of the three products), andperformed a variety of tasks, including: (a) independent test development; (b) automationof both internal and external test suite executions; (c) black box testing of softwareproducts; and (d) company-wide advocacy of software quality.

This model is presented in Figure 2 with the region between A and B correspondingto the ‘Testing and SQA phase’ of Figure 1.

In this model a new release is initiated in response to software maintenance, bothcorrective and perfective. The new release reflects input from the following groups: (a)technical support, in the form of a prioritized bug list; (b)core group, in the form ofnew features and enhancements; and (c) thetoolkit group, in the form of specialized

1998 John Wiley & Sons, Ltd. Softw. Test. Verif. Reliab.8, 177–190 (1998)

179COOPERATIVE TESTING PROCESS

Figure 1. Overall product development model.

enhancements (Figure 1). The core group modifications are tested by the core group andverified prior to delivery to the toolkit group. The toolkit group uses the latestinternalrelease from the core group and adds product line specific enhancements to produce acomplete product. After all agreed-upon changes are implemented, unit testing and someintegration testing are performed by the development team; the product is then submitted


180 M. GHIASSI

Table I. Size of software products (KLOC)

Product Release 1 Release 2

C compiler 141 151C11 compiler 181 194Debugger 152 169

Figure 2. Independent SQA organization model.

to the release management(RM) group (Figure 2). RM then installs the software andplaces it in the SQA test queue. The SQA group uses this release to perform regressionand black box testing, and to apply the release criteria to evaluate the product and issuetheir report. This report is provided to the management team for the final release (orrejection) decision. Every time the product is rejected by SQA, a bug report is sent to



the development group. This cycle is repeated until the product finally is deemed readyto ship (Figure 2). Included in the release criteria is a set of rules that classifies discoveredbugs by severity. All critical bugs are fixed. Non-critical bugs are documented and whenapplicable, workarounds are provided. A bug is considered ‘critical’ if it can cause theproduct to terminate unexpectedly, crash the host system, or result in silent bad codegeneration that can only be detected at run-time through incorrect behaviour. The finalversion is again submitted to the RM group and the ‘master/gold’ version is producedand delivered to manufacturing for duplication, packaging and final shipment.

1.2. Why change this model?The SQA independent model served a useful purpose for a while. Measurements showed

that the initial rejection rate by the SQA group was higher than expected and that productsrequired several rounds of SQA testing before the release criteria were finally met. Thecompany would miss announced release dates causing loss of customer goodwill. SinceSQA found bugs late in the process, fixing them took a long time—thus posing additionalobstacles to early release of the product. This approach would sometimes require ‘specialreleases’, called ‘engineering releases’, to a selected number of customers with specificproblems. These activities added to the overall workload of the development organization,potentially delaying their future releases—an outcome unacceptable to all parties involved.

An analysis of the process revealed several root causes of the delays. The first was thedivision of responsibilities for product testing. These responsibilities were divided betweenthe development and the SQA groups and were specified in the test plan. Although testplans were in place, not every component of the plan was closely monitored in execution.As development groups reached their scheduled due dates, or when the product develop-ment fell behind schedule, it became too easy for the developers to cut the test cycle—a practice that is frequently cited by other companies and other reports (Gustafson andKerr, 1982; McGregor and Korson, 1994; Rettig, 1991). SQA was sometimes perceivedto be the group that wouldinsert quality into the product—an ill-formed notion.

The second reason for the failure of the process was the lack of communication betweengroups. The development group submitted software to RM (an internal release) andreceived test tools and test suites from the SQA group. There were no specific ‘internalrelease criteria’ in place that would enforce a completed release to RM. SQA wouldproduce and distribute test results and QA reports that were returned to the developmentgroup for immediate action. These communications were often incomplete, and frequentmisunderstandings would occur, often resulting in missing files, missing test suites/cases,and/or incomplete or imprecise information. Miscommunication among groups could beattributed to the lack of team spirit in the process. Groups performed their taskssequentiallyand in isolation. Once the product was released to the SQA group, the development teamfelt detached from the process; similarly, the SQA group would not totally feel part ofthe product team until the product was formally released to them. Direct communicationduring the test cycle was infrequent at best and usually occurred only for negative reasons.

The third reason for the failure of this model was the sequential processing of theentire development cycle. Once the development group handed the product to RM for theSQA phase, the developers were assigned to the next project. When the SQA report wasfinally returned to the development groups, the team members were not always availableto fix the bugs. Moreover, if SQA testing took too long (mostly due to backlogs), some


182 M. GHIASSI

history would get lost and the project was no longer fresh in the developers’ minds.Since this process found bugs toward the end of the development process, the bug fixrate was very low (Table II).

The fourth reason was that all groups involved in testing would be more likely to runthose test tools that were totally automated. This introduced an unintended test duplicationthat would give the impression of extensiveness in testing without any recognition of theredundancy. Each group expected that another would run the less automated, moretedious tests. These were routinely left to SQA, who would often find additional bugs(Rettig, 1991).

Finally, existing configuration management tools were limited to the UNIX SourceCode Control System (SCCS) and were inadequate for use by large groups. Informationwas often lost during communication between groups, since not everything was under theSCCS control (such as test suites and documentation).

As a result, the existing development model had become unacceptable. Managementwanted a development model that would provide improved product quality and, equallyimportantly, improved product release predictability.

2. NEW COOPERATIVE PROCESS DESCRIPTIONThe new process was initiated in response to management concerns with product qualityand release predictability. As previously mentioned, more responsibility was needed onthe part of the developers for the quality of their code, including better testing. The newprocess also had to make it possible to discover bugs earlier in the development cycle(Humphrey, 1989; Hogan, 1993). Since the test suites would remain mostly unchanged,there had to be a clear, well-defined process for testing that specified what would betested, how to test it, and who would do the testing.

In particular, management wanted a process that would allow monitoring and auditingof the entire testing process in order to ensure complete and full execution of all testsuites at each stage. To operationalize their objectives, management altered the SQAorganization, the software development process, the ownership of the testing process, andthe necessary software organization infrastructure to support these objectives. The inde-pendent SQA group was restructured. Three quality engineers were transferred to thedevelopment groups (one quality engineer per product). Each of these engineers becamean additional test engineer within the development organization. The other six qualityengineers and the SQA manager remained in the SQA group. Three were assigned todevelop new tests and to automate the test suites. The remaining three were given a new

Table II. Bug fix rate comparison

Old process Old process Bugs New process New process Bugsnew bugs new bugs fixed new bugs new bugs fixed

Product found fixed (%) found fixed (%)

C compiler 2 0 0 9 7 77C11 compiler 14 2 14 10 8 80Debugger 31 18 58 20 19 95



role and the title of ‘SQA lead engineer/auditor’ with the primary responsibility ofoverseeing, coordinating and auditing the entire testing process. The staffing of theengineering organization remained the same, although the distribution of engineers betweenthe development and SQA organizations had changed.

Additionally, the organization acquired software tools necessary for improving softwaredevelopment infrastructure, including a new configuration management tool, RationalSoftware Corporation’s ClearCase (Rational, 1998). The entire engineering organizationwas trained in the use of these tools, and a formal document control system wasestablished under ClearCase.

Process changes were accomplished by creating additional check points (milestones)and by making the auditing tasks independent of the execution tasks. The additional checkpoints are Functionality Complete (FC) and Validation Complete (VC) milestones. TheSQA organization was assigned the role of independent auditor.

As in the old model, a new release is initiated in response to maintenance and/or newfeatures. A requirements specification defines precisely and clearly what will be containedin this release. In general, the requirements list includes the addition of new features, bugfixes and product enhancements. Although each release has its own unique set of tasks,management believes that the workload for subsequent releases are ‘roughly’ equivalent,in particular when considering stationary staffing levels and an evolutionary developmentmodel. When deciding on the size of each workload, each requested feature is designedand costed out. Any bug fix that can potentially impact the schedule is given specialattention. All tasks are then prioritized. Once the potential product changes are time andcost estimated and prioritized, they are scheduled. Some work will require specific expertisefrom certain individuals, while other tasks can be performed by any member of thedevelopment team. A project manager evaluates the team’s strengths and weaknesses andexploits them to maximize productivity and efficiency. The management knows the teamcannot possibly deliver all the requested features and bug fixes in one release, given thetime-to-market objective. With the exception of emergency features (such as those requiredby unforeseen market trends), new features are planned with a long-term strategy in mind.Managers commit to deliver several incremental releases of short, predictable duration(usually six months). They manage the requirements and priorities to create partitioningof tasks that can fit their currently available resources. In short, managers break theworkload down and make it fit into what appears to be ‘roughly’ equivalent workloadsper release. This document serves as the starting point for both the developers and thetesters (Figure 1).

The newly established cooperative testing process has added one additional documentto the requirements specification: the‘Release Criteria’. This document formally definesthe set of adequacy criteria necessary for obtaining ‘functionality complete’ (i.e. whendevelopment testing is done and full testing can begin—internal release criteria) and‘validation complete’, and other information pertinent to product release, such as schedules.Note that testing time becomes a formal part of the product development schedule. Theproject test plans were also enhanced to describe in detail the test suites that would berun, the combination of options to be used, the platforms, and versions of the operatingsystems. The test plan further outlines the responsibilities of each organization at everystage of the testing process and when each test suite was to be run. Completed test plansare reviewed by representatives from Engineering, Marketing, Technical Support, and bythe SQA group (the SQA lead engineer/auditor). These documents are then signed and


184 M. GHIASSI

placed under formal documentation control to enforce their integrity. A change can bemade only if acceptable to all parties. The functionality complete milestone cannot bereached if the agreed-upon requirements are not complete. Typical requirements for FCinclude all planned bug fixes and coding and testing of new features. Testing by thedevelopment group now includes complete unit tests, regression and integration testing,and a ‘clean build’ of the product. All sources (code, tests and documents) must be undersource control management and labelled. After FC is reached, the full product testing,called Validation Complete phase can begin. At this point, the label on the source codeis locked. This is to insure that all code remains untouched during this phase of testing.Figure 3 depicts this new process with the region between A and B corresponding to the‘Testing and SQA phase’ of Figure 1.

As part of the validation complete phase, the developers execute the entire test planand analyse the test results. During the VC phase, testing focuses on product integrationand functionality. The development team now runs all the existing automated regressionand integration tests, and analyses the results (in the older model most of this testing wasdone by the SQA team and independent from the developers). By shifting these testactivities, the new process places the responsibility for product quality back on thedevelopment team. Since the developers have just finished writing this code, it is freshin their minds, so fixes are faster and easier. Monitoring of the testing process andevaluation and certification of the results are the responsibilities of the SQA lead engineer.This person ensures that all tests (as described in the test plan) are executed to completionand their results analysed. The SQA lead is also responsible for attending to any errorsfound in the test suites and documenting all problems and fixes both in the product andin the test suites.

To keep communications open, the test team (composed of the developers, other testersand the SQA lead) meets regularly to discuss the status of their tests. After all tests arerun and all results are analysed, the team meets to decide which of the new bugs needto be fixed and which should just be documented (as before, all critical bugs are fixed).The developers perform code changes, code review them, and the team then decides whatre-testing needs to be done to test the changes. The process allows testing/fixing/re-testingto occur when necessary. If no new bugs are found during the re-testing, the validationcomplete phase is considered complete and the product proceeds to the release phase.

2.1. Impact on the role of the developers and SQA engineersUnder the old model, the development team handed the software over to SQA for

testing. Since the test process was time consuming, the development team was usuallyassigned to another project. When SQA issued its report, seldom was the entire teamavailable to work on the fixes. Those members who ended up back on the project neededsome refocusing/relearning before they became effective again. In the new process, theentire development team stays with the project performing the testing and the fixing ofbugs; hence, the developers maintain their knowledge of project details and do not requireadditional time to remain current.

The new process has had a significant impact upon the role of SQA. For example,now that the tedious job of executing and interpreting results is assigned to the developmentgroup, there is more time available for improving and developing test cases. As developersare charged with testing, the SQA staff has the time and priority to develop, acquire, and



Figure 3. Cooperative software testing model.

introduce additional test suites and tools that will make the entire testing process moreefficient and effective.

As testing teams become settled, the role of SQA is seen somewhat less antagonistically.It is believed and hoped that SQA will be viewed less as a roadblock and more as ateammate in the effort to ship a quality product.


186 M. GHIASSI

3. NEW MODEL EVALUATION

The new model was introduced to improve both the quality of the products and theproductivity of the entire engineering organization. In order to evaluate the effectivenessof the new processvis-a-vis the old, a few metrics have been introduced for outcomesmeasurement. However, establishing causal relationships between individual processchanges and observed outcomes is difficult in most organizational settings, and the complexactivity of software development is no exception. Furthermore, the focus of this researchhas been the analysis of the process of testingin an industrial environment. In suchenvironments, when a new process is introduced, new tools are often obtained to supportthe new activities. Since control and management of the test process was a paramountobjective, better configuration tools capable of supporting all activities of the new processwere acquired to ensure that the new system would proceed successfully. These toolsmay also play a role in any observed improvements. In interviews with the management,however, they stated that ‘addition of the new configuration management tools allowed(a) its use for some more complex tasks, and (b) provided management and the engineeringteams with an easy-to-use tool for creating and maintaining various product and processdocuments. These tools, however, did not provide any significant additional capabilitiesto improve product quality’.

The following metrics are defined to assess the relative effectiveness of the new process.

3.1. Bug fix rate

One major difference between the old and the new process is the timing of the producttesting. In both processes, unit testing is performed by the developers. However, in theold process most of the regression and integration testing was performed late in thedevelopment process and by the SQA team; this approach made bugs discovered duringthis phase very expensive to fix. Moreover, fixing bugs late in the process increases therisk of introducing new bugs. In the new process, the development team conducts regressionand integration testing, so bugs are found earlier and most may be fixed immediately.

Table II presents bug data for three products (a C compiler, a C11 compiler, and adebugger) and shows the number of bugs found and fixed (before the products werereleased) for each of these products. As the data indicate, when comparing two releasesof these products, a much larger number of discovered bugs were fixed in the new modeland the number of remaining bugs diminished between releases. As discussed earlier, theremaining bugs are listed and documented in the release notes. As the process is furthertuned, greater improvements in both bug discovery rate and bug fix rate are expected.These improvements obviously would contribute to both improved quality of the productsand more predictable development schedules.

3.2. Quality improvements

In order to assess the effectiveness of the process change on the quality of the product,field bug reports were monitored for up to two years after product release (subsequentreleases usually make these versions obsolete after two years). Bugs found and reportedfrom the field are verified by the technical support team and are logged as new bugs



Table III. Field-reported defects (2 years)

Old New ImprovementProduct model model (%)

C compiler 158 115 27C11 compiler 132 58 56Debugger 40 24 46

against the released product. Table III shows the number of field-reported bugs during atwo year field period for these products, a measure of quality improvement/degradation.The last column in Table III reports the percentage reduction in bugs in moving to thenew process, indicating a rather noticeable improvement in quality. The last column ofTable III shows improvements of 27 and 46% for two products (C compiler and debugger).The improvement in the C11 product was 56%. Table IV presents field-reported defectsper KLOC (un-commented) for the same time frame. This table also shows improvementsof 32 and 54% in field-reported bugs for the C compiler and the debugger, and 59% forthe C11 compiler in the new model.

Differences in the rate of improvement of the different products are partially due tothe nature and maturity of each product. The C compiler represents a relatively maturedevelopment language that is used by experienced embedded programmers, and is subjectto supporting both newly developed applications as well as existing ones. This product isthe most widely used of the three products. Testing of this product benefits from theavailability of standard industry test suites, as well as comprehensive test suites that aredeveloped in-house. Field-reported bugs found in this product can be attributed to the useof complex programming constructs in real life applications. These bugs were harder todetect, analyse and fix than those of the other two products. Lower improvement rates inthe field-reported bugs, therefore, can be attributed to the relative completeness of the in-house test suites for this product. The debugger, on the other hand, is a supporting toolused during the development cycle. This product is the flagship product of the companywhich has had the highest rate of code reuse between subsequent releases over the pastseveral releases. Although this product is complex, its users are more tolerant of bugsthan are those of a compiler. Developers will often find workarounds or find alternativemethods for debugging their code, when facing a debugger malfunction, rather than filinga bug report against the product. These two factors may explain why the debugger hasthe lowest field-reported defects per KLOC. The C11 compiler was a relatively newtechnology and a larger portion of the activities were new feature development rather

Table IV. Field-reported defects/KLOC (2 years)

Old New ImprovementProduct model model (%)

C compiler 1.121 0.762 32C11 compiler 0.729 0.299 59Debugger 0.263 0.142 54


188 M. GHIASSI

than bug fixes; additionally, early versions of a new technology tend to be more buggy.Applying the new testing process seems to be even more effective for new technologies.This can also be explained by the fact that more bugs were found during the testingcycle, and more of those bugs were fixed early. This information also confirms that thenew process improves the overall quality of the software.

3.3. Improved working relations

Once the new process was established, the antagonistic relationship between SQA andthe development group slowly faded away. The primary reason related to the ownershipof the quality of the products. The responsibility for quality in the new model was placedsquarely on the shoulders of the development team and not distributed across thedevelopment and SQA organizations. Improved infrastructure along with the increasedoverlap between the development and testing efforts gave developers more reasons tospend the time necessary to perform thorough testing and achieve higher quality. TheSQA was now perceived as part of the team (and not as the opposition), and finallytesting became a well-defined activity with its own milestones and schedule. The develop-ment team was now the owner of the testing process. This clarity removed time pressurefrom the testing phase.

3.4. Release date predictability

The starting point for a new release of a product is the same for both the old andthe new processes. During each product cycle the set of known bugs and the list ofrequested features were compiled. The development group would examine this list, andagree with the management team on a set of requirements.

Under the old process, as the deadline for release neared, often one of two difficultscenarios would unfold. In the first scenario, external commitments encouraged thecompany to release the product on the previously committed date, often reluctantly andwithout the desired level of testing. The development team often had to balance testingtime against the addition of new features. Two important causes of this problem were:(1) problems would tend to surface toward the end of the timeline; and (2) the overallpredictability of the delivery schedule was very poor in the first place, since managementcommitments were often guesses, not based on reliable historical or technical data. Whenunforeseen problems were encountered, the feature/bug list was pruned.

In the second release scenario, the product would be fully tested. However, since thetesting was back end loaded, with SQA performing independent testing, the product wouldgo through several rounds of testing. Additionally, inherent delays associated with eachinternal delivery after each round would further postpone the final release of the product,making the prediction of release dates a guessing game.

The new process identifies problems well ahead of the scheduled release, thus givingmanagement time to react. Testing is ‘in-lined’ so problems are identified before the FCstage. In-lining of the test process ensures the quality of the released product. Testing isdone during development, and whatever is coded is also adequately tested. Meeting FCadequacy criteria significantly reduces the risk and uncertainty of the remaining activities,thus allowing management to estimate more accurately the release dates of the products



and, when necessary, still have time to notify sensitive/crucial customers of possibleimpending delays.

In the new process, no external date is given until the product reaches FC. Once theproduct passes this phase the remaining activities become more predictable. This pre-dictability allows management to quote and maintain product release dates.

3.5. Difficulties and lessons learnedThe most difficult hurdle to overcome during the initial weeks of the new SQA

phase was changing the working relationships between the project managers and the SQAlead. One of the first things that seemed to happen was that project managers wanted todecrease the amount of testing to be done. Even though the test plan was agreed to, therewas ‘new evidence’ that certain tests would not find any problems and should beeliminated from the test plan. The SQA lead was now in the position of having to managethe project managers. Several conflicts arose as project managers felt that they shouldmake these decisions and not have to consult the SQA lead. Upper management had tointervene in these conflicts and empower the SQA lead with the responsibility to makethese decisions.

Test suite management became an issue as well. Since more developers were using theSQA provided test suites, more bugs/feature requests were submitted against these testsuites. Since many product groups were using the tests concurrently, it became difficultto schedule a time to make changes to the common code of the test suite withoutimpacting someone else’s testing. Other issues arose having to do with interpretation ofparts of the process. For example, a clean build was required in order for a product tobe declared at FC. However, ‘clean build’ was not formally defined. These issues, whichwere more prevalent during the initial implementation of the new process, were addressedand resolved in successive implementations.

Another challenge was the change in responsibilities of the SQA lead. The SQA leadwho had been mainly responsible for testing a particular product now had to makedecisions regarding several different products. Although other SQA group members wereavailable for consultation in their areas of expertise, some decisions were required on thespot. This tested the credibility of the SQA lead.

The job of the SQA lead changed in other ways as well. There were now many moreadministrative tasks (e.g. keeping track of each product in the toolkit and its associatedproblems, fixing bugs in the test suites, and auditing the results of the test cases). Theamount of independent testing done by SQA in the new process is much less than in theold. However, more interactive and usability testing is now done by SQA than before.

Most of these difficulties are being addressed. For example, putting the test suites underan advanced source management system alleviated some test management problems. Atechnical writer was used to document the new process fully to prevent misunderstandingsand misinterpretations. The solutions to other issues will most likely develop over timeas experience using the new process accumulates.

4. CONCLUSIONThis paper has described a change in testing process in a mid-sized software company.The key parameters of the process were quality ownership and when regression and


190 M. GHIASSI

integration testing were performed and by whom. This study recommends a new processin which the quality ownership is centralized and integrated within the developmentorganization. In this case study, transition to the new process resulted in improved bugdetection and bug fix rates during testing. The number of bugs detected in the field alsodropped significantly. This process has produced higher quality software products with amore predictable release schedule.

Other elements contributing to the success of this case study were the availability ofautomated test suites and a development environment infrastructure that supports productand process document control systems. This process has given managers better control ofthe entire product development process, resulting in a more organized and communicativesoftware development environment.

Finally, although the data in this case study strongly supports a shift to this cooperativeprocess, further research is needed to validate statistically the conclusions reached.

AcknowledgementsThe author would like to thank the anonymous referees for their suggestions and Lura Tamshen forher contribution to this research.

ReferencesDaly, E. (1977) ‘Management of software development’,IEEE Transactions on Software Engineering,

3(3), 229–242.Ghiassi, M. (1985) ‘Software quality: design it in from the start’,Computer Design,23(9), 91–97.Gustafson, G. G. and Kerr, R. J. (1982) ‘Some practical experience with a software quality assurance

program’, Communications of the ACM,25(1), 4–11.Hogan, G. W. (1993) ‘Testing positive: a testing strategy for the 1990s’,Proceedings of the 10th

International Conference and Exposition on Testing Computer Software, July, Washington, DC,pp. 29–35.

Humphrey, W. S. (1989)Managing the Software Process, Addison-Wesley, Reading, Massachu-setts, U.S.A.

McGregor, J.D. and Korson, T. D. (1994) ‘Integrated object-oriented testing and developmentprocesses’,Communications of the ACM,37(9), 59–77.

Plum (1994)The Plum Hall Validation Suite for ANSI C, Plum Hall Inc.Rational (1998)Beyond Version Control, Rational Software Corporation, Cupertino, California, U.S.A.Rettig, M. (1991) ‘Testing made palatable’,Communications of the ACM,34(5), 25–29.


Documents

Toward a cooperative software testing process