White Paper

This document and all its contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. Any unauthorized disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. © Cybage Software Pvt. Ltd

AUTOMATION TESTING: INVESTMENT MINIMIZATION STRATEGIES A Cybage perspective

Abstract Software testing is getting more and more complicated. Multiple factors are responsible for this situation. User Interfaces (UIs) are getting richer and complex, development cycles are shrinking, architecture complexity is increasing, and the number of devices and platforms one needs to test is increasing. And amid all these constraints, the timeline needed to assure the quality of the system is getting shorter. All these factors necessitate the employment of a test automation strategy that can satisfy these constraints and fulfill the required quality objectives. Automated testing, when implemented in the correct manner and purposefully, allows an

organization to use tools to speed up the testing that happens manually; and also perform certain testing which cannot be performed manually in a quick, repeatable, and efficient manner. Implementing automation testing comes with its own challenges; and in a majority of instances, requires considerable investment and efforts. This whitepaper discusses the typical automation testing processes that are used by organizations; the cost factors involved in implementing automation; lays down certain strategies that can be utilized for minimizing the investments; and maximizing ROI.

© Cybage Software Pvt. Ltd Page 2 of 8

Table of Contents 1. Introduction .......................................................................................................................................... 3

2. Automated Testing: Cost Factors .......................................................................................................... 3

3. Conclusion ............................................................................................................................................. 7

4. Cybage Test Automation Practice ......................................................................................................... 7

© Cybage Software Pvt. Ltd Page 3 of 8

1. Introduction

‘Time is money’ may be an old adage, but it is still highly relevant. Businesses today create applications of increasing complexity, and operate under tight time constraints. Technological advantage has become passé. ‘Time’ has become a dominant factor. An organization that can release a product to the market faster than its competitors has more certain chances of becoming the leading player in its segment. The time invested in a product or project can be divided roughly into analysis and design, and coding and testing. Empirical statistics suggest that close to 40 per cent of the total product development time is spent on testing. This period is in no way insignificant. Cutting the testing time allows the product to be released that much faster in the market. Imagine the difference this can make for an organization! It’s better ROI all the way. Automated testing is one of the key drivers that can help organizations in speeding up their testing processes. Automated testing involves using a combination of tools and/or technologies that, in its simplest form, performs the actions and verifies the state of an application; imitating the steps a manual test engineer would perform. Automated testing is also employed to perform testing which would be difficult and challenging to execute manually. The biggest benefit that automated testing brings is that once the manual tests are automated, they can be executed repeatedly and rapidly; especially during regression testing. This quick feedback on the state of the overall quality of the application can be extremely valuable in assessing the release-worthiness and identifying erroneous code at the earliest possible stage where it can be corrected with the least possible cost. Among all the strategies to shorten testing time, automated testing holds the most promise. This paper attempts to delve more into the area of employing automation in the QA domain, but in a cost-effective manner and with minimal investment.

2. Automated Testing: Cost Factors

Deploying automation in an organization necessitates investment. These investments can be broken down into two distinct categories—fixed costs and variable costs. Fixed costs do not vary on the basis of the number of test scripts developed. In contrast, variable costs have a direct co-relation with the volume of scripts developed, as well as the number of times they are executed.

The Gartner Group has

estimated that “Companies

spend upwards of 30%

of their annual IS budgets

and 60% of their annual

software development

budget on software

testing”.

© Cybage Software Pvt. Ltd Page 4 of 8

Major fixed cost factors: • Hardware costs • Automation tool license

costs • Automation framework

design and development costs

Major variable cost factors: • Script development costs • Script maintenance costs • Execution costs • Result analysis and defect

reporting costs Each of these cost factors exerts a substantial pull on the required investment; and while it is not possible to completely do away with most of them, an organization can employ certain strategies to minimize the investment that a test automation exercise entails. The remainder of this whitepaper will be devoted to describing the same.

Major Fixed Cost Factors

1. Hardware costs

One needs machines to execute the automation scripts. Depending on the pace at which automation scripts are expected to be executed, grid-based environment can also be considered for running the scripts in parallel on a number of machines simultaneously. From a cost-minimization perspective, existing machines that are used for manual testing can be employed for automation runs too; and more so at non-working hours such as nightly runs. To further reduce capital expenditure costs, cloud-based solutions can be considered. Cloud-based solutions are especially useful as they allow as many machines to be utilized as required, with the cost being proportional to the usage; saving tremendous amount of time and giving quick feedback on the health of the system.

2. Automation tool license costs Automation tools play the most significant role in any automation strategy. Often, the architecture, the technology stack, and the automation requirements of the application will dictate the choice of the automation tool that would be required. Licensed tools such as HP Quick Test Pro and VSTS are usually expensive. They come with different licensing models such as volume-based licenses and per seat licenses. Besides the one-time purchase cost, they also necessitate annual renewal of support and licenses to continue using the latest versions of the automation tool. These tool-related costs could be rather substantive sometimes and make up a large chunk of the overall investment required.

© Cybage Software Pvt. Ltd Page 5 of 8

A better strategy to employ here would be to check the feasibility of using open-source automation tools such as Selenium, FitNesse, and Watir. Some of these open-source automation tools are very stable and are used extensively by some of the biggest enterprises in the world. They also have a wide and dynamic community of practitioners who provide free support on the web and contribute extensively in developing the capabilities and features of the tool regularly. Cybage has been at the forefront of utilizing open-source automation tools across a number of projects and this approach has resulted in substantial cost savings for its clients.

3. Automation framework design and development costs Any automation tool will mostly provide the vanilla constructs required to automate the testing. These basic constructs include features to accomplish:

Record and playback of scripts

Develop scripts manually

Store the applications object repository or page object model

Provide a construct to allow tests to be executed with different data sets, and so on.

However, advanced features such as scheduling of tests on grid environment, recovery scenarios to execute in case of unresponsive test runs, ad hoc reporting, and codeless test scripting, are not provided by any tool. Rather, these features are left for implementation on the basis of specific requirements that an organization may have. An automation framework is what fills the gap and provides these missing features. It is not uncommon to see every automation project spend a good amount of cost and efforts in coming up with an appropriate test automation framework. The more sophisticated the automation framework is, the easier it will make the job of executing and maintaining the scripts. Conversely, sophisticated automation frameworks are more complex to develop, and require higher development, debugging, and testing efforts. A better way to handle this scenario would be to make use of a reusable test automation framework that can be utilized in tandem with the selected automation tool. A reusable test automation framework will bring in sophisticated and advanced capabilities that will make it faster and easier to deploy automation.

© Cybage Software Pvt. Ltd Page 6 of 8

Major Variable Cost Factors

1. Script development costs

The more automated tests one is able to develop and execute, the better would be the Return On Investment (ROI). The rate of development of test scripts is linked with the complexity of the required coding. Typically, developers who are conversant with the underlying programming language of the automation tool would be entrusted with the task of creating the scripts; and they tend to be lesser in number and more expensive than engineers having only manual testing skills. Extending this perspective, if the automation framework allows test scripts to be created by even manual testing, without necessitating them to have special programming skills, then the organization will see lowered script development costs, and quicker development of scripts based on the increased pool of engineers. This is possible to achieve only in scenarios where the automation framework has been developed to offer this ability to create ‘codeless scripting’. Advanced techniques such as keyword-driven testing and adoption of a common test case language while creating manual test cases, will make it easier and profitable to develop automation scripts rapidly.

2. Script maintenance costs During the lifecycle of a software project, the application undergoes a series of changes in the user interface and its functional aspects. Locations of controls such as push buttons, and radio buttons, may change. The label text used to identify controls could change; perhaps new controls could get added in the existing flow. These changes can be the result of new implementations, defect fixes, or feature updates. Whenever these changes occur at the application level, the respective automation scripts also need to be updated to ensure correct operation. Empirical evidence suggests that failure to modify scripts to keep them up-to-date is the single biggest reason for failure to use automation effectively. Depending on the way the automation framework has been built, making changes to existing scripts can either be a long-drawn and tedious process {where one change needs to be done manually in all the scripts where it is referenced} or an extremely quick task where change is needed to be made at only one place and all scripts get updated automatically. This capability is again required to be developed in the automation framework. Once it becomes available, an organization will tend to use automation more frequently since it would be up-to-date; and also find it easier to keep the scripts in sync with the changing application.

3. Execution costs In ROI calculations, the greater the number of times one is able to execute the test scripts, the more one is able to realize the return. Hence, in order to maximize this return, the automation runs should be quick and hassle-free. One also needs to consider scenarios in which a certain script may not execute correctly and could become unresponsive. This unresponsive behavior could manifest due to multiple reasons such as browser and machine not responding, and disk space getting exhausted. If the automation gets stuck while waiting for this script to get completed, then it would lead to unnecessary wastage of time and require constant manual

© Cybage Software Pvt. Ltd Page 7 of 8

monitoring for alleviation. This indirectly leads to decreased execution runs and increased execution costs. A recommended approach would be to provide the framework capabilities that would allow automatic scheduling of test runs along with monitoring capabilities that would let it identify stuck or non-responsive scripts, and take corrective action immediately. This would reduce manual intervention and allow automation scripts to be easily executed.

4. Result analysis and defect reporting costs When automation scripts fail, typically an engineer would be required to manually go through the test results, analyze the failures, and in case of valid defects, raise a ticket in the defect tracking tool. This exercise does take considerable efforts, particularly when you consider the frequent test runs that happen, and the huge number of scripts that projects typically have. The strategy recommended here, to minimize the efforts and subsequently the costs, is to have the automation framework highlight the test failures; and provide information which will make it easier for failures to be analyzed and defects to be logged; and in some cases, even log the defects automatically.

3. Conclusion Test automation offers a lot of benefits and has the potential to positively and radically impact the testing process of an organization. But it is also an expensive proposition to design, develop, and maintain. As a process, some areas in the test automation lifecycle can benefit from smart strategies that can be utilized to minimize the investment required, and positively impact the ROI.

4. Cybage Test Automation Practice Cybage Software has been at the forefront of utilizing test automation for its demanding clientele who have varied automation requirements. By virtue of its experience and expertise, Cybage has institutionalized the most optimum processes to be used across the entire automation lifecycle to minimize client investment, and maximize the return. Cybage has also developed its own state-of-the-art test automation framework that is reusable across projects and boasts of features such as scheduling, monitoring, codeless scripting, grid-based parallel execution, recovery scenarios, and comprehensive reporting among others. Salient features of the Cybage Test Automation Practice:

Experience in designing and implementing test frameworks (Modular, data-driven, keyword-driven, object repository, and descriptive, among others).

Technology expertise for implementing custom automation harness and utilities in different languages such as C#, Java, Perl, and Python, among others.

Proven expertise in industry’s leading testing tools such as HP QuickTest Pro, WinRunner, Rational Robot, and SilkTest.

Strong capabilities in emerging open-source testing tools such as FitNesse, Selenium, and Watir, among others.

Strong technology leadership and consultancy by QA industry veterans.

© Cybage Software Pvt. Ltd Page 8 of 8

About Cybage Software Pvt. Ltd. Founded in 1995, Cybage Software is a leading offshore software services company offering solutions that accelerate, simplify, and enrich business processes to give its clients an edge over competition. We are an SEI CMMI Level 5 version 1.3 and ISO 27001 appraised company based in Pune, India. Our success is built on a pool of 4,300 software professionals. Based on a remarkable record of quality, consistency, and outstanding technological prowess, we have partnered with more than 200 global software houses of fine repute. Our array of services includes Product Engineering (OPD), Enterprise Business Solutions, and Value Added Services. Cybage specializes in the implementation of the Offshore Development model.

The Cybage domain expertise spans across several business verticals such as Media and Entertainment, Travel and Hospitality, Healthcare and Life Sciences, Retail and Distribution, and Hi-Tech. Cybage has eight technology-focused Centers of Excellence (CoEs)—E-commerce, Enterprise Mobility, Customer Relationship Management, Business Intelligence, Enterprise Content Management, Cloud Computing, Education and Learning, and Supply Chain Management. Our unique model of operational efficiency, ExcelShore®, helps de-risk our approach and provides the best value per unit cost.

To know more about us, visit us at www.cybage.com or write to us at biz@cybage.com.