How to increase the efficiency of manual testing

Business white paper

Table of contentsExecutive summary ..................................................3Introduction ............................................................3The real challenges of manual testing .........................5Reporting a defect ...................................................7Data injection .........................................................8Mirror Testing .........................................................9Reporting .............................................................10Conclusion ............................................................ 11

Even though nearly every QA team in the world still relies on manual testing, the actual process of manual testing hasn’t improved much in the past 10 or even 20 years.

3

Executive summaryManual testing remains the most popular method for validating the functionality of software applications. It is simple, straightforward, and doesn’t require advanced technical knowledge or programming skills. However, as technologies have progressed and applications become more complex, the process of manual testing has stayed mostly unchanged. It is still a very resource-intensive and often repetitive effort that requires monotonous reiteration of the test steps and manually entering large sets of data. Manual testing can also be highly error prone, resulting in overlooked defects and false positives. Over the years, several solutions have been introduced to the market to help run manual tests, but they still failed to solve the fundamental problems that plague most manual testing teams: data-driven testing, accurate defect reporting, better communication with developers, and replaying the same tests in multiple environments.

This paper explores the challenges of traditional manual testing methods and discusses the advantages of HP Sprinter— a unique solution for maximizing the efficiency and precision of manual testing, streamlining communication between testing and development teams, and reducing test cycles.

IntroductionApplication testing has been around for decades. Ever since the first software applications were built, there was a need to validate an application’s functionality before releasing it into production. The original validation method was manual testing. A software tester—or sometimes a business user—through a series of steps, would interact with an application and document any problems encountered along the way. Naturally, this process was often monotonous, time consuming, and error prone.

Advances in technology brought new ways of application testing, including automated functional testing. At first glance, automation looked like the answer to all of the problems with manual testing. It didn’t rely on a business user to execute repetitive test steps; it was scalable and reusable; and it could run tests all night, unattended, and test different business processes with various sets of data. However, QA teams quickly realized that not all applications are good candidates for automated testing, and that the best testing strategies always involved a combination of automated and manual testing elements.

Manual testing is here to stay. There will always be applications or parts of applications that should be tested manually by a business user who is familiar with

4

the business process flow. And there will always be times in the application lifecycle where quick manual exploratory testing will provide just enough information for developers to capture early defects, before they turn into larger problems downstream. But even though nearly every QA team in the world still relies on manual testing, the actual process of manual testing hasn’t improved much in the past 10 or even 20 years. It is still a very resource-intensive, time-consuming, and often tedious effort, which involves endless repetitions of the same steps, and copying and pasting countless sets of data. Several test management tool vendors have offered manual test runners in the past, but business users are reluctant to accept them. Most testers still prefer to print the test steps from the tool’s repository and continue to run tests in the old way, often losing the traceability of the test execution results.

Most importantly, traditional manual runners don’t solve the main problems that plague manual testing: data entry, inaccurate and inadequate defect reporting, and replaying tests with multiple configurations.

HP sets out to change the way QA organizations view manual testing. HP Sprinter software is a truly revolutionary approach to all of the aspects of manual testing: from maximizing the “real estate” on your screen for easier interaction with an application, to accelerating the capturing and reporting process of relevant defects, injecting test data, and even replicating test steps across multiple environments on remote machines. Intuitive and easy to use, HP Sprinter is one of the only solutions that helps preserve the benefits of manual testing, while significantly improving its efficiency, accuracy, and speed.

Figure 1: HP Sprinter features an easy-to-use, unobtrusive interface

Test data entry can now be completed in a matter of seconds, significantly reducing test execution time and helping organizations release applications faster.

5

The real challenges of manual testingThe advantages of manual testing are undeniable. It doesn’t involve complex preparations, script recording, or coding. It can be performed by a business analyst or any other application end user without requiring any particular technical knowledge, specialized tools, or intricate setup. Manual testing can be run at any stage of an application lifecycle, making it even more popular with agile development teams.

But traditional manual testing has some very serious drawbacks as well. Not only is it monotonous and repetitive, it can also be highly error prone, where a missed step or incorrect data can lead to overlooked defects and false results. Data-driven manual testing also presents a considerable challenge: To validate the

application’s functionality with different sets of data—such as login names or credit card numbers—testers usually print out a spreadsheet with available data and then manually enter this information into an application. Alternatively, they may toggle between an open spreadsheet and the application under test, copying and pasting data into required fields. But when schedules get tighter and testers are under pressure to finish validating the application’s functionality before the release, they often resort to entering bogus data into form fields, missing potentially critical problems.

It is also extremely difficult to achieve good coverage of different environments and platforms when testing manually. Testers simply don’t have the time to repeat the same steps over and over, on different machines running various combinations of

Figure 2: HP Sprinter offers multiple ways to log defects to give developers detailed information about the issue

“Sprinter’s Mirror capability allows for much faster (>80% productivity) Web application Browser Compatibility testing”

Anonymous customer, TechValidate survey, October 2010

6

browsers and operating systems, to look for differences in application behavior under diverse conditions. QA managers end up choosing a single—and typically the most common—platform to test the application on, ignoring other possible combinations and use cases.

Finally, when defects are found, manual testers typically don’t have the tools or skills to properly communicate the problem. As a result, the developer who is investigating a particular defect has to try and reproduce it with only minimal information. If the developer knew which steps the tester took before arriving at a problem, which conditions the application was tested under, or what data was entered, the resolution time could have been much faster. Instead, developers often have to rely on poorly constructed and incomplete defect records, which cause unnecessary delays and frustrations. This is especially true in agile

exploratory testing, where testers “poke around” in an early build of an application without following any formal written test steps or procedures. If a defect is found, testers often cannot even recall which path they took through an application to arrive at that point. Naturally, recreating such defects is a challenging task for R&D.

Introducing HP Sprinter: a revolutionary approach to manual testing HP Sprinter is a unique, innovative solution designed to increase efficiency, coverage, and precision of manual testing. A component of HP Application Lifecycle Management (ALM) and HP Quality Center (QC), HP Sprinter is an easy-to-use manual testing tool that features an intuitive, unobtrusive interface and a series of advanced features to help testers improve testing speed, accuracy, and documentation. With HP Sprinter, testers no longer need to flip between

Figure 3: Data injection helps to save a significant amount of time and helps to reduce errors

7

multiple screens looking for test-case descriptions. HP Sprinter can run as a collection of sidebars on the screen, which can be expanded or minimized as needed. It can also show test steps as one-line subtitles, maximizing the screen “real estate” needed for interacting with the application under test.

HP Sprinter runs on your local computer. And, because it interacts with HP ALM software and HP QC software, it has full access to the tests that have been defined in your test plan, as well as complete information about the application under test. When you first open a test in HP Sprinter, you can select which application you will be testing and review its configuration. HP Sprinter automatically starts the application or browser, and opens the application under test.

Reporting a defectWhen creating a defect report, manual testers typically don’t like to spend time typing up large amounts of data to help developers reproduce the issue. HP Sprinter solves this problem by automatically including an array of relevant information in the defect description, to help illustrate the issue and the actions that led up to uncovering it. HP Sprinter allows you to either submit a defect into the HP ALM or HP QC repository during the test run, or to continue with the run flow and submit a defect when the run is completed. All of the information about steps, user action screenshots, and detailed descriptions that were available during the run is also accessible when the run is finished.

Figure 4: Mirroring allows you to simultaneously test multiple configurations, saving time and increasing coverage

8

The tester can configure which types of information HP Sprinter is going to include into the defect report. You can either submit a Smart Defect that would automatically include all of the selected data, or open a new defect dialog with no predefined content. To help illustrate the problem, you can include a detailed list of user actions—including every button click and piece of data entered—that you took prior to discovering a defect. You can add a short movie replaying all of the user actions and application responses over the past several minutes. You can also choose to include a screen capture, complete with annotations and markups. The annotations feature is also very useful when testing the application’s UI elements. For instance, you can measure the length of a form field using a measuring tool in the annotations workspace.

The ability to attach user actions and a movie of all recent activities is invaluable in exploratory testing and testing in agile environments. Not only can a developer easily recreate the problem, the list of steps can be used later to create a more formal test script.

The intuitive HP Sprinter UI and smart settings allow testers to submit a great

amount of relevant defect information without slowing down the testing process or creating any additional steps. All of the defect data can help testers to better communicate with application developers, and to save the developer’s time in understanding, reproducing, and resolving the problem. Defects submitted through HP Sprinter are automatically linked to the test run and to any associated requirements in HP ALM or HP QC, and therefore create traceability and support the concept of requirements-driven testing.

Data injectionMost test scenarios require testers to fill out long forms, such as entering shipping information or submitting credit card numbers. Data entry is indeed one of the most time-consuming, tedious, and error-prone activities in manual testing. HP Sprinter automates test data entry by using the process of data injection, saving valuable testing time, increasing coverage, and reducing errors. Data sets can be prepared in advance, and can be added from the file system or from the Test Resources module in HP ALM or HP QC. Data sets can be associated with the application under test, and can be used directly by HP Sprinter. Data files can be either in Microsoft® Excel or Comma

9

Separated Values (.csv) format, with the column headers set up to match the form fields in the application under test.

Data injection automatically injects data from the row in the data set into the matching fields in the application, populating the form fields in a fraction of the time it would have taken to manually enter the data. In the next iteration of the test, you can choose values from the next row, validating the application with multiple sets of data. You can also control the order of the data injection, which is especially beneficial in testing applications with dynamic fields. This process, which used to take manual testers several minutes for each form, can now be completed in a matter of seconds, significantly reducing test execution time and helping organizations release applications faster.

Mirror TestingOne of the most significant limitations of manual testing is that there’s never enough time to run tests on multiple machines, using different configurations and platforms.

With HP Sprinter, you can execute tests on your primary machine—such as your local laptop or desktop—and HP Sprinter automatically repeats the exact same test on different secondary machines. This feature is known as Mirror Testing. It allows you to replicate all of the actions that you take on your primary machine on multiple computers with different configurations. This distinctive capability allows QA teams to instantly increase test coverage without having to spend the time and resources manually repeating tests on different browser and OS combinations.

HP Sprinter automatically shows you the list of available machines that are connected to your primary computer. You can choose, for example, to have one machine set up to test the application in Internet Explorer (IE) 7.0 and another machine set up to test the application in Mozilla Firefox or IE 8.0. HP Sprinter connects to and initializes all of the selected machines, and then starts the application and continuously displays health indicators and run status for each machine. Each mouse click, field entry, or page

10

advance that you perform on your primary machine is automatically replicated on each of the secondary machines. All actions are kept in sync, and HP Sprinter compares the results on all of the computers that are running the test. To enhance performance, HP Sprinter does not compare results automatically after each user action, but can be configured to compare results on demand at critical points in the test. As soon as HP Sprinter detects a difference, you can view and compare the machines, and capture screenshots of all of the machines that display inconsistent results. From there, you can either submit a defect and automatically include the description of the difference, or instruct HP Sprinter to ignore the difference. Once the “Ignore” rule is created, HP Sprinter automatically recompares the machines.

Today’s applications are required to run in a variety of environments and support different types and versions of browsers, operating systems, and configurations. Without HP Sprinter, it is nearly impossible to manually test all of the required combinations. The ground-breaking mirroring technology not only reduces time, but helps to increase testing coverage and to release applications with more confidence.

ReportingWhen the run ends, it is automatically updated in HP ALM and HP QC, and a detailed run report is created. The report includes the run summary and all of the steps that were performed, including actual results that were entered during the run, and screen captures that were added. You can also see a list of any defects that were submitted during the run. Alternatively, you can review the run results through the Storyboard view. In the Storyboard view, the run is represented in a series of screen captures that were taken for every user action that was performed during the run. The report includes a list of all user actions that were performed during the test run, making it easier to create a format test from an informal or exploratory test. The run report is also saved in HP ALM, which helps to keep all of the members of the testing and development teams on the same page with the status and progress of the testing process, and helps them to make informed decisions about whether the application is ready to be released.

11

ConclusionManual testing continues to be a popular method for validating application functionality. To help accelerate the manual testing process and to increase coverage, speed, and accuracy, HP Sprinter provides an easy-to-use, unobtrusive tool that is seamlessly integrated with HP ALM and HP QC. HP Sprinter features an intuitive UI that helps to maximize the screen “real estate” of the application under test, and a variety of powerful advanced features designed to remove the major inefficiencies and difficulties of the manual testing process. Through data injection, detailed

defect reporting, test run mirroring, and other superior features, HP Sprinter is changing the way QA teams view manual testing. Instead of a tedious, time-consuming, and error-prone process, manual testing with HP Sprinter can be accurate, fast, and complete, helping to increase application quality, reduce testing time, and improve communication between different stakeholders in the organization.

Improve testing speed and accuracy with HP Sprinter; visit www.hp.com/go/sprinter

Get connectedwww.hp.com/go/getconnected

Get the insider view on tech trends, alerts, and HP solutions for better business outcomes

© Copyright 2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Microsoft is a U.S. registered trademark of Microsoft Corporation.

4AA3-1859ENW, Created November 2010

Share with colleagues