Selecting Automated Testing Tools

8/6/2019 Selecting Automated Testing Tools

1/12

A SmartBear White Paper

Selecting the right automated sotware testing tool involves having a list o requirements to review

during your evaluation. Without a list o requirements, you may waste time downloading, installing

and evaluating tools that only meet some o your requirements. This White Paper outlines common

requirements or automated testing tools which you can build rom to create a list that reects the

specifcs o your sotware testing needs.

Selecting Automated Testing Tools

Contents

Introduction........................................................................................... 2Support or Various Applications and Platorms ..................... 2

Support or Various Operating Systems...................................... 3

Support or Various Testing Types................................................. 3

Creating Automated Tests Without Programming ................ 4

Automated Test Scripting................................................................. 5

Recording Automated Sotware Tests......................................... 5

Creating Automated Tests that are Resistant

to Changes ............................................................................................. 6

Support or Controls and Objects .................................................. 6

Support or Various Test Data Sources........................................ 7

Running Automated Sotware Tests and

Synchronization.....................................................................................7

Automatically Compare Results o Automated

Test Actions.............................................................................................8

Ensuring Sotware SuccessSM

www.smartbear.com/testcomplete

Logging Automated Test Results ..............................................8Exporting Reports ...........................................................................8

Additional Needed Automated Testing Tool Features.....9

Testing Multi-Language Applications .....................................9

Automated Testing Tool: Support or Team Work .............9

Command Line and OLE Automation Support ................ 10

Integration with Team System and Build Sotware.......10

Technical Support ........................................................................ 10

Pricing Policy .................................................................................. 10

Free Trial Version .......................................................................... 11

8/6/2019 Selecting Automated Testing Tools

2/12

Introduction

Testing is a critical part o the sotware development process. There are a lot o dierent automated sotware

testing tools currently on the market. Some o these tools are only able to perorm specifc kinds o testing and

only work with specifc languages, like, or example, Java application unit testing. Other products support a wide

range o applications and oer more eatures and unctionality.

When you start your search or the right automated sotware testing tool, its important to create a list o

requirements to review when choosing a tool or evaluation. I you dont have a list o requirements, you may

waste time downloading, installing and evaluating tools that only meet some o your requirements, or may not

meet any o them.

Creating a list o requirements is not easy, since you have to take all o your testing needs into account. We have

prepared a list o requirements, which are described below. O course, the list does not include every situation,

but this list is a good start. You can modiy and add to it to get a list that reects the specifcs o your sotware

testing environment.

Support or Various Applications and Platorms

Some automated testing tools only support one type o application, like, or instance, only Java or .NET ap-

plications, while other tools support more application types. In general, the best option is to choose a tool that

supports all development tools currently used in your organization, or all tools that you plan on using in the near

uture. Even i you build your application with one compiler, you might use other compilers in the uture.

Selecting one multi-compiler sotware testing tool instead o several compiler-specifc tools may save you

money, but you must decide which is more important and whether it will save you time and energy when creat-

ing tests or uture application versions. Purchasing and using several sotware testing tools requires additional

time or training and creating tests. Do you have that much time? Does your company have enough budgeted

to pay or this?

TestComplete is one o the automated sotware testing tools that support the testing o any Windows program

created with any compiler. It works with both 32-bit and 64-bit Windows, .NET, Java and web applications. It

can also test applications running on Windows Mobile-based PDAs.

TestCompletes automated test engine provides extended support or the ollowing applications:

C#, Visual Basic .NET, VCL .NET and any other .NET application or any version o .NET Framework (1.0, 1.1,

2.0, 3.0, 3.5 or 4.0)

WPF (XAML) and XBAP applications (.NET Framework 4) Java applications (including Swing, SWT, AWT and WFC applications)

Visual C++ 6.0-9.0 applications (including ATL, MFC and WTL applications)

Visual Basic 6.0

Delphi and C++Builder applications (both VCL and CLX applications)

2

8/6/2019 Selecting Automated Testing Tools

3/12

Flex, Flash, Silverlight and JavaFX applications

Windows Mobile applications

Qt applications

wxWidgets applications

This extended support means that TestComplete can access the internal objects, methods and properties o the

application under test. QA engineers can use all o these elements in their automated tests. For instance, your

tests can execute an objects internal methods to initiate a unit test.

TestComplete also provides access to methods and properties o web page elements displayed in Internet Ex-

plorer, Fireox, Netscape Navigator or any browser based on the Microsot Web Browser control. It can test web

pages on the client computer and is not dependant on the server technology. You can use it to test web pages

generated by IIS, ASP, ASP.NET, PHP, CGI, AJAX, Atlas and other applications.

Support or Various Operating Systems

Choose an automated testing tool that supports as many operating systems as possible. Make sure that thetool supports recent operating system versions. The act that your customers do not run your application under

Windows Vista does not mean that they will not use it in the uture. I your application works under this operat-

ing system, your sotware testing tool should be able to work under it as well. In other words, the more Windows

versions the sotware testing tool supports, the better.

TestComplete supports both old and new Windows operating systems: Windows 2000, Windows XP, Windows

Vista, Windows 7, Windows Server 2003 and Windows Server 2008, including both 32 and 64-bit versions o

these operating systems. TestComplete also provides automated testing support or Windows Mobile applica-

tions running on PDAs. The range o supported operating systems on PDAs includes Windows Mobile 2003

(Pocket PC 2003), Windows Mobile 2003 SE (Pocket PC 2003 SE), Windows Mobile 5.0 and Windows Mobile 6.0.

Support or Various Sotware Testing Types

When choosing an automated testing tool, check whether it supports the test types that you are going to run.

Can it simulate user actions over the tested applications user interace, in other words, perorm unctional

testing? Can it run unit tests built into the tested application? Can it run tests created with NUnit, JUnit, DUnit

or MSTest? Does it support distributed testing? Does it have special eatures or synchronizing tests that run

simultaneously on several network workstations? Does it support load testing?

Also, i you test your products using both manual and automated methods, you may want to choose a tool that

provides specifc support or manual tests. Also, dont orget other important actors, such as, data-driven testingsupport and the ability to use the tool or regression testing. There are a lot o automated sotware testing tools

that support only one or two testing types, or instance, only unit testing or unctional testing. Some vendors

require separate products or dierent testing types. For instance, support or load or distributed testing can be

implemented with separate products or add-ons. I you wish to support these testing types, you will have to add

the price o these products or add-ons to the price that you are going to pay or the product.

3

8/6/2019 Selecting Automated Testing Tools

4/12

TestComplete supports unit, unctional (or UI), regression, data-driven, manual and distributed testing o Win-

dows applications. It provides special support or unit tests created in NUnit, JUnit, DUnit, MSTest, which allows

you to run these tests directly rom your TestComplete projects. This lets you use unit tests that were created

with these tools earlier in your development cycle. With TestComplete you can also perorm load, stress and scal-

ability testing o Web servers and services. To perorm coverage testing, check the application or resource and

memory usage and you can fnd bottlenecks in your application by using another one o SmartBears products

called AQtime. AQtime is tightly integrated with TestComplete, so you can run coverage, perormance and

memory leak tests directly rom your TestComplete projects.

Creating Automated Tests Without Programming

An automated testing tool must let you create tests ast and eectively. The test creation techniques must be

sufciently powerul to allow perorming various automated testing tasks. At the same time the automated

testing tool must be easy-to-use and clear even or inexperienced QA personnel. A good automated testing tool

may support several ways or creating automated tests. For instance, it may provide a simple, easy-to-use visual

automated test editor and some other eatures, like automated test scripts, or perorming advanced tasks.

Many automated testing tools only provide scripting capabilities or creating automated tests. Creating auto-

mated test scripts require programming experience and may be difcult or inexperienced testers. TestComplete

provides a simple but powerul alternative to automated test scripts: keyword tests (also called keyword-driven

testing).

Keyword tests consist o operations that simulate user activities on the application under test (like clicking the

mouse buttons or typing on the keyboard) or perorm other automated testing actions: launching tested ap-

plications, posting messages to the test log, running other automated tests (or instance, low-level procedures or

unit tests), use checkpoints to veriy specifc application unctionality and more.

Keyword tests are very simple and easy to create. Testers do not need programming skills to create them. You

can record these keyword-driven tests or modiy them visually in a special editor. The editors intuitive UI, wiz-

ards and dialogs help even inexperienced users create powerul automated tests with ease. Keyword tests are an

ideal eature or creating unctional tests quickly.

But keyword tests are not the only automated tests that can be recorded or created visually. TestComplete can

record automated test scripts, load tests and low-level procedures (special procedures that simulate low-level

events like pressing and releasing o mouse buttons or keyboard keys). Load tests and low-level procedures can

then be modifed visually with special editors. Recorded automated test scripts can also be modifed in the code

editor.

4

8/6/2019 Selecting Automated Testing Tools

5/12

Automated Test Scripting

Many automated sotware testing tools also provide scripting capabilities. Scripts are very important when per-

orming automated testing and they let users create automated tests or specifc application eatures, provide a

variety o automated testing actions and perorm those operations that cannot be done with standard eatures

built into the automated testing tool. Automated test scripts are useul even i the automated testing tool pro-

vides easier ways to create automated tests, because script code typically provides more powerul unctionality,

it lets you perorm those actions that other automated testing types cannot.

I you preer scripting or your automated testing, it is important to pay attention to the scripting language the

automated testing tool uses. Using standard languages, such as VBScript or JScript rather than their dialects,

will decrease the time needed to get acquainted with a product. Another beneft is that you can fnd inormation

on these languages rom third-party web sites as well as MSDN.

Two other important eatures to think about when choosing a tool that uses scripting is the code editor and

script debugger. Check whether the editor supports eatures that simpliy and speed up script writing: syntax

highlighting, outlining, code completion, bookmarks, Ctrl-click navigation between script routines and many

others. Check i the tools script debugger includes the needed debugging unctionalities, such as breakpoints,

step-over and step-into execution, Call Stack window and Evaluate dialog, etc. Using all o these eatures you

can easily fnd and correct bugs or fnd workarounds.

TestComplete supports several scripting languages: VBScript, JScript, DelphiScript, C++Script and C#Script, so,

you can choose the language you know best and thus save more time.

TestComplete includes a powerul code editor that supports all modern code editor eatures: syntax highlighting,

outlining, code completion, code templates, bookmarks and many others. TestCompletes built-in script debug-

ger supports ordinary and conditional breakpoints, step-over, step-into and run-to-the-cursor eatures, the Watchand Call Stack windows and special eatures or evaluating variables and objects during the script run.

Recording Automated Sotware Tests

Functional (or user interace) testing implies simulating user actions on the application under test. Implement-

ing these tests even or a small application is a tiresome process. When choosing an automated testing tool,

make sure the tool can record and play back user actions on the tested application. This eature will signifcantly

streamline test development.

TestComplete records user actions as keyword tests or automated test scripts, which you can then edit and play

back an unlimited number o times. Fast test development is not the only beneft. Recording also helps you

understand how to create a keyword test or how to write automated test script code, how to address application

objects and how to perorm testing actions on them. Wed like to note that TestComplete not only records and

plays back tests; it also records and plays back HTTP trafc. This eature lets you easily create load tests or web

servers and services. TestComplete can also flter out trafc sent and received by specifc applications during

trafc recording.

5

8/6/2019 Selecting Automated Testing Tools

6/126

Creating Automated Tests that are Resistant to Changes

Normally, you start testing your application while it is still in development. So, it is quite possible that ater you

make changes to the application, the current tests will become obsolete and youll have to update the test to

work with the changed application windows and controls. Modiying the tests may take the same amount o

time as modiying the application itsel. Thats why smart teams like to create exible tests that will be resistant

to the changes made to the application under test.

One approach that decreases the dependence on changes made to the tested application is the object-oriented

testing model. Each window or control o the application under test is represented by an appropriate object with

the name, or instance, [WinFormsObject(Form1)], so, to simulate user actions on this window or control, you

call the objects methods and properties. For instance, simulating a mouse click on a button can be represented

as a call to the Click method o the test object that corresponds to this button. I you use this approach, there is

no need to change your test ater the tested application has changed. For instance, i developers change a but-

tons width, height or position, the Click method will still work. I you simulate a click by coordinates, changing

the buttons size or position would cause changes in the automated test. Nevertheless, in some cases, you may

need to work with coordinates as well, so, it is nice i the automated testing tool supports both object-oriented

and coordinate-oriented methods.

Another approach that lets you create robust tests is object mapping. As a rule, automated sotware test-

ing tools address applications objects by their names or captions. However, during application development,

programmers may change controls captions, so you will have to modiy your tests in order or them to match

the changed controls. That is why a good automated testing tool should be able to map an object name to a

custom name and use this custom name to address the object in tests. However, the tool must be able to map

an object not only by its caption or identifer. In this case, i a developer changes the tested application, you will

only have to modiy the mapping conditions while the scripts and other tests remain the same.

TestComplete includes special eatures that let you map application windows, controls and objects by their prop-

erties. Testers can map a short name to the tested object and then use this mapped name in automated tests.

This simplifes working with objects, makes the created tests shorter and easier to understand. There is another

beneft o name mapping - it makes automated tests independent o object properties, which may change rom

one build to another. Also, TestComplete supports the object-oriented approach to simulate user actions. Simu-

lating mouse events by coordinates is also supported.

Support or Controls and Objects

Windows applications use controls to display inormation and interact with users. The operating system provides

dozens o various controls that can be used in tested applications: edit boxes, list views, check boxes, menus,

buttons and much more. This number is also extended with custom components oered by third-party vendors.

These can be non-standard components, like a tree list or grid, or third-party analogs to standard components:

custom menus, toolbars and many others. When choosing an automated testing tool, make sure it includes spe-

cial eatures that simpliy the work with controls used in your application. It is important to determine whether

8/6/2019 Selecting Automated Testing Tools

7/127

these eatures are or both standard and third-party controls.

Another important actor is access to methods and properties o the applications internal objects and controls.

This access provides methods and properties or simulating user actions and checking the results o test actions.

For example, you can use the internal methods and properties o a grid control to change the value in a grids

cell and then check i the change was successul.

TestComplete provides specifc methods and properties that greatly simpliy your work with standard Win32

controls, web page controls, WPF controls and third-party components provided by Microsot, Borland, Com-

ponentOne, Developer Express, Inragistics, Syncusion, Telerik and other vendors. In addition, TestCompletes

automated test engine can access the internal methods and properties o application objects and use these

methods and properties to perorm the desired testing tasks.

Supports Various Test Data Sources

When perorming unctional testing, its a common practice to simulate user input into the tested applications

data orms. Thorough testing implies that you eed varied data to the input felds and that the tested applica-

tion processes it correctly. When choosing an automated testing tool, check which data ormats it can use, such

as text fles, XML fles, database tables, and others.

With TestComplete, you can obtain data rom text and XML fles, Excel worksheets and databases like SQL

Server, Oracle and MySQL. TestComplete includes special program objects and test operations that allow you to

iterate through data records in external fles and easily create data loops in your automated tests.

You can create and populate these data storages manually, or you can use TestCompletes Data Generator to

generate the desired test data automatically and save it either to TestCompletes Table variable or to the speci-

fed Excel fle. This approach helps you decrease the time spent on preparing test data. For more inormation on

using this eature, see the Using Data Generators section in TestCompletes help.

TestComplete also allows you to use script arrays, variables and constants to keep data or automated tests.

Running Automated Sotware Tests and Synchronization

The state o the application under test and the execution environment may dier rom the state and conditions

you expected. For instance, some network resource may be unavailable, or the desired window is not created; the

web page loads very slowly or an unexpected modal dialog box appears. These and other similar actors may

cause the automated sotware testing tool to decide that the test has ailed and it should be stopped. A good

automated testing tool must be able to handle these situations in the appropriate manner. For instance, it must

be able to wait or the needed windows, close unexpected dialogs automatically and perorm other actions

that a human would do to continue the test.

TestComplete can handle these situations properly. For example, methods that work with windows and web

pages can wait until a window appears on screen or until a web page is loaded; methods that work with process-

es can wait until the desired process starts and so on. TestComplete also includes special eatures that let you

8/6/2019 Selecting Automated Testing Tools

8/128

speciy how the tool should handle unexpected windows without interrupting the test run.

Automatically Compare Results o Automated Test Actions

An automated testing tool should include eatures or automatically comparing the results o test actions. For

instance, i during the test run the tested application saves a customer list to a fle, the automated testing tool

should be able to compare this fle with a baseline copy and determine whether the save was successul and the

saved data is correct. The automated testing tool should be able to compare objects and controls properties,

images, fles and other objects.

TestComplete includes special tools, objects and unctionality that you can use to compare fles, images and

object properties. The comparison can be implemented manually, or generated by TestComplete when you are

inserting a checkpoint in your test. TestComplete includes special dialogs and wizards that can be used to create

checkpoints or properties, objects, fles, images, databases, tabular controls, XML documents, web services and

web pages. By using the powerul test implementation tools, you can create automated tests that will compare

complex objects and can also use regular expressions or comparisons.

Logging Automated Test Results

An automated sotware testing tool should keep a log o all testing actions and provide a detailed report or

them. The log o a high-end sotware testing tool must contain dierent types o messages: errors, warnings and

inormative messages. In this case, you can easily determine what went wrong and why the test ailed.

TestComplete includes a powerul test log that contains warning, error or inormative messages as well as fles

and images. The images are captured by the Test Visualizer subsystem and are automatically posted to the log

during the test run, or they can be posted using special operations or scripting methods. The captured images

illustrate test commands and simpliy the analysis o the automated test results.

Exporting Reports

Anautomated testing tool should give users the opportunity to export test results to external fles. For instance,

it may be useul, when you need to view test results on other computers that do not have this testing tool

installed. Also, you should check whether the testing tool can export the test log so that you can send the results

to your boss or colleagues.

TestComplete stores test results in XML fles and allows you to easily export the test log data to several ormats

(XML+HTML, Excel, MHT), pack the results with WinZip, WinRar or some other archive tool and send the results

via e-mail. You can perorm these operations rom the TestComplete UI or rom tests.

Also, TestComplete provides access to the stored results. Since the test log uses the XML ormat, you can parse

the results and retrieve the desired inormation.

TestComplete also allows you to create a bug report or test log data and post the report to an issue-tracking

system.

8/6/2019 Selecting Automated Testing Tools

9/129

Additional Needed Automated Testing Tool Features

To create powerul sotware tests, the test recording and playback eatures may not be enough. Choose a tool

that provides many additional eatures. For instance, you may need the ollowing:

A library o useul objects and unctions to perorm various operations. For example, reading data rom and

writing it to fles, working with databases, working with date and time values, regular expressions and manyothers.

The ability to call Win32 API unctions.

The ability to call unctions that reside in DLLs and .NET assemblies.

The ability to call routines that reside in Java classes.

The ability to work with COM servers.

Access to the shell (the WSH scripting object) and the ability to call the operating system commands.

OCR unctionality to retrieve data rom applications windows and controls.

The ability to create custom add-ons that will add specifc unctionality to your sotware test projects.

TestComplete includes all o these eatures and oers many others.

Testing Multi-Language Applications

I you have customers throughout the world and your company develops or is planning to develop localized ver-

sions o your applications, you have to check whether the sotware testing tool can work with localized applica-

tions and operating systems and whether it supports Unicode. I the tool does not support oreign languages,

then, most likely, you will not be able to use it to test localized sotware.

TestComplete supports Unicode and, thereore, it is possible to test multi-language sotware with it.

Automated Testing Tool: Support or Team WorkTesters are members o the development team. They work together and communicate with developers, manag-

ers and other various people. When choosing an automated testing tool, you may be interested in eatures that

support team work such as concurrent work with project fles, sharing script code among several projects and

integration with source code control and issue-tracking systems.

TestComplete supports all o these eatures. It allows you to share your test projects with several testers. You

can easily link script units rom one project to another. You can check project fles in and check them out rom a

source control system directly rom TestComplete. Finally, you can add a bug report to an issue-tracking system

like Microsot Visual Studio Team Foundation, Bugzilla, OnTime or JIRA, directly rom the test log viewer, and

thus notiy developers about the problem immediately.

8/6/2019 Selecting Automated Testing Tools

10/1210

Command Line and OLE Automation Support

Choose an automated testing tool that you can launch automatically, or instance, at a certain time or ater the

application build is over. You will be able to do this only i your automated testing tool has the command line or

OLE interace where you can speciy run parameters.

TestComplete supports command line arguments and can work with COM. Other applications, or instance,

SmartBearsAutomated Build Studio, use these eatures to call TestComplete automatically.

Integration with Team System and Build Sotware

Automated sotware testing tools are used along with build programs. For instance, the application builder can

run tests at the end o each build. When choosing an automated testing tool, check whether it can be used with

build programs.

TestComplete can be integrated with build sotware, or example, with SmartBears Automated Build Studio and

with Microsot Visual Studio, including version 2010 o this popular Microsot IDE. Also, TestComplete projects

can be called rom Visual Studio test projects. TestComplete also includes a special assembly that you can use to

run TestCompletes sotware tests rom MSBuild projects.

Technical Support

A very important aspect o selecting an automated sotware testing tool is the quality o the technical support

provided. Prompt responses to your messages prepared by a proessional support team will assist in resolving

problems and will save you time and energy.

Pricing Policy

When choosing an automated sotware testing tool, you have to determine whether its price is reasonable.

You should take into account not only the product price, but also the price o add-ons (i they are needed), the

support ee as well as training and upgrade prices. To get this inormation, you may need to call the companys

ofce. Some vendors have special prices that you can only learn about when talking to a representative rom

their sales department over the phone. SmartBear oers a clear pricing policy. You can fnd TestComplete prices

on our web site: TestComplete Pricing.

As you can see, SmartBear oers a eature-rich product at an aordable price. As a result, not only

large companiesuse TestComplete in their everyday work, but also small teams can aord it. Also, you can save

money by purchasing TestExecute licenses rather than TestComplete licenses or all o the computers that you

are going to run sotware tests on, but that are not or creating and debugging sotware tests.

Training is a paid option. For more inormation on this, see our TestComplete Training Web page.

8/6/2019 Selecting Automated Testing Tools

11/12

Free Trial Version

An important actor is the availability o a trial version. Even i the product seems to meet your requirements, the

best way to determine this is to download and use it beore purchasing. Since you are new to the product, you

may get questions and need some assistance with perorming tasks. So, another important actor is whether the

company provides support or trial-version users.

SmartBear provides support or trial-version users. The trial version is a ull-eatured version, but it has a 30-day

limitation and there are limitations on the size o your scripts.

You may also enjoy these other resources in the SmartBear Sotware Quality Series:

11 Best Practices or Peer Code Review

Uniting Your Automated and Manual Test Eorts

6 Tips to Get Started with Automated Testing

Be Smart andjoin our growing community o over 100,000 development, QA and IT proessionals in 90

countries.

Try TestComplete Free or 30 Days

Scan and download your ree trial to see why

testers choose SmartBear or automated testing.

11

8/6/2019 Selecting Automated Testing Tools

12/12

SmartBear Sotware, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915+1 978.236.7900 www.smartbear.com 2012 by SmartBear Sotware Inc.Specifcations subject to change. TC-WP-SATT-042012-WEB

About SmartBear Sotware

SmartBear Sotware provides tools or over one mil-

lion sotware proessionals to build, test, and monitor

some o the best sotware applications and websites

anywhere on the desktop, mobile and in the cloud.

Our users can be ound worldwide, in small businesses,

Fortune 100 companies, and government agencies.

Learn more about the SmartBear Quality Anywhere

Platorm, our award-winning tools, or join our active

user community at www.smartbear.com, on Facebook,

or ollow us on Twitter @smartbear.