Automation Testing Tools

Automation Testing ToolsAutomation Testing Tools

Sanjay Patel

Muhammad Rafi

Mahesh Vaghasia

AgendaAgenda

Overview Information Gathering Tools and Vendors Evaluation Criteria Tools Evaluation Matrix Conclusion

OverviewOverview

“Automated Testing” means automating the manual testing process currently in use. This requires that a formalized "manual testing process" currently exists in the company or organization. Minimally, such a process includes: – Detailed test cases, including predictable "expected results",

which have been developed from Business Functional Specifications and Design documentation.

– A standalone Test Environment, including a Test Database that is restorable to a known constant, such that the test cases are able to be repeated each time there are modifications made to the application.

Information GatheringInformation Gathering

Testers working– Motorola labs

– Siemens Medical Solutions

Online resources– IEEE and ACM papers

– Vendor’s web-site

– Forums

Information GatheringInformation Gathering

Following are sample questions asked to tester who have been using some the testing tools:

1. How long have you been using this tool and are you basically happy with it?

2. How many copies/licenses do you have and what hardware and software platforms are you using?

3. How did you evaluate and decide on this tool and which other tools did you consider before purchasing this tool?

4. How does the tool perform and are there any bottlenecks?

5. What is your impression of the vendor (commercial professionalism, on-going level of support, documentation and training)?

Tools and VendorsTools and Vendors

Robot – Rational Software WinRunner 7 – Mercury QA Run 4.7 – Compuware Visual Test – Rational Software Silk Test – Segue QA Wizard – Seapine Software

Tools OverviewTools Overview

Robot – Rational Software– IBM Rational Robot v2003 automates regression, functional and

configuration testing for e-commerce, client/server and ERP applications. It's used to test applications constructed in a wide variety of IDEs and languages, and ships with IBM Rational TestManager. Rational TestManager provides desktop management of all testing activities for all types of testing.

WinRunner 7 – Mercury– Mercury WinRunner is a powerful tool for enterprise wide

functional and regression testing. – WinRunner captures, verifies, and replays user interactions

automatically to identify defects and ensure that business processes work flawlessly upon deployment and remain reliable.

– WinRunner allows you to reduce testing time by automating repetitive tasks and optimize testing efforts by covering diverse environments with a single testing tool.


QA Run 4.7 – Compuware– With QA Run, programmers get the automation capabilities they

need to quickly and productively create and execute test scripts, verify tests and analyze test results.

– Uses an object-oriented approach to automate test script generation, which can significantly increase the accuracy of testing in the time you have available.

Visual Test 6.5 – Rational Software– Based on the BASIC language and used to simulate user actions

on a User Interface. – Is a powerful language providing support for pointers, remote

procedure calls, working with advanced data types such as linked lists, open-ended hash tables, callback functions, and much more.

– Is a host of utilities for querying an application to determine how to access it with Visual Test, screen capture/comparison, script executor, and scenario recorder.


Silk Test – Segue– Is an automated tool for testing the functionality of enterprise

applications in any environment. – Designed for ease of use, Silk Test includes a host of productivity-

boosting features that let both novice and expert users create functional tests quickly, execute them automatically and analyze results accurately.

– In addition to validating the full functionality of an application prior to its initial release, users can easily evaluate the impact of new enhancements on existing functionality by simply reusing existing test casts.

QA Wizard – Seapine Software– Completely automates the functional regression testing of your

applications and Web sites. – It’s an intelligent object-based solution that provides data-driven testing

support for multiple data sources. – Uses scripting language that includes all of the features of a modern

structured language, including flow control, subroutines, constants, conditionals, variables, assignment statements, functions, and more.

Evaluation CriteriaEvaluation Criteria

Record and Playback Object Mapping Web Testing Object Identity Tool Environment Support Extensible Language Cost Integration Ease of Use Image Testing Database Tests Test/Error Recovery Data Functions Object Tests Support

Record and Playback

WinRunner

Allows recording in context sensitive or analogue mode. Uses TSL (Test Script Language), which provides usual array of functions and syntax. Has function for WAP.

QA Run

Learn facility for recording and playback. Playback scripts in command line or via toolbar. RunAWL is Compuware’s script execution module that allows you to select scripts to run without having to open the script editor.

Silk Test

Two ways to record: quickstart window and from toolbar. Uses an identifier and a tag to identify each object individually. Allows multi-tags.

Visual Test‘tape cassette’ icon on the toolbar starts the recording session. Each recording session is given a scenario name.

Robot

Record test using a red GUI button. Uses a language very similar to Visual Basic called SQABasic. User defined allows you to build a mixture of data types similar to a ‘struct’ in C or a ‘record’ is Pascal.

QA Wizard

Uses a visual and menu driven approach for creating scripts. People with very little knowledge in scripting can record and playback.

Tools

Cost

WinRunner

Being one of the most reputable test tools tends to be expensive. Price depends on number of licenses purchased. Java add-in increases the cost to anywhere from $ 4000- 6000 per license

QA RunPrice is comparable to the price of Robot. Each license cost $2000-3500.

Silk Test

Is on the expensive side. Segue products tend to be more expensive than the others. Price is a disadvantage here. Anywhere from $3000-5000 per license.

Visual Test

The cheapest of all the tools. You can buy it for close to $1000. You don’t need multiple licenses, which makes it cheaper.

RobotDepends on number of licenses purchased. Varies from $2500-4000 per license.

QA Wizard Each license varies from $3000-$5000

Tools

Object Mapping

WinRunnerObject mapping is available via the rapidtest wizard and virtual object wizard. Allows you to map non-standard objects or painted controls

QA Run

Has a good facility for object mapping. All identified objects use a hybrid of properties to uniquely identify themselves. You can get QA Run to automatically create name for unidentified testing objects

Silk Test

You can map the custom win to the so-called meta classes i.e., BUTTON, STATIC, and MDI. You can map an object to a class you think it behaves like.

Visual TestObject mapping can be done from tools or from scenario recording. Object mapping facilities can be accessed through the tools->general options menu.

RobotObject mapping facilities can be accessed through the tools->general options menu. Classes to map include Java, HTML, PeopleSoft Tools and standard windows controls.

QA WizardYou can map objects to pre-defined classes. Comes with most of the standard window controls

Tools

Environment Support

WinRunnerOut of the box supports, all windows based system. With add-ins, support Java based applications.

QA Run

Supports NS-DK, Oracle, PowerBuilder, Visual Basic, Uniface (Compuware’s own 4GL), SAP, Java, Netscape, Internet explorer, and most windows applications

Silk Test

Supports HTML, XML, JavaScript, Java, ActiveX, Windows 2000, Windows 98, Windows 95 controls, Visual Basic, Internet Explorer, and Netscape.

Visual TestSupports Microsoft centric environments e.g. Internet explorer, Windows based control and similar.

RobotSupports HTML, DHTML, Java, Visual Basic, Visual C++, Oracle Developer/2000, Delphi, SAP, PeopleSoft and Sybase PowerBuilder.

QA Wizard Supports HTML, DHTML, XML, Java, and Visual Basic.

Tools

Extensible LanguageWinRunner You can extend WinRunner’s facilities by creating or using DLL’s.

QA Run Like other tools, this tool is also extendable through the use of DLL.

Silk Test

You have the ability to extend the 4Test language in several ways. DLL invocation allows you to call functions stored in DLL’s. It offers inheritance and abstract classes. You can add your own verification properties to classes.

Visual Test

Declaring a DLL function can extend Visual Test. One neat thing is that you can use pointers so functions that require variables to be passed by reference can be used even though the majority is byVal.

Robot

Robot’s language can be extended in two ways: One by declaring a function in DLL and then calling it. You can also extend the language by creating classes in languages like: VB and then creating an instance of the class using the CreateObject function.

QA Wizard It can be extended by using DLL function.

Tools

Tools Rec

ord

& P

layb

ack

Web

Tes

ting

Env

iron

men

t Sup

port

Cos

t

Eas

e O

f Use

Dat

abas

e Te

sts

Dat

a F

unct

ions

Obj

ect M

appi

ng

Obj

ect I

dent

ity T

ool

Ext

ensi

ble

Lang

uage

Inte

grat

ion

Imag

e Te

stin

g

Test

/Err

or r

ecov

ery

Obj

ect T

ests

Supp

ort

Tot

al S

core

WinRunner 2 1 1 3 2 1 2 1 2 2 1 1 2 1 1 23

QA Run 1 2 2 2 2 1 2 1 1 2 1 1 2 1 2 23

Silk Test 1 2 2 3 3 1 2 1 2 1 3 1 1 1 2 26

Visual Test 3 2 2 1 3 3 3 2 1 2 2 2 2 2 2 32

Robot 1 2 2 2 2 1 2 1 1 2 2 1 2 1 2 24

QA Wizard 2 3 1 1 2 1 1 1 1 1 1 1 1 1 2 20

3 = Basic 2 = Good 1 = Excellent

Vendors/Tools Strength Weakness

Rational (Robot, Visual Test)

Economical, Lots of supporting tools, good extensible language through VB, good data creation facilities, good online community.

Support, Language is a little basic out of the box and a bit confusing with tools breakdown.

Mercury (WinRunner)

Possibly the most popular test tool, plenty of jobs, good support, good online community, Good cross browser support.

No Java support out of the box, expensive.

Compuware (QA Run)

Strong professional services, good integration with development products, economical.

Poor online community.

Seque (Silk Test)Good development language, good online community, recovery system.

Bad helpdesk, expensive.

Seapine (QA Wizard)Strong Java support, Very easy to use.

Not used widely in the market, poor online community.

Strength & Weakness of ToolsStrength & Weakness of Tools

Tool Selection RecommendationTool Selection Recommendation

Tool evaluation and selection is a project in its own right.

It can take between 2 and 6 weeks. It will need team members, a budget, goals and timescales.

There will also be people issues i.e. "politics". Start by looking at your current situation

– Identify your problems – Explore alternative solutions – Realistic expectations from tool solutions – Are you ready for tools?


Make a business case for the tool – What are your current and future manual testing costs?

– What are initial and future automated testing costs?

– What return will you get on investment and when?

Identify candidate tools– Identify constraints (economic, environmental, commercial,

quality, political)

– Classify tool features into mandatory & desirable

– Evaluate features by asking questions to tool vendors

– Investigate tool experience by asking questions to other tool users Plan and schedule in-house demonstration by vendors

– Make the decision


Choose a test tool that best fits the testing requirements of your organization or company.

An "Automated Testing Handbook" is available from the Software Testing Institute (www.ondaweb.com/sti), which covers all of the major considerations involved in choosing the

right test tool for your purposes.

ReferencesReferences

1. www.rational.com

2. www.mercury.com

3. www.seque.com

4. www.seapine.com

5. www.compuware.com

6. www.testing.com

7. www.sqa-test.com

Documents

Automation Testing Tools