Software Test Automation Overview

Rohan Bhattarai

Contents Overview of STA BEP and ROI Tools Automation

Framework

Contents Overview of STA

History Myths and Truths Why Automation Projects Fail? What is TA? Why TA is needed? Manual vs. Automated Pros & Cons What to Automate? What Not to Automate?

BEP and ROI BEP BEP - Example ROI Classic ROI Real ROI Benefits of ROI ROI Calculator

Tools Choosing the right tool – The Strategic

Approach Choosing the right tool – The Strategic

Steps Tools Classification Short List Tools – Functional Short List Tools – Performance Evaluate Vendors Functional Test Tools - Analysis Feasibility Analysis

Automation Framework Where to start? What is AF? Benefits of AF How does it work? Architecture Framework Approaches

Record and Playback Script Based Approach Keyword Driven Approach Data Driven Approach Hybrid Keyword and Data Driven Approach

Contents Overview

History Myths and Truths Why Automation Projects Fail? What is TA? Why TA is needed? Manual vs. Automated Pros & Cons What to Automate? What Not to Automate?

History

Moving swiftly past the hype

Historically Automation is perceived as a “Silver Bullet” of the Testing world

“The term has been adopted into a general metaphor, where “silver bullet” refers to any straight forward solution perceived to have extreme effectiveness. The phrase typically appears with an expectation that some new technology or practice will easily cure a major prevailing problem”.

History Historical trends in test automation frameworks:

1993 – 20011st Gen – Modularity

Driven

2001 – 20052nd Gen – Data

Driven

2005 – 20083rd Gen – Keyword Driven

2008 – 20114th Gen – Hybrid

Keyword Data Driven

• 1993 – 1999: WinRunner x.x• 1999 – 2001: WinRunner 6.x• 2001 – 2004: WinRunner 7.x,

RobotJ 1.0, XDE Tester 1.0• 2004 – 2005: QTP 7.x, RFT 6.x• 2004 : Selenium 1

• 2005 – 2006: QTP 8.x, RFT 7.x• 2006 – 2008: QTP 9.x, RFT 8.x• 2007: Selenium 2• 2008 – 2010: QTP 10.x, RFT 8.1• 2010 – 2011: QTP 11, RFT 8.2

Myths and Truths1. Test Automation is simple, that every tester

can do it

Promoted by the Sales people by simply saying: Record the script Enhance the script by adding functions and data

driving Run the scripts Report results

Under this influence QA manager can proudly say “All our testers are doing test automation”.

Myths and Truths But in Reality: TA is a software development task

It should be designed, developed and tested Need to have some kind of a programming

background to implement test automation. TA is not as complex as C++/C#/Java development.

TA components are assets that should be treated like application source code

Don’t fall into tool vendor sales pitch …remember Record & Playback is not real test automation

Myths and Truths2. Commercial TA tools are expensive

Under the influence of this myth some companies, especially the small ones: Try to develop their own test automation tools Use scripting languages like Perl and Ruby Use shareware test tools Do not consider test automation at all

Myths and Truths But in Reality: Commercial TA tools are not

that expensive

Per seat license for the most expensive automation tool is $8K, which can be used for 5 years.

Maintenance/Support fees are 20% of tool cost or $1,600 per year

The cost of this tool is $8K/5+$1,600 = $3,200 per year The automation developer cost with overhead is $100K

per year The cost of this tool is just 3% of the person who uses

it, but productivity gain can be very significant

Myths and Truths Truth: 92% FAIL to meet target ROI

2004 2010 (Estimated)

Industry: Test Automation (Net Worth)

$1 Billion $ 6.3 Billion

Automation Projects (Failure Cost)

$ 0.6 Billion $ 3.8 Billion

40%

60%

Automation Projects

WorkingFailure

Source 1: http://www.nytimes.com/2006/07/26/technology/26hewlett.html Source 2: http://www.slideshare.net/Jonathon_Wright/hybrid-keyword-data-driven-automation-frameworks-jonathon-wright

32%

8%

ROI

Learning from past experience

Why do Automation Projects typically Fail? IDT study(www.idtus.com)

Lack of Time37%

Lack of Expertise20%

Lack Of Budget

17%

Tool In-compatibil-

ity11%

Misc15%

Why do Automation Projects typically Fail? Lack of defined automation

methodology Automation is not treated as a

legitimate project with the necessary planning / resources

Test Automation is typically performed at the end of the SDLC

After the initial success the automation scripts are not maintained for future builds

Why do Automation Projects typically Fail? Testers are typically untrained in test tools

and programming techniques No modularization (reusable functions) in

automation scripts Automated tests cases are usually designed

based on front end functionality (black box testing)

What is Software Test Automation? Test Automation is the use of software to

execute tests without Human intervention It refers to the activities and efforts that

intend to automate engineering tasks and operations in a software test process using well-defined strategies and systematic solutions.

Not like Rube Goldberg cartoons

Why TA is needed? Objectives:

To free engineers from tedious and redundant manual testing operations

To speed up a software testing process, and to reduce software testing cost and time during a software life cycle

To increase the quality and effectiveness of a software test process by achieving pre-defined adequate test criteria in a limited schedule

Key to Success

To reduce manual testing activities and redundant test operations using a systematic solution to achieve a better

testing coverage.

Manual vs. Automated Testing Manual Testing:

Testing time is consuming and tedious Inefficient in today’s shorter SDLC Delay the ability to thoroughly test an application Critical bugs escape undetected What happens when multiple platforms involved

Automated Testing: Higher efficiency that accelerate the testing cycle and

promote software quality Optimizes software quality and testing efficiency by delivering

Reusability Predictability and Consistency Productivity

Enables accurate assessment of quality level

Pros and Cons

Pros:SpeedReusabilityAccuracyRun AnytimeEfficiency

Pros and Cons

Cons:Significant InvestmentMaintenanceNot as RobustError DetectionCannot Think

What to Automate? Regression Tests: Stabilized tests that verify stabilized

functionality

Tests rerun often: Tests that are executed regularly vs. rarely

Tests that will not expire shortly: Most tests have a finite lifetime during which its automated script must recoup the additional cost required for its automation

Tedious/Boring tests:

tests with many calculations and number verifications

repetitive tests performing the same operations over and over

tests requiring many performance measurements

Just plain boring tests

Reliably repeatable

What NOT to Automate? Unstable functionality: Not reliably repeatable

Rarely executed tests: poor Return-On-Investment

Tests that will soon expire: poor Return-On-Investment

Requiring in-depth business analysis:

some tests require so much business specific knowledge that it becomes prohibitive time wise to include every verification required to make its automated script robust enough to be effective

exceedingly complex tests are sometimes not possible to automate because computers cannot think

Contents BEP and ROI

BEP BEP - Example ROI PayBack Period Classic ROI Real ROI Benefits of ROI ROI Calculator

BEP

Break-Even Point (BEP) is the point at which cost or expenses and revenue are equal: there is no net loss or gain

BEP - Example

Preparation (V)(in Mins)

Execution (D)(in Mins) ROI

using Manu

al

Resources (R ) for (n) Automated TestsRn = Aa / Am = (Va + n*Da) / (Vm +

n*Dm)

TestManual Automated Manual Automated 1 5 10 20

Scenario 1 30 60 11 1.1 33% 149% 77% 51% 33%

Scenario 2 30 60 11 1.1 33% 149% 77% 51% 33%

Scenario 3 30 60 9 0.9 36% 156% 86% 58% 37%

Scenario 4 30 60 10 1 34% 153% 81% 54% 35%

Scenario 5 30 60 10 1 34% 153% 81% 54% 35%

Scenario 6 30 60 10 1 34% 153% 81% 54% 35%

Scenario 7 30 60 15 1.5 27% 137% 64% 42% 27%

Scenario 8 30 60 30 3 5% 105% 42% 27% 19%

Scenario 9 30 60 22 2.2 16% 120% 51% 33% 22%

Scenario 10 30 60 12 1.2 31% 146% 73% 48% 31%

Total 300 600 140 14 28% 142% 71% 47% 31%

ROI Return on Investment ROI = BENEFIT/COST ROI = (total benefit – total cost) / (total cost) ROI = (cost of manual – cost of automation) /

cost of automation Where,

Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost

Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost

Looks right, Doesn’t it?

Classic ROI Problems with classic ROI calculation:

You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT.

You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually

So then…what is real ROI?

Real ROI ROI value is not the value of Automation vs.

Cost of executing these tests manually Automation ROI value is the benefit of this

type of testing, and it can be: Reducing Time to Market Increased Test Efficiency (Productivity) Increased Test Effectiveness

Benefits of ROI Reduced Time to Market

Products delivered quickly Makes people available to work on other projects Higher margins, if no competitive products in

market Productivity and Effectiveness

More testing gets done faster, increasing the odds of finding defects

Defects found early have better chances of being fixed

Manual Testers can concentrate on clever ways to finding defects, instead of typing test inputs and verify output.

Benefits of ROI About 7% of bug fixes create new bugs,

sometimes in already tested parts of the system. With automation you can rerun tests for those modules. This almost never happens when testing is done manually.

ROI Calculator ROI Calculator:

Source 1: http://www.aspiresys.com/testautomationroi/index.php

Source 2: http://www.elbrus.com/services/test_automation_roi_calc/

ROI Calculator

ROI Calculator

Contents Tools

Choosing the right tool – The Strategic Approach

Choosing the right tool – The Strategic Steps Tools Classification Short List Tools – Functional Short List Tools – Performance Evaluate Vendors Functional Test Tools - Analysis Feasibility Analysis

Tools

There is no single best testing tool; rather, different tools are more or less appropriate in different environments.

Tools Over 300 Test Tools are available (

http://www.softwareqatest.com) Load/Performance tools – 54 Web Functional/Regression – 60 Java Test tools - 48 Other Web tools – 76

Which tool is right for you?

Choosing the right tool – The Strategic Approach• Is there an organizational methodology for

test automation?• Which applications/processes?• What is the impact to current project

schedules?• What is the effort in maintaining

automated tests?• What are the costs?• What about tools integration?• What about Continuous Integration?

Choosing the right tool – The Strategic Steps• Step 1: Define and Refine Requirements• Step 2: Communicate the Impact• Step 3: Develop Evaluation Methodology • Step 4: Select, Procure, and Implement

Step 1: Define and refine requirement Create a list of organizational requirements

What problems do you want the tool to solve? What capabilities will the tool need to be

effective in your environment? Other lifecycle tools?

What constraints, budgetary or otherwise? Identify compatibility issues

What operating systems does your application support?

What is the development environment? Does your application integrate with third-party

software? Does the application use custom controls?

Step 1: Define and refine requirement Identify tool audience

Who will use the tool on a day-to-day basis? What is the level and mix of user skill levels? Is your organization willing to invest in training?

Define technical or business requirements Does your organization have additional

requirements? Software standards Technical standards Procurement rules Preferred vendor rules

Step 1: Define and refine requirement Identify budget constraints

How much can we afford? How much is this worth?

New requirements may surface based on research

Did not know about Forgot to include

Step 2: Communicate the Impact Automated testing is part of the larger

strategic application development endeavor Communicate the effects of implementing a tool

Chance to discuss and mitigate concerns

How tool may change job description Commitment to training Implementation strategy

Discussion may imply additional requirements Business, functional, technical, or operational

Step 3: Develop the evaluation methodology How will tools be compared? Are there specific features that may

differentiate one tool from another? Are there specific things that can eliminate a

tool from consideration? Preferred vendor list can reduce evaluation

scope Demos and evaluations are time-consuming Identify a representative set of activities to

accomplish with the tool during the evaluation

Step 4: Select, procure, and implement Make an informed selection Follow organization’s procurement process Develop the implementation plan

What When Why Who How

Step 4: Select, procure, and implement Develop an implementation plan

Enterprise applications requiring multiple releases Applications that must produce a consistent set of

results using stable data These characteristics fully leverage reusability and

predictability benefits of automated testing

Take implementation one step at a time Take time for training Keep focus on staff issues and reactions

Step 4: Select, procure, and implement Develop a test plan

Describes scope, approach, resources and schedule for all automated and manual activities Rule of Thumb: (Test scripts) 40% manual - 60%

automated

Create and deploy your automated tests Be selective with the automation of test scripts

Verify the most critical functionality Are the most likely to expose defects Are expensive or impossible to perform manually

Use the first automated suites you build for Smoke testing Regression testing

Decision?How to calculate the cost of functional test

automation

If a test script will be run every week for the next 2 years, automate the test if thecost of automation is less than the cost of manually executing the test 104 times.

Cost of automation Cost of manually executing the test as manytimes as the automated test will be executed

Cost of test automation

Cost of tool(s)

Labor costs of script creation

Labor costs ofscript maintenance

Automate if

Tools Classification

Test Tool Types Basic Descriptions of Different Types of Test Tools

Test InformationManagement

Systematic solutions and tools support test engineers and quality assurance people tocreate, update, and maintain diverse test information, including test cases, test scripts,test data, test results, and discovered problems.

Test Execution andControl

Systematic solutions and tools help engineer set up and run tests, and collect andvalidate test results.

Test Generation Systematic solutions and tools generate program tests in an automatic way.

Test CoverageAnalysis

Systematic solutions and tools analyze the test coverage during a test process basedon selected test criteria.

Performance Testingand Measurement

Systematic solutions and tools support program performance testing and performancemeasurement.

Software Simulators Programs are developed to simulate the functions and behaviors of external systems,or dependent subsystems/components for a under test program.

Regression Testing Test tools support the automation performance of regression testing and activities,including test recording and re-playing.

Tools Classification

Types of Test Tools Test Tool Vendors Test Tools

Problem Management Tools Rational Inc. ClearQuest, ClearDDTS

Microsoft Corp. PVCS Tracker

Imbus AG Imbus Fehlerdatenbank

Test Information Management Tools

Rautional Inc. TestManager

Mercury Interactive TestDirectory

Test Suite Management Tools Evalid TestSuiter

Rational Inc. TestFactory

SUN JavaTest, JavaHarness White-Box Test Tools McCabe & Associates McCabe IQ2

Junit

IBM IBM COBOL Unit TesterIBM ATC- Coverage Assistant- Source Audit Assistant- Distillation Assistant- Unit Test Assistant

Tools ClassificationTest Execution Tools OC Systems Aprob

Softbridge ATF/TestWright

AutoTester AutoTester

Rational Inc. Visual Test, Rational Functional Tester

SQA Robot

Mercury Interactive WinRunner, Quick Test Prof

Sterling Software Vision TestPro

Compuware QARun

Seque Software SilkTest

RSW Software Inc. e-Test

Cyrano Gmbh Cyrano Robot Code Coverage Analysis Tools Case Consult Corp. Analyzer, Analyzer Java

OC Systems Aprob

IPL Software Product Group Cantata/Cantata++

ATTOL Testware SA Coverage

Compuware NuMega TruCoverage

Software Research TestWorks Coverage

Rational Inc PureCoverage

SUN JavaScope

ParaSoft TCA

Software Automation Inc Panorama

Tools Classification

Load Test and Performance Tools Rational Inc. Rational Performance Tester

InterNetwork AG sma@rtTest

Compuware QA-Load

Mercury Interactive LoadRunner

RSW Software Inc. e- Load

SUN JavaLoad

Seque Software SilkPerformer

Client/Server Solutions, Inc. Benchmark Factory

Regression Testing Tools IBM Regression Testing Tool(ARTT)

Distillation Assistant

GUI Record/Replay Software Research eValid

Mercury Interactive Xrunner

Astra Astra QuickTest

AutoTester AutoTester, AutoTester One

Short List Tools - FunctionalToolVendor Test Suite - Companion Tools

TestPartnerCompuware QACenter Enterprise Edition+

QuickTest Professional

Mercury Quality Center

e-TesterEmpirix e-TEST suite

Rational Functional Tester

IBM Rational Suite

WebFTRadView TestView Suite

SilkTestBorland(Segue)

SilkCentral Test Manager

QA Wizard ProSeapine TestTrack Pro

Short List Tools - Performance

ToolVendor Test Suite - Companion Tools

QALoadCompuware QACenter Enterprise Edition+

LoadRunnerMercury Quality Center

e-LoadEmpirix e-TEST suite

Rational Performance Tester

IBM Rational Suite

WebLOADRadView TestView Suite

SilkPerformerBorland(Segue)

SilkCentral Test Manager

ForecastFacilita ForecastWeb, ForecastNet, ForecastDB

Evaluate VendorRisky bets Contenders

Currentofferings

StrategyWeak Strong

StrongLeaders

Strong performers

IBM

Empirix

Seapine

Compuware

Mercury

RadView

Segue

Functional Test Tools - Analysis

Tool Pros ConsIBM/Rational Functional Tester (RFT)

•Built as Eclipse Plug-In with full IDE and Java support•Supports Web 2.0, Java or .NET applications•Full GUI Object Map repository

•Insufficient browser support•Licensed product

HP/Mercury Quick Test Pro (QTP)

•Supports Web 2.0, Java or .NET applications•Full GUI Object Map repository•Seamless integration with QualityCenter

•VisualBasic scripting is limited•No IDE (may change in new release)•Licensed Product

Selenium RC & IDE

•Good browser support•Good language support (Java, Ruby,C# )•Can be easily extended as JUnit suite•Open-source

•No GUI Object repository•Only web application support

Feasibility Analysis FA Matrix Available Operational Feasibility Technical Feasibility Economic Feasibility Schedule Feasibility

FA Matrix

FA Matrix

Contents Automation Framework

Where to start? What is AF? Benefits of AF How does it work? Architecture Framework Approaches

Record and Playback Script Based Approach Keyword Driven Approach Data Driven Approach Hybrid Keyword and Data Driven Approach

Where to Start?

First find out Then you can work out

What needs to be tested?

What needs be automated?

What can be tested? What can be automated?

What could be tested? What could be automated?

“Start SMALLthink BIG”

Quick winsshould beavoided

NEVER expect toautomate 100%

“Under promise,Over deliver?”

Keep it simple,wherever possible

Focus on keycritical businessprocesses

What is Automation Framework? Framework – independent of application or

environment under test A Test Automation Framework is a set of

assumptions, concepts and tools that provide support for Automated Software Testing.

A reusable set of libraries or classes for a software system (or subsystem).

A correctly implemented Test Automation Framework can further improve ROI by reducing the development and maintenance costs.

Benefits of Framework Ease of Use – easy to learn and easy to use Time – faster than capture/replay and scripting approach Maintainability – significantly reduces the test maintenance

effort Reusability – due to modularity of test cases and library

functions Manageability - effective test design, execution, and traceability Accessibility – to design, develop & modify tests whilst

executing Availability – scheduled execution can run unattended on a 24/7

basis Reliability – due to advanced error handling and scenario

recovery Flexibility – framework independent of AUT or environment Measurability – customizable reporting of test results ensure

quality Increases the ROI (Return on Investment)

How does it work? Different Implementations

One Example of Keyword Driven Framework could be: Spreadsheets, Spreadsheets, Spreadsheets Test Objects Keywords and Methods = Toast! Parameters Description or Call the 911?

Architecture

Framework Approaches Record and Playback Script Based Approach Keyword Driven Approach Data Driven Approach Hybrid Keyword and Data Driven Approach

Manual Testing – Looking back

+ easy & cheap to start

+ flexible testing

- expensive every execution

- no auto regression testing

- less coverage measurement

Record and Playback

+ flexible testing

- expensive first execution

+ auto regression testing

- fragile tests break easily

- less coverage measurement

Script Based Approach+/- test impl. = programming

+ automatic execution

+ auto regression testing

- fragile tests break easily?

(depends on abstraction)

- less coverage measurement

Data Driven Approach Automation is data-centric User defines just data sets to drive tests with Will have an external data source (DB tables,

Excel spreadsheets, XML for data sets) Flow control (navigation) is normally done by

the test script not by the data sourceEx: data set exercises creation of new sales

accounts functionality; stored in a DB table account_dataCompanyName PrimarySalesPerson Street Zip City State

Genesis Inc. Phil Collins 5775 Main st 30075 Atlanta GA

RollingStones Inc. Mick Jagger Jr. 2332 Washington st 02111 Boston MA

Keyword/Action Driven Approach+ abstract tests

+ automatic execution

+ auto regression testing

- robust tests

- less coverage measurement

Keyword/Action Driven Approach Automation is action-centric De-compose your test cases/modules into granular re-

usable keywords The idea is for non-coders to be able to create

automated test cases with action keywords User defines flow control of the test via action

keywords

Example: Test Case “Verify Checking Account Balance”1. Enter Username and Password and Click submit button step

1 is action Login2. Enter “Phil Collins” as a Sales Person and Click Submit button3. Verify the Sales Person was successfully created and Logout

So you may want to choose the following re-usable action keywords:

EnterText, Click, Login, VerifyExists

Benefits of Keyword Driven Approach This Framework addresses the most

common problem with test automation: Automation Engineers do not have domain knowledge and the End Users (Subject Matter Experts/Test Engineers) usually do not have automation expertise.

When properly implemented and maintained, it presents a superior ROI because each business event is designed, automated and maintained as a discrete entity.

Keywords can then be used to design test cases, but the design and automation overhead for the keyword has already been paid.

Benefits of Keyword Driven Approach Reduced the cost and time spent maintaining and

updating tests The modular structure of keyword-driven testing

means that new tests can easily be created from pre-existing modules

The test team is capable of entirely automating tests, even without programming knowledge

Can be easily modified to use with different test tool Reusability across different projects

Classic Example:Object Action Data

Textfield (username)

Enter Text <username>

Keyword/Action Driven Approach May have an external data source (DB tables, Excel

spreadsheets, XML for data sets) with action keywords

Step

Description Page Action

Module

Type Object Expected

1 Login Home LoginUserLogin N/A

.id:=LoginSubmit

.text:=Login Successful

2

Enter New Sales Person data

CreateSalesPerson EnterText Field

.text:=SalesPerName

 .value:=Phil Collins

3Click Submit

CreateSalesPerson Click

Button

.id:=SubmitSalesPer

.url:=.*createdSalesPerStatus.html

4

Verify Sales Person Creation Successful

CreateSalesPersonStatus VerifyExi

sts DIV.id:=CreationStatus

.value:=User Created Successfully

Hybrid Keyword and Data Driven Approach Combines the best of both worlds

User defines data sets to drive tests with User also defines flow control of the test via

action keywords May have an external data source (DB tables,

Excel spreadsheets, XML for data sets) with action keywords in addition to generic and test case specific data sets

Architecture

Open2Test Automation Framework

Open2Test Automation Framework

Open2Test Automation Framework

Model Based Approach+ abstract tests

+ automatic execution

+ auto regression testing

+ auto design of tests

+ systematic coverage

+ measure coverage of model and requirements

- modelling overhead

Emerging Approach

References http://www.slideshare.net/Jonathon_Wright/hy

brid-keyword-data-driven-automation-frameworks-jonathon-wright?src=related_normal&rel=805408

http://www.ibm.com/developerworks/rational/library/591.html

http://www.keane.com/resources%2Fpdf%2FWhitePapers%2FWP_ROIforTestAutomation.pdf

Summarizing… Overview of STA

History Myths and Truths Why Automation Projects Fail? What is TA? Why TA is needed? Manual vs. Automated Pros & Cons What to Automate? What Not to Automate?

BEP and ROI BEP BEP - Example ROI Classic ROI Real ROI Benefits of ROI ROI Calculator

Tools Choosing the right tool – The Strategic

Approach Choosing the right tool – The Strategic

Steps Tools Classification Short List Tools – Functional Short List Tools – Performance Evaluate Vendors Functional Test Tools - Analysis Feasibility Analysis

Automation Framework Where to start? What is AF? Benefits of AF How does it work? Architecture Framework Approaches

Record and Playback Script Based Approach Keyword Driven Approach Data Driven Approach Hybrid Keyword and Data Driven Approach

THANK YOU

Rohan Bhattarai