Successful Test Automation: A Manager’s View

TB Full Day Tutorial

10/14/2014 8:30:00 AM

"Successful Test Automation: A

Manager’s View"

Presented by:

Mark Fewster

Grove Consultants

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Mark Fewster

Grove Software Testing Ltd. Mark Fewster has more than thirty years of experience in software testing ranging from test management to test techniques and test automation. For the past two decades, Mark has provided consultancy and training in software testing, published papers, and co-authoredSoftware Test Automation and Experiences of Test Automation with Dorothy Graham. A popular speaker at conferences worldwide, Mark has won the Mercury BTO Innovation in Quality Award. He is currently helping ISTQB define the expert level certification for test automation. Speaker Presentations

Written by Grove Software Testing Ltd. www.grove.co.uk Version 3_1 © Grove Software Testing, 2014

StarWest 2014

Managing Successful Test Automation A one-day tutorial

Managing Successful Test Automation

Contents Session 0: Introduction to the tutorial Objectives, What we cover (and don’t cover) today

Session 1: Planning and Managing Test Automation Test automation objectives (and exercise) Responsibilities Pilot project Measures for automation Return on Investment (ROI) (and exercise)

Session 2: Testware Architecture Importance of a testware architecture What needs to be organised

Session 3: Pre- and Post-Processing Automating more than tests Test status

Session 4: Scripting Techniques Objectives of scripting techniques Different types of scripts Domain specific test language

Session 5: Automated Comparison Automated test verification Test sensitivity Comparison example

Session 6: Final Advice, Q&A and Direction Strategy exercise Final advice Questions and Answers

Abstract Many organisations have invested a lot of time and effort into test automation but they have not achieved the significant returns that they had expected. Some blame the tool that they use while others conclude test automation doesn't work well for their situation. The truth is often very different. These organisations are typically doing many of the right things but they are not addressing key issues that are vital to long term success with test automation. Mark Fewster describes the most important issues that you must address, and helps you understand and choose the best approaches for your organization—no matter which automation tools you use. Management issues including responsibilities, automation objectives and return on investment are covered along with technical issues such as testware architecture, pre- and post-processing and automated comparison techniques. The target audience for this tutorial is people involved with managing test automation who need to understand the key issues in making test automation successful. Technical issues are covered at a high level of understanding; there are no tool demos! Biography Mark has over 30 years of industrial experience in software testing ranging from test management to test techniques and test automation. In the last two decades Mark has provided consultancy and training in software testing, published papers and co-authored two books with Dorothy Graham, "Software Test Automation” and “Experiences of Test Automation”. He is a popular speaker at national and international conferences and seminars, and has won the Mercury BTO Innovation in Quality Award. Mark has served on the committee of the British Computer Society's Specialist Interest Group in Software Testing (BCS SIGiST) and is currently helping ISTQB in defining the expert level certification for test automation.

presented by Mark [email protected] © Mark Fewster & Dorothy Graham 2014

0-1


Prepared by

Dorothy Graham [email protected]

© Mark Fewster and Dorothy Graham 2014

Mark FewsterGrove Software Testing Ltd.

[email protected]

and

0.1

Objectives of this tutorial

● help you achieve better success in automation■ independent of any particular tool

● mainly management and some technical issues■ objectives for automation■ showing Return on Investment (ROI)■ importance of testware architecture■ practical tips for a few technical issues■ what works in practice (case studies)

● help you plan an effective automation strategy0.2


0-2

Tutorial contents1) planning & managing test automation2) testware architecture3) pre and post processing4) scripting techniques5) automated comparison6) final advice, Q&A and direction0.3

Shameless commercial plug

Part 1: How to do automation - still relevant today, though we plan to update it at some point!

Latest book

(2012)

testautomationpatterns.org0.4


0-3

What is today about? (and not about)

● test execution automation (not other tools)● We will NOT cover:

■ demos of tools (time, which one, expo)■ comparative tool info (expo, web)■ selecting a tool*

● at the end of the day■ understand technical and non-technical issues■ have your own automation objectives■ plan your own automation strategy

* Mark will email you Chapter 10 of the STA book on request – [email protected]

1-Managing

presented by Mark [email protected]

© Mark Fewster &Dorothy Graham 2014

Planning & Managing Test Automation


1 Managing 2 Architecture 3 Pre- and Post

4 Scripting 6 Advice5 Comparison

1.1


Contents

Managing

1 2 3

4 5 6

Test automation objectivesResponsibilitiesAutomation in agilePilot projectMeasures for automationReturn on Investment (ROI)1.2

1-Managing



An automation effort

● is a project (getting started or major changes)■ with goals, responsibilities, and monitoring■ but not just a project – ongoing effort is needed

● not just one effort – continuing ■ when acquiring a tool – pilot project■ when anticipated benefits have not materialized■ different projects at different times

►with different objectives● objectives are important for automation efforts

■ where are we going? are we getting there?1.3

fasttesting

slowtesting

Effectiveness

Low

High

EfficiencyManual testing Automated

Efficiency and effectiveness

poorfast

testing

poorslow

testing

goodgood

greatestbenefit

not good but commonworst

better

1.4

1-Managing



Common test automation objectives

● choose your most important objective:1. faster testing2. run more tests3. reduce testing costs4. automate x% of tests5. find more bugs6. other: _______________________________

cheaper testing, less effort, fewer testers

more testing, more oftenincrease test coverage

reduce elapsed time, shorten schedule, time to market

better testing,improve software quality

automate 100% of testing

Exercise

1.5

Same testsautomated

edit tests (maintenance) set-up execute

analysefailures clear-up

Manualtesting

More matureautomation

Faster testing?

1.6

1-Managing



Run more tests?

● which is better:■ 100 1-minute tests or 1 60-minute test?

● not the only aspect:■ failure analysis may be much longer for the 60-minute test!

25% 25% 25% 25%

Setup Com Uniq TD100 1-min tests:unique testing = 15 sec x 100 = 25 mins

25% 50% 25%

Setup Unique TD1 60-min test:unique testing = 50%= 30 mins +

1.7

Run more tests?

● 3 sets of tests■ A: 100 tests, easy to automate■ B: 60 tests, moderately difficult to automate■ C: 30 tests, hard to automate

● what if the next release leaves A unchanged but has major changes to B and C?“Good automation is not found in the

number of tests run but in thevalue of the tests that are run”

1.8

1-Managing



Reduce testing costs?

Cost

Time (months / years)

Automationeffort

Total effort

Yes, but not to zero!

1.9

Manual testing effort(without automation)

Manual testing effort(with automation)

Automate x% of the manual tests?

manualtests automated

tests

tests notworth

automatingexploratory

test automation

manual tests automated (% manual)

tests (& verification) not possible to do

manually

tests notautomated

yet

1-Managing



What is automated?

regression tests exploratory testing

likelihood offinding bugs

most oftenautomated

1.11

Find more bugs?

● tests find bugs, not automation● automation is a mechanism for running tests● the bug-finding ability of a test is not affected by the manner in which it is executed● this can be a dangerous objective

■ especially for regression automation!Automated tests Manual Scripted Exploratory Fix Verification

9.3% 24.0% 58.2% 8.4%

Experiences of Test Automation, Ch 27, p 503, Ed Allen & Brian Newman 1.12

1-Managing



When is “find more bugs” a good objective for automation?

● objective is “fewer regression bugs missed”● when the first run of a given test is automated

■ MBT, Exploratory test automation, automated test design■ keyword-driven (e.g. users populate spreadsheet)

● find bugs in parts we wouldn’t have tested?■ indirect! (direct result of running more tests)

1.13

Good objectives for test automation

● realistic and achievable● short and long term● regularly re-visited and revised● measurable● should be different objectives for testing and for automation● automation should support testing activities

Pattern: SET CLEAR GOALS1.14

1-Managing



Trying to get started: Tessa Benzie

■ consultancy to start automation effort►project, needs a champion – hired someone► training first, something else next, etc.

■ contract test manager – more consultancy►bought a tool – now used by a couple contractors►TM moved on, new QA manager has other priorities

■ just wanting to do it isn’t enough►needs dedicated effort►now have “football teams” of manual testers

Chapter 29, pp 535, Experiences of Test Automation 1.15


Contents

Managing

1 2 3

4 5 6


1-Managing



What is an automated test?

● a test!■ designed by a tester for a purpose

● test is executed■ implemented / constructed to run automatically using a tool■ or run manually

● who decides which tests to run?● who decides how a test is run?

1.17

Test manager’s dilemma

● who should undertake automation work■ not all testers can automate (well)■ not all testers want to automate■ not all automators want to test!

● conflict of responsibilities ■ (if you are both tester and automator)■ should I automate tests or run tests manually?

● get additional resources as automators?■ contractors? borrow a developer? tool vendor?

1.18

1-Managing



Relationships

19

engine:test tool

passengers:test cases

car: testinfrastructure

driver:tester

mechanic:test automator

Testers

● test the software■ design tests■ select tests for automation

► requires planning / negotiation● execute automated tests

■ should not need detailed technical expertise● analyse failed automated tests

■ report bugs found by tests■ problems with the tests may need help from the automation team

Automators

● automate tests (requested by testers)● support automated testing

■ allow testers to execute tests■ help testers debug failed tests■ provide additional tools

● predict■ maintenance effort for software changes■ cost of automating new tests

● improve the automation■ more benefits, less cost

Responsibilities

1.20

1-Managing



Good testing

● is effective■ finds most of the faults (90+%)■ gives confidence

● is efficient■ uses a few small test cases

● is flexible■ can use different subsets of test cases for different test objectives

Good automation

● easy to use■ flexible: supports different requirements for automation■ responsive: quick changes when needed

● cheap to use■ build & maintain automated tests■ failure analysis

● improve over time1.21

Testing versus automation

Roles for automation

● Testware architect■ designs the overall structure for the automation

● Champion■ “sells” automation to managers and testers

● Tool specialist / toolsmith■ technical aspects, licensing, updates to the tool

● Automated script developers■ write new scripts as needed (e.g. keyword)■ debug automation problems

1.22

1-Managing




Contents

Managing

1 2 3

4 5 6


Agile automation: Lisa Crispin

■ starting point: buggy code, new functionality needed,whole team regression tests manually■ testable architecture: (open source tools)

►want unit tests automated (TDD), start with new code► start with GUI smoke tests - regression►business logic in middle level with FitNesse

■ 100% regression tests automated in one year► selected set of smoke tests for coverage of stories

■ every 6 mos, engineering sprint on the automation■ key success factors

►management support & communication►whole team approach, celebration & refactoring

1.24Chapter 1, pp 17-32, Experiences of Test Automation

1-Managing



Automation and agile

● agile automation: apply agile principles to automation■ multidisciplinary team■ automation sprints■ refactor when needed

● fitting automation into agile development■ ideal: automation is part of “done” for each sprint

►Test-Driven Design = write and automate tests first■ alternative: automation in the following sprint ->

►may be better for system level tests1.25See www.satisfice.com/articles/agileauto-paper.pdf (James Bach)

Automation in agile/iterative development

1.26

Amanual testing of

this release (testers)A B

B CA

FEDCBA

regression testing (automators automate the best tests)

run automated tests (testers)

1-Managing



Requirements for agile test framework

● support manual and automated testing ■ using the same test construction process

● support fully manual execution at any time■ requires good naming convention for components

● support manual + automated execution ■ so test can be used before it is 100% automated

● implement reusable objects● allow “stubbing” objects before GUI available

1.27Source: Dave Martin, LDSChurch.org, email

A tale of two projects: Ane Clausen

■ Project 1: 5 people part-time, within test group►no objectives, no standards, no experience, unstable►after 6 months was closed down

■ Project 2: 3 people full time, 3-month pilot►worked on two (easy) insurance products, end to end►1st month: learn and plan, 2nd & 3rd months: implement► started with simple, stable, positive tests, easy to do► close cooperation with business, developers, delivery►weekly delivery of automated Business Process Tests

■ after 6 months, automated all insurance products1.28Chapter 6, pp 105-128, Experiences of Test Automation

1-Managing




Contents

Managing

1 2 3

4 5 6


Pilot project

● reasons■ you’re unique■ many variables / unknowns / options■ learn how to start / improve

● benefits■ find the best way for you■ solve problems once■ establish confidence (based on experience)■ set realistic targets

● objectives■ demonstrate tool value■ gain experience / skills in the use of the tool■ identify changes to existing test process■ set internal standards and conventions■ refine assessment of costs and achievable benefits

1.30

1-Managing



Characteristics of a pilot project

1.31

Planned

Important

Learning

Objective

Timely

resourced, targets, contingency

full time work, worthwhile tests

informative, useful, revealing

quantified, not subjective

short term, focused

P

I

L

O

T

What to explore in the pilot

● build / implement automated tests (architecture)■ different ways to build stable tests (e.g. 10 – 20)

● maintenance■ different versions of the application■ reduce maintenance for most likely changes

● failure analysis■ support for identifying bugs■ coping with common bugs affecting many automated tests

1.32Also: naming conventions, reporting results, measurement

1-Managing



After the pilot…

● having processes & standards is only the start■ 30% on new process■ 70% on deployment

►marketing, training, coaching► feedback, focus groups, sharing what’s been done

● the (psychological) Change Equation■ change only happens if (x + y + z) > w

1.33

Source: Eric Van Veenendaal, successful test process improvement


Contents

Managing

1 2 3

4 5 6


1-Managing



Why measure automation?

● to justify and confirm starting automation■ business case for purchase/investment decision, to confirm ROI has been achieved e.g. after pilot■ both compare manual vs automated testing

● to monitor on-going automation “health”■ for increased efficiency, continuous improvement■ build time, maintenance time, failure analysis time, refactoring time

►on-going costs – what are the benefits?■ monitor your automation objectives

1.35

Useful measures

● a useful measure:“supports effective analysis and decision making, and that can be obtained relatively easily.”Bill Hetzel, “Making SoftwareMeasurement Work”, QED, 1993.● easy measures may be more useful even though less accurate (e.g. car fuel economy)● ‘useful’ depends on objectives, i.e. what you want to know

1.36

1-Managing



EMTE – what is it?

● Equivalent Manual Test Effort■ given a set of automated tests,■ EMTE is how much effort would it take

► to run those tests manually● note

■ you would not actually run these tests manually■ EMTE = is the effort you would have spent if you had run the tests manually■ EMTE can be used to show some test automation benefit

1.37

Monitoring test automation health

● important to do (often neglected)■ need to distinguish between test automation progress and test automation health

● progress examples■ number of tests automated■ coverage achieved by automated tests■ number of test cycles executed / release

● health examples■ benefits■ build cost■ analysis cost■ maintenance cost 1.38

ROI = benefit – costcost

1-Managing



Internal objectives for test automation

● provide automation services■ efficient building, use and maintenance of automated tests■ decrease automation costs over time■ increase automation benefits

► savings, ease of use, flexibility● measured ROI (appropriate for test objectives)

■ Equivalent Manual Test Effort (EMTE)► additional test hours► hours of unattended tests performed

■ proportion of unattended testing■ increased coverage■ reduce elapsed time

1.39

Measure benefit

● equivalent manual test effort (EMTE)■ hours of additional testing■ hours of unattended testing

● number of tests■ tests executed■ additional (new) tests■ repeated tests

● number of test cycles■ additional cycles

● increased coverage1.40

Relate target to total cost of automation, e.g. benefit 10 times total cost

Suggestion

the degree to which automation has supportedtesters in achieving their objectives

1-Managing



An example comparative benefits chart

1.41

01020304050607080

exec speed times run data variety tester work

manaut

ROI spreadsheet – email me for a copy

14 x faster 5 x more often 4 x more data 12 x less effort

Measure build effort

● time taken to automate tests■ hours to add new or existing manual tests■ average across different test types

● proportion of equivalent manual test effort■ e.g. 1 hour to automate 30 minute manual test= 2 times equivalent manual test effort

1.42

Target: < 2 times:decreasing 10% per year

Suggestion

put your own(more appropriate)figures here

1-Managing



Measure failure analysis effort

● analysis effort for each test■ captured in defect report■ effort from first recognition through to resumption of test execution■ average hours (or minutes) per failed test case

►needs comparison of same with manual testing►must also monitor defect reporting effectiveness

– e.g. how many ‘non reproducible’ reports

1.43

Target: X minutes?Trend:stable

Suggestionput your own(more appropriate)figure here

Measure maintenance effort

● maintenance effort of automated tests■ percentage of test cases requiring maintenance■ average effort per test case■ percentage of equivalent manual test effort

1.44

Target: < 10%Trend:stable or decreasing

Suggestion

put your own( appropriate)figure here

1-Managing



Recommendations

● don’t measure everything!● choose three or four measures

■ applicable to your most important objectives● monitor for a few months

■ see what you learn● change measures if they don’t give useful information

1.45


Contents

Managing

1 2 3

4 5 6


1-Managing



Is this Return on Investment (ROI)?

● tests are run more often● tests take less time to run● it takes less human effort to run tests● we can test (cover) more of the system● we can run the equivalent of days / weeks of manual testing in a few minutes / hours● faster time to market

1.47

these are (good) benefitsbut are not ROIROI = (benefit – cost)

cost

Examples of ROI achieved

● Michael Snyman, S African bank (Ch 29.13)■ US$4m on testing project, automation $850K■ savings $8m, ROI 900%

● Henri van de Scheur, Database testing (Ch 2)■ results: 2400 times more efficient

● Stefan Mohacsi, Armin Beer: European Space Agency (Ch 9)■ MBT, break even after four test cycles

1.48

from: Experiences of Test Automation book

1-Managing



How important is ROI?

● ROI can be dangerous■ easiest way to measure: tester time■ may give impression that tools replace people

● “automation is an enabler for success, not a cost reduction tool”►Yoram Mizrachi, “Planning a mobile test automation strategy that works, ATI magazine, July 2012

● many achieve lasting success without measuring ROI (depends on your context)■ need to be measure benefits (and publicize them)

1.49


Managing

1 2 3

4 5 6Note in your Summary Sheet the key points for you from this session

Summary: key points

1.50

• Assign responsibility for automation (and testing)• Use a pilot project to explore the good methods• Know your automation objectives• Measure what’s important to you• Show ROI from automation Now complete the exercise!

2-Architecture



Testware Architecture



Ref. Chapter 5: Testware Architecture“Software Test Automation”


2.1


Contents

Architecture

1 2 3

4 5 6

Importance of a testware architectureWhat needs to be organised

2.2

2-Architecture



Testware architecture

● organisation of, and relationship between, artefacts■ scripts, input, test data, test descriptions, expected results, actual results, log files, etc.

● if you create an automated test:■ do you know what already exists that you can use?

► scripts, data, tools■ do you know where to put the artefacts?

► script(s), data, expected results, etc.■ do you know what names to use?

● if you execute an automated test:■ do you know where to find all the test results?■ do you know how to analyse a test failure? 2.3

Testware architecture

test

war

e ar

chite

ctur

e

Testers

Test Execution Tool

High Level Keywords

Structured Scripts

structuredtestware

TestAutomator(s)

write tests (in DSTL)

runs scripts

abstraction here= easier to change tools and maintain

= long life

abstraction here= easier to write automated tests

= widely used

(tes

t fra

mew

ork)

2.4

2-Architecture



Architecture – abstraction levels

● most critical factor for success■ worst: close ties between scripts, tool & tester

● separate testers’ view from technical aspects■ so testers don’t need tool knowledge

► for widespread use of automation► scripting techniques address this

● separate tests from the tool – modular design■ likely changes confined to one / few module(s)■ re-use of automation functions■ for minimal maintenance and long-lived automation

2.5

Localised regimes

● “everyone will do the sensible thing”■ most will do something sensible, but different

● “use the tool however it best suits you”■ ignores cost of learning how best to automate

● problems include:■ effort wasted repeatedly solving the same problems in different ways■ no re-use between teams■ multiple learning curves

2.6

2-Architecture



Easy way out: use the tool’s architecture

● tool will have its own way of organising tests■ where to put things (for the convenience of the tool!)■ will “lock you in” to that tool – good for vendors!

● a better way (gives independence from tools)■ organise your tests to suit you■ as part of pre-processing, copy files to where the tool needs (expects) to find them■ as part of post-processing, copy back to where you want things to live

2.7

Tool-specific vs generic scripts

A

Tool A

A

A

A

B

Tool B

B

B

B

Tool A

A

G

G

G

Tool B

B

New env

2.8

2-Architecture



Test-specific vs reused scripts

T1

Test 1

T1

T1

T2

Test 2

T2

T2

T1

Test 1

R

R

T2

Test 2 Test 1 Test 2

R

R

Test Definition

2.9

TESTNAME: <name of test>

PURPOSE: <single sentence explaining test purpose>

MATERIALS:<a list of the artefacts used by this test>

RESULTS:<a list of the artefacts produced by this test>

SETUP:<sequence of keywords implementing setup actions>

TEARDOWN:<sequence of keywords implementing teardown actions>

EXECUTION:<sequence of keywords implementing the test actions>

Test definition

2.10

2-Architecture



Test definition additional info

● measures■ expected run time (of the automated test)■ EMTE (equivalent manual test effort)■ others?

● attributes, such as test selector tags■ tag specific sets of tests so they can be selected to be run■ examples: smoke tests, short tests, bug fix tests. long tests, specific environment tests

2.11

General control script

For each test to be executed

Read keyword definitionVerify keyword definition

Check specified materials existExecute setup

Execute test

Perform post-execution comparisonExecute teardownCheck specified results exist

Report results

EndFor

2.12

2-Architecture



Learning is incremental: Molly Mahai

■ book learning – knew about investment, not replace people, don’t automate everything, etc.■ set up good architecture? books not enough■ picked something to get started

►after a while, realised limitations► too many projects, library cumbersome

■ re-designed architecture, moved things around■ didn’t know what we needed till we experienced the problems for ourselves

► like trying to educate a teenager2.13Chapter 29, pp 527-528, Experiences of Test Automation


Contents

Architecture

1 2 3

4 5 6

Importance of a testware architectureWhat needs to be organised

2.14

2-Architecture



A test for you

● show me one of your automated tests running■ how long will it take before it runs?

● typical problems■ fails: forgot a file, couldn’t find a called script■ can’t do it (yet):

► Joe knows how but he’s out, ►environment not right, ►haven’t run in a while, ►don’t know what files need to be set up for this script

● why not: run up your framework, select test, GO2.15

Key issues

● scale■ the number of scripts, data files, results files, benchmark files, etc. will be large and growing

● shared scripts and data■ efficient automation demands reuse of scripts and data through sharing, not multiple copies

● multiple versions■ as the software changes so too will some tests but the old tests may still be required

● multiple environments / platforms2.16

2-Architecture



Terms

- Testware artefacts

Testware

Products By-Products

Test Materials Test Results

logsstatus

summarydifferences

logsstatus

summarydifferences

actual results

scripts

data

inputs

expected results

doc (specifications)

env utilities

2.17

Sharedscript

saveas.scp

Sharedscript

open.scp

Scribble

Testware for example test case

diffs.txt

Differences

Test script:- test input

countries.scp

Log

log.txt

countries2.dcm

ExpectedOutput

TestSpecification

testspec.txt

countries.dcm

InitialDocument

countries2.dcm

EditedDocument

Compare

2.18

2-Architecture



Testware by type

Testware

countries2.dcm

Products By-Products

Test Materials Test Results

saveas.scp

countries.dcm

countries.scp

testdef.txt

open.scp

countries2.dcm

log.txt

diff.txt

status.txt

2.19

Benefits of standard approach

● tools can assume knowledge (architecture)■ they need less information; are easier to use; fewer errors will be made

● can automate many tasks■ checking (completeness, interdependencies); documentation (summaries, reports); browsing

● portability of tests■ between people, projects, organisations, etc.

● shorter learning curve2.20

2-Architecture



Testware Sets

● Testware Set:■ a collection of testware artifacts■ four types:

►Test Set - one or more test cases►Script Set - scripts used by two or more Test Sets►Data Set - data files used by two or more Test Sets►Utility Set - utilities used by two or more Test Sets

■ good software practice: look for what is common, and keep it in only one place!■ Keep your testware DRY!

2.21

Testware library

● repository of master versions of all Testware Sets■ “uncategorised scripts are worse than no scripts”

► Onaral & Turkmen● CM is critical

■ “If it takes too long to update your test library, automation introduces delay instead of adding efficiency” ► Linda Hayes, AST magazine, Sept 2010

2.22

d_ScribbleTypical v1d_ScribbleTypical v2d_ScribbleVolume v1s_Logging v1s_ScribbleDocument v1s_ScribbleDocument v2s_ScribbleDocument v3s_ScribbleNavigate v1t_ScribbleBreadth v1t_ScribbleCheck v1t_ScribbleFormat v1t_ScribbleList v1t_ScribbleList v2t_ScribbleList v3t_ScribblePrint v1t_ScribbleSave v1t_ScribbleTextEdit v1t_ScribbleTextEdit v2u_ScribbleFilters v1u_GeneralCompare v1

Versionnumbers

2-Architecture



Separate test results

Testsuite

A single test suite ...

Softwareunder test

... may be used on subtlydifferent versions of software ...

Testresults

... and producedifferent setsof test results ...

... that wewish to keep

2.23

Incremental approach: Ursula Friede

■ large insurance application■ first attempt failed

►no structure (architecture), data in scripts■ four phases (unplanned)

►parameterized (dates, claim numbers, etc)►parameters stored in single database for all scripts► improved error handling (non-fatal unexpected events)►automatic system restart

■ benefits: saved 200 man-days per test cycle►€120,000!

2.24Chapter 23, pp 437-445, Experiences of Test Automation

2-Architecture




Architecture

1 2 3

4 5 6

Summary: key points

2.25

• Structure your automation testware to suit you• Testware comprises many files, etc. which need to be given a home• Use good software development standardsNote in your Summary Sheet the key points for you from this session

3-Pre and Post Processing



Pre- and Post-Processing



6 Advice4 Scripting

Ref. Chapter 6: Automating Pre- and Post-Processing“Software Test Automation”

5 Comparison

3.1


Contents

Pre and Post

1 2 3

4 5 6

Automating more than testsTest statusContents

3.2




What is pre- and post-processing?

● Pre-processing■ automation of setup tasks necessary to fulfil test case prerequisites

● Post-processing■ automation of post-execution tasks necessary to complete verification and house-work

● These terms are useful because:■ there are lots of tasks, they come in packs, many are the same, and they can be easily automated

3.3

Automated tests/automated testing

Select / identify test cases to runSet-up test environment:

• create test environment• load test data

Repeat for each test case:• set-up test pre-requisites• execute• compare results• log results• analyse test failures• report defect(s)• clear-up after test case

Clear-up test environment:• delete unwanted data• save important data

Summarise results

Automated tests

Select / identify test cases to runSet-up test environment:

• create test environment• load test data

Repeat for each test case:• set-up test pre-requisites• execute• compare results• log results• clear-up after test case

Clear-up test environment:• delete unwanted data• save important data

Summarise resultsAnalyse test failuresReport defects

Automated testing

Automated processManual process 3.4




Examples

● pre-processing■ copy scripts from a common script set (e.g. open, saveas)■ delete files that shouldn’t exist when test starts■ set up data in files■ copy files to where the tool expects to find them■ save normal default file and rename the test file to the default (for this test)

● post-processing■ copy results to where the comparison process expects to find them■ delete actual results if they match expected (or archive if required)■ rename a file back to its normal default

3.5

Outside the box: Jonathan Kohl

■ task automation (throw-away scripts)►entering data sets to 2 browsers (verify by watching)► install builds, copy test data

■ support manual exploratory testing■ testing under the GUI to the database (“side door”)■ don’t believe everything you see

►1000s of automated tests pass too quickly►monitoring tools to see what was happening► “if there’s no error message, it must be ok”

– defects didn’t make it to the test harness– overloaded system ignored data that was wrong

Chapter 19, pp 355-373, Experiences of Test Automation 3.6




Automation +

execution

comparison

traditional test

automation

DSTLstructured

testwarearchitecture

loosen youroracles

ETA,monkeys

manual

testing

3.7


Contents

Pre and Post

1 2 3

4 5 6

Automating more than testsTest status

3.8




Test status – pass or fail?

● tool cannot judge pass or fail■ only “match” or “no match”■ assumption: expected results are correct

● when a test fails (i.e. the software fails)■ need to analyse the failure

► true failure? write up bug report► test fault? fix the test (e.g. expected result)►known bug or failure affecting many automated tests?

– this can eat a lot of time in automated testing– solution: additional test statuses

3.9

Test statuses for automation

• other possible additional test statuses– test blocked– environment problem (e.g. network down, timeouts)– set-up problems (files missing)– test needs to be changed but not done yet

Compare to No differences found Differences found(true) expected outcome Pass Failexpected fail outcome Expected Fail Unknowndon’t know / missing Unknown Unknown

3.10





Note in your Summary Sheet the key points for you from this session

Pre and Post

1 2 3

4 5 6

Summary: key points

• Pre- and post processing to automate setup and clear-up tasks• Test status is more than just pass / fail

3.11

4-Scripting



Scripting Techniques


Ref. Chapter 3: Scripting Techniques“Software Test Automation”



4.1


Contents

Scripting

1 2 3

4 5 6

Objectives of scripting techniquesDifferent types of scriptsDomain specific test language

4.2

4-Scripting



Scripting

● the backbone of automation● four broad approaches

■ linear■ structured■ data-driven■ keyword-driven

● issues■ more code means more maintenance■ programmers like programming

► may write new code even when unnecessary● do you know your script to test ratio?

test specific programming required

framework supports abstraction

4.3

Objectives of scripting techniques

● implement your testware architecture● to reduce costs

■ make it easier to build automated tests► avoid duplication

■ avoid excessive maintenance costs► greater reuse of functional, modular scripting

● greater return on investment■ better testing support■ greater portability

► environments & hardware platforms● enhance capabilities

■ achieve more testing for same (or less) effort► testing beyond traditional manual approaches

probably best achieved by data-driven or keyword-driven4.4

4-Scripting



re-usable modules / script library

Progression of automation implementation

low-levelinstructions

structuredcode

datadriven

keyworddriven

manual testscripts

inputdata

testdefinitions

get quote - motor, age, …create policy – motor, age, …

TechnicalTestAnalyst

TestAnalyst

easier maintenance

Source: Mark Fewster, Grove Consultants (grove.co.uk) 4.5


Contents

Scripting

1 2 3

4 5 6


4.6

4-Scripting



Example application

Tax Calculator

Name:

EarningsJanuary

FebruaryMarch

Tax band:Tax due:

input fields

calculated outputs

£290

B£45

Fred

£100£80

£110

Test 1:1. open application2. enter name & earnings3. save results4. close application

4.7

TestTool

Structured scripting

(manual) testprocedures

TestTool

software under test

create

testscripts

scriptlibrary

high-levelinstructions

and test datalow-level“how to”

instructions

automatorstesters

software under test

4.8

4-Scripting



About structured scripting

● script library for re-used scripts■ part of testware architecture implementation

► shared scripts interface to software under test►all other scripts interface to shared scripts

● reduced costs■ maintenance

► fewer scripts affected by software changes■ build

► individual control scripts are smaller and easier to read (a ‘higher’ level language is used)4.9

Example structured scripts

10

Sub Test1()Call OpenApplication(“TaxCalculator”)Call CalculateTax(“Pooh”, 100, 150, 125)Call SaveAsAndClose(“Test 1 Results”)

End Sub

Sub OpenApplication(Application)Workbooks.Open Filename:= _

"ATT “ & Application & ”.xls“End Sub

Sub CalculateTax(Name, M1, M2, M3)ActiveCell.FormulaR1C1 = NameRange("C6").SelectActiveCell.FormulaR1C1 = M1Range("C7").SelectActiveCell.FormulaR1C1 = M2Range("C8").SelectActiveCell.FormulaR1C1 = M3

End Sub

Sub SaveAsAndClose(Filename)ActiveWorkbook.SaveAs Filename:= _

Filename & “.xls", FileFormat:= _xlNormal, Password:="“

ActiveWorkbook.CloseEnd Sub

Supporting scripts:

Main test script:

additional tests canbe created more easily

4-Scripting



Usable (re-usable) scripts

● to re-use a script, need to know:■ what does this script do?■ what does it need?■ what does it deliver?■ what state when it starts?■ what state when it finishes?

● information in a standard place for every script■ can search for the answers to these questions

4.11

TestToolTestTool

automatorstesters

Data driven

(manual) testprocedures softw

are under test

scriptlibrary

low-level“how to”

instructions


test data

create

datafiles

create

controlscripts

4.12

4-Scripting



About data driven

● test data extracted from scripts■ placed into separate data files

● control script reads data from data file■ one script implements several tests by reading different data files (reduces script maintenance per test)

● reduced build cost■ faster and easier to automate similar test procedures■ many test variations using different data

● multiple control scripts required■ one for each type of test (with varying data)

4.13

Example data-driven script

14

Name Earn1 Earn2 Earn3Pooh 100 150 125Piglet 75 90 80Roo 120 110 65

data file: TaxCalcData.csvSub RunTests()

For each Row in fileCall OpenApplication(“TaxCalculator”)Open TaxCalcData.csvRead NameRead Earn1Read Earn2Read Earn3Call CalculateTax(Name, Earn1, _

Earn2, Earn3)Call SaveAsAndClose(Test & “ Results”)

Next RowEnd Sub

main script (runs all tests in data file)

4-Scripting



TestToolTestTool

automatorstesters

Keywords (basic)softw

are under test

(manual) testprocedures

create

testdefinitions

controlscript


and test data

scriptlibrary low-level “how to”

instructions and keywords

single control script:“interpreter” / ITE

4.15

About keywords

● single control script (Interactive Test Environment)■ improvements to this benefit all tests (ROI)■ extracts high-level instructions from scripts

● ‘test definition’■ independent of tool scripting language■ a language tailored to testers’ requirements

► software design►application domain►business processes

● more tests, fewer scriptsUnit test: calculate

one interest payment

System test: summarise interest for one customer

Acceptance test: end of day run, all interest payments 4.16

4-Scripting



Example keyword-driven scripts

CalculateTax Pooh, 100, 150, 125CalculateTax Piglet, 75, 90, 80CalculateTax Roo, 120, 110, 65

Keyword file (option 1)

CalculateTaxPooh, 100, 150, 125Piglet, 75, 90, 80Roo, 120, 110, 65

Keyword file (option 2)

Keyword file (Robot Framework)Test Case Action Arg1 Arg2 Arg3 Arg4Check Tax Calculate Tax Pooh 100 150 125

Calculate Tax Piglet 75 90 80Calculate Tax Roo 120 110 65

CalculateTax Datafile.txtKeyword file (option 3)

Name Earn1 Earn2 Earn3Pooh 100 150 125Piglet 75 90 80Roo 120 110 65

Datafile.txt

4.17

Minimising automation code

● more code means more maintenance■ better to reuse existing code than write new code

► that does the same thing■ achieved by

► clear objectives– e.g. automate for minimal maintenance

►use of appropriate scripting approach– abstraction

► careful design– consider sets of tests, not individual tests

► consistency (standards)4.18

4-Scripting




Contents

Scripting

1 2 3

4 5 6


4.19

procedures/definitions

test


and test data

TestTool

Merged test procedure/test definition

software under test

create

controlscript


and test data

scriptlibrary

low-level “how to”instructions and keyword scripts

single control script:“interpreter” / ITE

language for testers 4.20

4-Scripting



Domain Specific Test Language (DSTL)

● test procedures and test definitions similar■ both describe sequences of test cases

►giving test inputs and expected results● combine into one document

■ can include all test information■ avoids extra ‘translation’ step■ testers specify tests regardless manual/automated■ automators implement required keywords

4.21

Keywords in the test definition language

● multiple levels of keywords possible■ high level for business functionality■ low level for component testing

● composite keywords■ define keywords as sequence of other keywords■ gives greater flexibility (testers can define composite keywords) but risk of chaos

● format■ freeform, structured, or standard notation

► (e.g. XML)4.22

4-Scripting



Example use of keywords

Create a new account, order 2 items and check out

Firstname Surname Email address Password

Create Account Edward Brown [email protected] apssowdr

Item Num Items Check Price for Items

Order Item 1579 3 15.30

Order Item 2598 12.99

Total

Checkout 28.29

4.23

Documenting keywords

Name Name for this keywordPurpose What this keyword doesParameters Any inputs needed, outputs producedPre-conditions

What needs to be true before using it, where valid

Post-conditions What will be true after it finishesError conditions

What errors it copes with, what is returned

Example An example of the use of the keyword

Source: Martin Gijsen. See also Hans Buwalda book & articles 4.24

4-Scripting



Example keyword Create account

*mandatory

Name Create account

Purpose Creates a new account

Parameters *First name: 2 to 32 characters

*Last name: 2 to 32 characters

*Email address: also serves as account id

*Password: 4 - 32 characters

Pre-conditions Account doesn't exist for this person

Post-conditions Account created (including email confirmation)

order Screen displayed

Error conditions Account already exists

Example (see example)

4.25

Example keyword Order item

*mandatory

Name Order item

Purpose Order one or more of a specific itemParameters *Item number: 1000 to 9999, in catalogue

Number of items wanted: 1 to Max-for-item. If blank, assumes 1

Pre-conditions Valid account logged inItem in stock (sufficient for the order)Prices available for item (including discounts for many)

Post-conditions Item(s) appear in shopping basketNumber of available items decreased by number ordered

Error conditions Insufficient items in stockExample (see example)

4.26

4-Scripting



Implementing keywords

● ways to implement keywords■ scripting language (of a tool)■ programming language (e.g. Java)■ use what your developers are most familiar with!

● ways of supporting a DSTL■ commercial, open source or home-grown framework■ spreadsheet or database for test descriptions

4.27

Frameworks

● commercial tools■ ATRT, ATSM, Axe, Certify, eCATT, FASTBoX, GUIdancer, Liberation, Ranorex, te52, TestComplete, TestDrive, TestingWhiz, Tosca Testsuite, zest

● open source■ Cacique, FitNesse, Helium, Jameleon, JET, JSystem, Jubula, Maveryx, Open2Test, Power Tools, QAliber Test Builder, Rasta, Robot Framework, SAFS, SpecFlow, STAF, TAF, TAF Core, TestMaker, Xebium

● I can email you my Tool List ■ test execution and framework tools■ [email protected]

4.28

4-Scripting



TestTool

software under test

scriptlibraries

tool dependenttool independent

Execution-tool-independent framework

frame-work

testprocedures/definitions

tool independentscripting language

AnotherTestTool

sut

TestTool

software under test

scriptlibraries

some tests run manually

4.29



Scripting

1 2 3

4 5 6

Summary

● objectives of good scripting■ reduce costs, enhancecapabilities

● many types of scripting■ structured, data-driven,keyword

● keyword/DSTL the mostsophisticated■ yields significant benefits

● increased productivity■ customised front end,tool independence

4.30

5-Comparison



Automated Comparison


Ref. Chapter 4: Automated Comparison“Software Test Automation”



5.1


Contents

Comparison

1 2 3

4 5 6

Automated test verificationTest sensitivity

5.2

5-Comparison



Perverse persistence: Michael Williamson

■ testing Webmaster tools at Google (new to testing)■ QA used Eggplant (image processing tool)■ new UI broke existing automation■ automate 4 or 5 functions■ comparing bitmap images – inaccurate and slow■ testers had to do automation maintenance

►not worth developers learning tool’s language■ after 6 months, went for more appropriate tools■ QA didn’t use the automation, tested manually!

► tool was just running in the backgroundChapter 17, pp 321-338, Experiences of Test Automation 5.3

Checking versus testing

■ checking confirms that things are as we think►e.g. check that the code still works as before

■ testing is a process of exploration, discovery, investigation and learning►e.g. what are the threats to value to stakeholders, give information

■ checks are machine-decidable► if it’s automated, it’s probably a check

■ tests require sapience► including “are the checks good enough”

Source: Michael Bolton, www.developsense.com/blog/2009/08/testing-vs-checking/5.4

5-Comparison



General comparison guidelines

● keep as simple as possible● well documented● standardise as much as possible● avoid bit-map comparison● poor comparisons destroy good tests● divide and conquer:

■ use a multi-pass strategy■ compare different aspects in each pass

5.5

Two types of comparison

● dynamic comparison■ done during test execution■ performed by the test tool■ can be used to direct the progress of the test

► e.g. if this fails, do that instead■ fail information written to test log (usually)

● post-execution comparison■ done after the test execution has completed■ good for comparing files or databases■ can be separated from test execution■ can have different levels of comparison

► e.g. compare in detail if all high level comparisons pass5.6

5-Comparison



Scribble

Comparison types compared

diffs.txt

Differences

Log

log.txt

countries2.dcm

ExpectedOutput

countries.dcm

InitialDocument

countries2.dcm

EditedDocument

Compare

Test script:-test input-comparisoninstructions

scribble1.scp

Error messageas expected?

dynamic comparison

post-executioncomparison 5.7

Comparison process

● few tools for post-execution comparison● simple comparators come with operating systems but do not have pattern matching

■ e.g, Unix ‘diff’, Windows ‘UltraCompare’● text manipulation tools widely available

■ sed, awk, grep, egrep, Perl, Tcl, Python● use pattern matching tools with a simple comparator to make a ‘comparison process’● use masks and filters for efficiency

5.8

5-Comparison




Contents

Comparison

1 2 3

4 5 6

Automated test verificationTest sensitivityContents

5.9

Test sensitivity

● the more data there is available:■ the easier it is to analyse faults and debug

● the more data that is compared:■ the more sensitive the test

● the more sensitive a test:■ the more likely it is to fail■ (this can be both a good and bad thing)

5.10

5-Comparison



Sensitive versus specific(robust) test

Unexpectedchange occurs

Sensitive testverifies theentire outcome

Test outcome

Specific testverifies thisfield only

Test is supposedto change onlythis field

5.11

Unexpectedchange occursfor every test

Three tests,each changesa different field

If all tests aresensitive, they

all show theunexpected change

If all tests arespecific, theunexpected

change ismissed

Testoutcome

Too much sensitivity = redundancy

5.12

5-Comparison



Using test sensitivity

● sensitive tests:■ few, at high level■ breadth / sanity checking tests■ good for regression / maintenance

● specific/robust tests:■ many, at detailed level■ focus on specific aspects■ good for development A good test

automation strategy will plan

a combination of sensitive and specific

tests 5.13



Comparison

1 2 3

4 5 6

Summary: key points

• Balance dynamic and post-execution comparison• Balance sensitive and specific tests• Use masking and filters to adjust sensitivity

5.14

5-Comparison



Comparison example

● Expected output■ Date 11 May 2011■ Order No X43578■ Login J Smith■ Add 1 mouse■ Add 1 mountain bike■ Add_to_total $15.99■ Add_to_total $249.99■ Total_due $265.98■ Logout J Smith

● Actual output■ Login M Jones■ Order No X54965■ Add 1 mountain bike■ Add_to_total $249.99■ Add 1 toaster■ Add_to_total $35.45■ Logout J Smith■ Total_due $285.34■ Date 16 Aug 2011

Has this test passed?5.15

Simple automated comparison

● Expected output■ Date 11 May 2011■ Order No X43578■ Login J Smith■ Add 1 mouse■ Add 1 mountain bike■ Add_to_total $15.99■ Add_to_total $249.99■ Total_due $265.98■ Logout J Smith

● Actual output■ Login M Jones■ Order No X54965■ Add 1 mountain bike■ Add_to_total $249.99■ Add 1 toaster■ Add_to_total $35.45■ Logout J Smith■ Total_due $285.34■ Date 16 Aug 2011

Fail

Fail

Fail

Fail

Fail

Fail

Fail

Fail

Fail

5.16

5-Comparison



Filter: alphabetical order

● Expected output■ Add 1 mountain bike■ Add 1 mouse■ Add_to_total $15.99■ Add_to_total $249.99■ Login J Smith■ Logout J Smith ■ Date 11 May 2011■ Order No X43578■ Total_due $265.98

● Actual output■ Add 1 mountain bike■ Add 1 toaster■ Add_to_total $249.99■ Add_to_total $35.45■ Login M Jones■ Logout J Smith■ Date 16 Aug 2011■ Order No X54965■ Total_due $285.34

Pass

Fail

Fail

Fail

Fail

Pass

Fail

Fail

Fail

1st Pass is a coincidence, 2nd Pass is actually a bug!5.17

Filters: replacement by object type

● Expected output■ Add <item>■ Add <item>■ Add_to_total $15.99■ Add_to_total $249.99■ Login J Smith■ Logout J Smith ■ Date <date>■ Order No <orderno>■ Total_due $265.98

● Actual output■ Add <item>■ Add <item>■ Add_to_total $249.99■ Add_to_total $35.45■ Login M Jones■ Logout J Smith■ Date <date>■ Order No <orderno>■ Total_due $285.34

This has helped eliminate things we aren’t interested in

Pass

Pass

Fail

Fail

Fail

Pass

Pass

Pass

Fail

5.18

5-Comparison



Filters: replacement for everything?

● Expected output■ Add <item>■ Add <item>■ Add_to_total $<amt>■ Add_to_total $<amt>■ Login <name>■ Logout <name> ■ Date <date>■ Order No <orderno>■ Total_due $<amt>

● Actual output■ Add <item>■ Add <item>■ Add_to_total $<amt>■ Add_to_total $<amt>■ Login <name>■ Logout <name>■ Date <date>■ Order No <orderno>■ Total_due $<amt>

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Pass

Everything passed – isn’t this great? Actually, no – not checking name or total

5.19

Variables – what needs to be done

● Expected output■ Add 1 <item>■ Add 1 <item>■ Add_to_total T1=$15.99■ Add_to_total T2=$249.99■ Login NAME=J Smith■ Logout J Smith=NAME? ■ Date <date>■ Order <orderno>■ Total_due $265.98 =T1+T2?

● Actual output■ Add 1 <item>■ Add 1 <item>■ Add_to_total T1=$249.99■ Add_to_total T2=$35.45■ Login NAME=M Jones■ Logout J Smith =NAME?■ Date <date>■ Order <orderno>■ Total_due $285.34 =T1+T2?

“Variable=” implemented as “store but ignore in comparison”“=Variable/expression” implemented as “check it is equal to”

P

P

P

P

P

F

P

P

F

5.20

6-Advice



Final Advice and Direction




6.1

What next?

● we have looked at a number of ideas about test automation today● what is your situation?

■ what are the most important things for you now?■ where do you want to go?■ how will you get there?

● make a start on your test automation strategy now■ adapt it to your own situation tomorrow

6.2

6-Advice



Strategy exercise

● your automation strategy / action plan■ review the exercises

►automation objectives, 3rd page► responsibility►measurement►architecture

● Your strategy■ identify the top 3 changes you want to make to your automation■ note your plans now

6.3

Dealing with high level management

● management support■ building good automation takes time and effort■ set realistic expectations

● benefits and ROI■ make benefits visible (charts on the walls)■ metrics for automation

► to justify it, compare to manual test costs over iterations►automation health: on-going continuous improvement

– build cost, maintenance cost, failure analysis cost– coverage of system tested

6.4

6-Advice



Dealing with developers

● critical aspect for successful automation■ automation is development

►may need help from developers►automation needs development standards to work

– testability is critical for automatability– why should they work to new standards if there is “nothing in it for them”?

■ seek ways to cooperate and help each other► run tests for them

– in different environments– rapid feedback from smoke tests

►help them design better tests?6.5

Standards and technical factors

● standards for the testware architecture■ where to put things■ what to name things■ how to do things

►but allow exceptions if needed● new technology can be great

■ but only if the context is appropriate for it (e.g. Model-Based Testing)● use automation “outside the box”

6.6

6-Advice



On-going automation

● you are never finished■ don’t “stand still” - schedule regular review and re-factoring of the automation■ change tools, hardware when needed■ re-structure if your current approach is causing problems

● regular “pruning” of tests■ don’t have “tenured” test suites

► check for overlap, removed features►each test should earn its place

6.7

Information and web sites

■ www.TestAutomationPatterns.org■ www.AutomatedTestingInstitute.com

►TestKIT Conference, Washington DC■ tool information

► commercial and open source: http://testertools.com►open source tools

– www.opensourcetesting.org– http://sourceforge.net– http://riceconsulting.com (search on “cheap and free tools”)

►LinkedIn group: QA Automation Architect■ www.ISTQB.org

►Expert level in Test Automation (in progress)6.8

6-Advice





Advice

1 2 3

4 5 6

Summary: successful test automation• assigned responsibility for automation tasks• realistic, measured objectives (testing ≠ automation)• technical factors – architecture, levels of abstraction, DSTL, scripting, comparison, pre and post processing• management support, ROI, continuous improvement

Free book!

6.9

and now …

● any final questions / comments?● please evaluate this tutorial (vs its objectives)

■ high mark – thanks very much!■ low mark – please explain (so we can improve)

► (see Session 0 for tutorial objectives, what is covered and what is intentionally excluded)● email Dot for ROI spreadsheet / tool list ([email protected])● email Mark for automation consultancy & questions ([email protected])

6.10

6-Advice



Any more questions?Please email me!

[email protected]

Thank you for coming todayI hope this will be useful for you

All the best with your automation!


6.11

Technology

Successful Test Automation: A Manager’s View