Introducing Keyword-driven Test Automation

orial

Presented by:

Hans Buwalda

Brought to you by:

340 Corporate Way, Suite Orange Park, FL 32073 888‐2

MI PM Tut4/7/2014 1:00 – 4:30 PM

“Introducing Keyword‐Driven Test Automation”

LogiGear

300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

http://starcanada.techwell.com/�

Hans Buwalda has been working with inform hnology since his high school r,

s

Hans Buwalda LogiGear

ation tec

years. In his thirty year career, Hans has gained experience as a developer, manageand principal consultant for companies and organizations worldwide. He was a pioneerof the keyword approach to testing and automation, now widely used throughout the industry. His approaches to testing, like Action Based Testing and Soap Opera Testing,have helped a variety of customers achieve scalable and maintainable solutions for large and complex testing challenges. Hans is a frequent speaker at STAR conferenceand is lead author of Integrated Test Design and Automation: Using the Testframe Method.

http://starcanada.techwell.com/sme-profiles/hans-buwalda�

2/4/2014

1

STARCANADA 2014Toronto, Canada

Tutorial MIMonday, April 7, 13.00 – 16.30

IntroducingKeyword-driven Test Automation

© 2014 LogiGear Corporation. All Rights Reserved

Hans BuwaldaLogiGear

hans @ logigear.com

Introduction

− industries

roles in testing


− roles in testing

2/4/2014

2

About LogiGear

Software testing company, around since 1994

Testing and test automation expertise, services and

www.logigear.comwww.testarchitect.com

g p ,tooling− consultancy, training− test development and automation services− "test integrated" development services

Aims to be thought leader, in particular for large and complex test projects


p p j

Products:− TestArchitect™, TestArchitect for Visual Studio™− integrating test development with test management and automation− based on modularized keyword-driven testing

About Hans

Dutch guy, living and working in California since 2001, as CTO of LogiGear

Background in math, computer science, management

Original career in management consultancy, since 1994 focusing on testing and test automation− keywords, agile testing, big testing, . . .


www.happytester.com

hans @ logigear.com

2/4/2014

3

Topics for this tutorial

Introduction to keyword driven testing− including "Action Based Testing", my own flavor of it...

Comparison to other techniques for automationp qRecommendations for a successful application of keyword driven testing− test design− automation− organization

Some ideas for specific situations:− data driven testing


data driven testing− non-ui testing− multi-media− protocols− initial data

Not everything will be equally interesting, or accessible, to everybody

Testing Under Pressure

DEADLINE

specification development test


2/4/2014

4

Testing Under Pressure

DEADLINE

specification development test

Develop tests in time:


Develop tests in time:• Test design• Auditing, acceptance• Preparations• Automation

The 5% Rules of Test Automation

No more than 5% of all test cases should beNo more than 5% of all test cases should be executed manually

No more than 5% of all efforts around testing should involve automating the tests


testing should involve automating the tests

2/4/2014

5

Why a High Automation Degree ??

The best way to prepare for efficiency in the crunch zone− good manual test cases can help too, but marginally

Buy time to do more "exploratory testing", and better test development

Credible pay-off for the cost of introduction of automation− initial costs are: tooling, learning curve, adaptation of existing tests

Automation is better positioned to identify “bonus bugs”− on average 15% of fixes cause new bugs− many of these bugs are hard to find without integral testing

• often a result of violating overall architectures• the bugs occur because data is left in inconsistent state


Automated tests have a better chance of being kept up to date if they form the majority of the testware

Automation can be re-run, for example as part of the continuous integration process− either specific based on code changes, or integral, to also catch bonus bugs

Why < 5% Automation Efforts ??

Automation should not dominate testing− it is not a goal in itself− may never be a bottleneck

automation should deliver, not dominate…may never be a bottleneck

Testers should be able to focus on testing− better tests (higher ambition level)− communication with stake holders

High automation efforts can aggravate the


High automation efforts can aggravate the “crunch zone”, instead of relieving it− “invitation to Murphy’s law”

2/4/2014

6

Record and Playback

select window "Logon"enter text "username", "administrator"enter text "password", "testonly"push button "Ok"push button "Ok"select window "Main"push button "New Customer"expect window "Customer Information"select field "First Name"type "Paul"select field "Last Name"type "Jones"


type Jonesselect field "Address"type "54321 Space Drive"

.

.

.

Scripting

Test CaseDesign

Test CaseAutomation

TEST DESIGNER

AUTOMATION ENGINEER


Test CaseExecution

AUTOMATION ENGINEER

MR. PLAYBACK

2/4/2014

7

Example scripting

/// <summary>/// AddItems ‐ Use 'AddItemsParams' to pass parameters into this method./// </summary>

State of the art, but stuff for coders . . .

/// </summary>public void AddItems(){

#region Variable DeclarationsWinControl uICalculatorDialog =

this.UICalculatorWindow.UICalculatorDialog;WinEdit uIItemEdit =

this.UICalculatorWindow.UIItemWindow.UIItemEdit;#endregion

// Type '{NumPad7}' in 'Calculator' Dialog


Keyboard.SendKeys(uICalculatorDialog,this.AddItemsParams.UICalculatorDialogSendKeys,ModifierKeys.None);

// Type '{Add}{NumPad2}{Enter}' in 'Unknown Name' text boxKeyboard.SendKeys(uIItemEdit,

this.AddItemsParams.UIItemEditSendKeys,ModifierKeys.None);

}

Keywords, Action Words

number name quantitynew product P-9009 Sledge Hammer 5

number quantityadd quantity P-9009 20

number name quantitynew product P-9009 Sledge Hammer 5

number quantityadd quantity P-9009 20

actions, each with a keyword and arguments read from top

to bottom

fragment from a test with actions

Common in automation tools nowadays (but with different styles)

Identify tasks for both test development and automation

q yadd quantity P-9009 3add quantity P-9009 6

number quantitycheck quantity P-9009 34

q yadd quantity P-9009 3add quantity P-9009 6

number quantitycheck quantity P-9009 34

"34" is the expected value here

to bottom


y p

The test developer creates tests using actions

Each action consists of a keyword and arguments

The automation task focuses on automating the actions

Each action is automated only once

2/4/2014

8

Potential benefits of keywords

More tests, better tests− more breadth− more depth

Fast, results can be quickly available− the test design directly drives the automation

Separates the tests from the technical scripting language− easier to involve business subject matter experts− the action format allows for easy readability

Less efforts for automation− even "script free" in many cases


Automation more stable and maintainable− limited and manageable impact of changes in the system under test

Develop tests more early in the life cycle− deal with execution details later

Risks of keyword approaches

Often seen as silver bullet, complications are underestimated− often treated as a technical "trick"− testers can get squeezed and marginalized

• developers and users dictating tests• automation engineers dictating actions

− testers can end up with an automation responsibility, thus becoming pseudo programmers

The method needs understanding and experience to be successful


− pitfalls are many, and can have a negative effect on the outcome

Lack of method and structure can risk manageability− maintainability not as good as hoped− results can be disappointing, approach will be blamed

2/4/2014

9

Combining Approaches . . .

Use keywords for the automation-ready description of test cases

Use scripting to set up structured automation for the actions


Use record and playback to record keywords

Enter a user id that is greater than 10 characters, enter proper information for all other fields, and click on the "Continue" button

There should be an error message stating that "User Id must be less than 10 characters".

Comparing FormatsMost values are implicit. The tester has to figure them out during execution….

classic format

Execution instructions are repeated in multiple test cases

and click on the Continue button characters .

Enter a User Id with special character's), enter proper information for all other fields and click on the "Continue" button

An error message should be displayed indicating that "User Id cannot contain some special characters".

Enter the information, with a password of 4 characters and click on the "Continue" button

Check for an error message saying: "Password must contain at least 5 characters".

keywords


user id messagecheck registration dialog aaaaabbbbbc User Id must be less than 10 characters

user id messagecheck registration dialog résoudre User Id cannot contain some special characters

password messagecheck registration dialog test Password must contain at least 5 characters

2/4/2014

10

Keywords is not just test automation

Can also be used for other than testing:− data entry chores− training purposesg p p

Can also be used for manual testing− for example with a manual testing dialog− even can show instructions, with placeholders for values

Actionlogin <user> <password>Enter "<user>" in the user name field, and "<password>" in the password field.

Actionlogin <user> <password>Enter "<user>" in the user name field, and "<password>" in the password field.


Test lineuser name password

login hansb secret

Test lineuser name password

login hansb secret

What the manual tester would seeEnter "hansb" in the user name field, and "secret" in the password field.What the manual tester would seeEnter "hansb" in the user name field, and "secret" in the password field.

Keywords need a method

By themselves keywords don't provide much scalability− they can even backfire and make automation more cumbersome− a method can help tell you which keywords to use when, and how to

i thorganize the process

Today we'll look at Action Based Testing (ABT)− addresses test management, test development and automation− large focus on test design as the main driver for automation success

Central deliveries in ABT are the "Test Modules"


Central deliveries in ABT are the Test Modules− developed in spreadsheets− each test module contains "test objectives" and "test cases"− each test module is a separate (mini) project, each test module can

involve different stake holders

2/4/2014

11

Don't just automate manual testing




2/4/2014

12



Good automated testing is not the same as automating good manual testing. . .

Action Based Testing (ABT)

Uses the keyword format as a basis for a method− covers test management, test development and automation− with a large focus on test design as the main driver for automation

success− method is specific, but concepts are generic

The central product in ABT is the "Test Module", not the test case− like chapters in a book− test cases are part of the test modules, they are typically the result

(rather than the input) of test development


(rather than the input) of test development− test development is seen as having both analytical and creative

aspects− developed as spread sheets, external from the automation, with a

well defined flow− easier to manage: each test module is a separate (mini) project, each

test module can involve different stake holders

2/4/2014

13

Test Module Plan

Test Module 1 Test Module 2 Test Module N

Overview

Tests

Objectives

Tests Tests

Actions

. . .Objectives Objectives


AUTOMATION

user passwordlog in jdoe StarEast

first last brand modelrent car John Renter Ford Escaperent car John Renter Chevrolet Volt

last totalcheck bill Renter 140.42

interaction test business testwindow control value

enter log in user name jdoeenter log in password StarEast

window control property expectedcheck prop log in ok button enabled true

car guy

Example of business level test module

Consists of an (1) initial part, (2) test cases and (3) a final partFocus is on business functionality, with a clear business scopeNavigation details are avoidedNavigation details are avoided

TEST MODULE Car Rental Paymentsuser

start system john

TEST CASE TC 01 Rent some cars

first name last name carrent car John Doe Ford Escape


rent car John Doe Ford Escaperent car John Doe Chevvy Volt

last name amountcheck payment Doe 140.4

FINAL

close application

2/4/2014

14

Example of an interaction level test module

Lay-out the same, with an initial part, test cases and a final partInteraction details that are the target of the test are not hiddenFocus is not on business ("is the payment amount correct"), but on ( p y )interaction ("do I see the payment amount")

TEST MODULE Screen Flowuser

start system john

TEST CASE TC 01 Order button

window buttonclick main create order


windowcheck window exists new order

FINAL

close application

Variables and expressions with keywords

TEST CASE TC 02 Rent some more cars

car availableget quantity Chevvy Volt >> voltsget quantity Chevvy Volt volts

first name last name carrent car John Doe Chevvy Voltrent car John Doe Chevvy Volt

car expectedcheck quantity Chevvy Volt # volts - 2


This test does not need an absolute number for the available cars, just wants to see if a stock is updatedAs a convention we denote an assignment with ">>"The "#" indicates an expression

2/4/2014

15

Data driven testing with keywords

DATA SET carscar first last valueChevvy Volt John Doe 40000

TEST CASE TC 03 Check stocksdata set

use data set /cars yFord Escape Mary Kane 22500Chrysler 300 Jane Collins 29000Buick Verano Tom Anderson 23000BMW 750 Henry Smyth 87000Toyota Corolla Vivian Major 16000

car availableget quantity # car >> quantity

first name last name carrent car # first # last # car

car expectedcheck quantity # car # quantity - 1

repeat for data set


The test lines will be repeated for each row in the data setThe values represented by "car", "first" and "last" come from the selected row of the data set

Automating keyword tests

Keywords are useful, but technical not complex. It is not hard to make a simple keyword interpreter. Many test tools also have keyword options in some form or another.

Function InterpretWhile not end of test

Read next lineSplit the line into argumentsLook up the keyword in the "action list"Execute the function belonging to the keyword

ReportInterpreter


g g yReport the results of this line

Repeat for next lineEnd

2/4/2014

16

Example: script for an action "check sort order"

The following action script will verify whether the rows in a table are sorted:

def action checkSortOrder():

# get table object, column number and column countwindowName = LIBRARY.NamedArgument("window")tableName = LIBRARY.NamedArgument("table")columnName = LIBRARY.NamedArgument("column")table = ABT.OpenElement(windowName, tableName)column = table.GetColumnIndex(columnName)rowCount = table.GetRowCount()

# check the sort order, row by rowprevious = table.GetCellText(0, column)f i i (1 C t)

find the table in the UI

if a value is smaller than before, fail the test

get arguments from the test line

_ ()


for i in range(1, rowCount):current = table.GetCellText(i, column) if current < previous :

LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0)return

previous = current

LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)

if all rows are ascending, pass the test

Using the new action

By keeping an action generic it can be applied for a variety of situations

S l f i " h k t d "Some examples of using "check sort order":

window table columncheck sort order view orders orders table ID

window table columncheck sort order annual results regions revenue


window table columncheck sort order inventory cars price

window table columncheck sort order registration students last name

2/4/2014

17

Example application


A Test Module for the application


We click a tree node, and then do a checkThe actions here are built-in in the framework

2/4/2014

18

Making a new "action"

name of the new action

create a node path

the arguments of the new action


This action definition uses existing actions to create a new action called "check bitrate"Argument names can be used in cell expressions, that start with "#", and support the usual string and numeric operators

pfrom the first two argumentsthe expected value is

given by the 3rd argument

Using the action in a test


These test lines don't care about the navigation in the UI of the application, the focus is functional: verify dataSuch functional tests are easier to read with high level actions, and the reduced dependency on navigation makes them (much) easier to maintain in the long term

2/4/2014

19

4000

In a good application of the keywords approach a large increase in test cases (like doubling the amount) should result in a modest increase in actions, and a minor increase, if any, in programmed action functions.

Scalability

4000tests

2000


250actions

tests

22 functions

200actions

20 functions

Identifying controls


Identify windows and controls, and assign names to themThese names encapsulate the properties that the tool can use to identify the windows and controls when executing the tests

2/4/2014

20

Mapping an interface

INTERFACE ENTITY libraryinterface entity setting title {.*Music Library}

name class labelinterface element title text Title:interface element title text Title:interface element artist text Artist:interface element file size text File size (Kb):

name class positioninterface element playing time text textbox 4interface element file type text textbox 5interface element bitrate text textbox 6

name class positioninterface element music treeview treeview 1


An interface mapping (common in test tools) will map windows and controls to namesWhen the interface of an application changes, you only have to update this in one placeThe interface mapping is a key step in your automation success, allocate time to design it well, in particular naming and choosing identifying properties

Some Tips to Get Stable Automation

Press for an automation-friendly system design− consider this a key requirement ("must have")− having testing and automation in agile teams helpshaving testing and automation in agile teams helps

Test the automation, in particular complex actions− before running tests with them

Keep an eye on the test design


Use "active" timing, don't use hard coded waits

2/4/2014

21

Automation friendly system design (examples)

Timing hooks− to know when a UI or a background process is ready

Hidden identifiers for controlsHidden identifiers for controls− like "name" in Java and "automation id" in WPF

Access to data input− ability to set up special values for calculations− functions to shut off or steer random generators

Behind-the-scenes access to displayed data


− graphic representation is different from value correctness− in a game: knowing where the monsters are

Emulation features, like time-travel and fake locations

(note: be aware of security considerations. Non-UI access should be safe)

Look for properties a human user can't see, but a test tool canThis approach can lead to speedier and more stable automation− interface mapping is often bottleneck, and source of maintenance problems− with predefined identifying property values in interface map can be created without "spy" tools

not sensitive to changes in the system under test

Automation-friendly design: hidden properties

− not sensitive to changes in the system under test− not sensitive to languages and localizations

Examples:− "id" attribute for HTML elements− "name" field for Java controls− "AccessibleName" or "Automation ID" properties in .Net controls (see below)


number

2/4/2014

22

Mapping the interface using hidden identifiers

INTERFACE ENTITY libraryinterface entity setting automation id MusicLibraryWindow

ta name ta class automation idinterface element title text TitleTextBoxinterface element title text TitleTextBoxinterface element artist text SongArtistTextBoxinterface element file size text SizeTextBoxinterface element playing time text TimeTextBoxinterface element file type text TypeTextBoxinterface element bitrate text BitrateTextBox

ta name ta class automation idinterface element music treeview MusicTreeView


Instead of positions or language dependent labels, an internal property "automation id" has been usedThe interface definition will be less dependent on modifications in the UI of the application under testIf the information can be agreed upon with the developers, for example in an agile team, it can be entered (or pasted) manually and early on

Active Timing

Passive timing− wait a set amount of time− in large scale testing, try to avoid passive timing altogether:

• if wait too short, test will be interruptedif it t l ti i t d• if wait too long, time is wasted

Active timing− wait for a measurable event− usually the wait is up to a, generous, maximum time− common example: wait for a window or control to appear (usually the test tool will do

this for you)

Even if not obvious, find something to wait for...


Involve developers if needed− relatively easy in an agile team, but also in traditional projects, give this priority

If using a waiting loop− make sure to use a "sleep" function in each cycle that frees up the processor (giving the

AUT time to respond)− wait for an end time, rather then a set amount of cycles

2/4/2014

23

Active Timing

How much passive timing do you have in your scripts?If you're not sure, find out...... and let me know

"First action I took upon my return was to evaluate the percentage of


First action I took upon my return was to evaluate the percentage of passive time in our code and found passive time 68% versus active time 32%. Needless to say our automation test cases were very expensive time operations and now I know why..."

Raed Atawneh, 2012 (extract)

Things to wait for...

Wait for a last control or elements to load− developers can help knowing which one that is

Non UI criteriaNon-UI criteria− API function− existence of a file

Criteria added in development specifically for this purpose, like:− "disabling" big slow controls (like lists or trees) until they're done loading− API functions or UI window or control properties

Use a "delta" approach:


Use a delta approach:− every wait cycle, test if there was a change; if no change, assume that the

loading time is over:− examples of changes:

• the controls on a window• count of items in a list• size a file (like a log file)

2/4/2014

24

Test Design

Effective test breakdown (into test modules)− make sure every test module has a clear focus

keep different kinds and levels of tests separate− keep different kinds and levels of tests separate

Right level of actions− as “high level” if possible, hiding as many details as

much as possible− but not if the details are relevant for the test


I have become to believe that successful automation is not a technical challenge. It is most of all a test design challenge.

Why Better Test Design?

Quality and manageability of test − many tests are often quite "mechanical" now− one to one related to specifications, user stories or requirements,

hi h ft i k b t l k iwhich often is ok, but lacks aggression− no combinations, no unexpected situations, lame and boring− such tests have a hard time finding (interesting) bugs

Better automation− when unneeded details are left out of tests, they don't have to be

maintained− avoiding "over checking": creating checks that are not in the scope of

a test but may fail after system changes


a test, but may fail after system changes− limit the impact of system changes on tests, making such impact

more manageable

I have become to believe that successful automation is usually less of a technical challenge as it is a test design challenge.

2/4/2014

25

The Three “Holy Grails” of Test Design

Metaphor to depict three main steps in test design

Using "grail" to illustrate that there is no one perfect solution, but that it matters to pay attention (to search)

About quality of tests, but most of all about scalability and maintainability in big and complex projects

Organization of tests into test modules


Right approach for each test module

Proper level of detail in the test specification

What's the trick...


2/4/2014

26

What's the trick...

Have or acquire facilities to store and organize your content

Select your stuff

Decide where to put what− assign and label the shelves

P t it th


Put it there

If the organization is not sufficient anymore, add to it or change it

Breakdown Criteria

Main Criteria− Business tests versus interaction tests− Data versus presentation ("is data correct" versus "can I view data")

Architecture of the system under test (client server protocol sub− Architecture of the system under test (client, server, protocol, sub systems, components, modules, ...)

− Functionality (customers, finances, management information, ...)− Kind of test (navigation flow, negative tests, response time, ...)− Ambition level (smoke test, regression, aggressive, …)

Additional Criteria


− Stakeholders (like "Accounting", "Compliance", "HR", ...)− Complexity of the test (put complex tests in separate modules)− Technical aspects of execution (special hardware, multi-station, ...) − Overall project planning (availability of information, timelines, sprints, ...)− Risks involved (extra test modules for high risk areas)

2/4/2014

27

Approach 1: Workshop

Gather a meeting with relevant participants− test developers− domain expertsdomain experts− automation engineer (focus on efficiency of automation)− experienced moderator− also consider: developers, managers

If necessary, provide training of participants before the discussion


before the discussion

Approach 2: Design and Feedback

One or two experienced test designers create a first draft

The draft is delivered/discussed to relevant parties

Ask the parties to verify:1. Structure: does it make sense


2. Completeness: are all relevant areas covered

Based on feedback, further modify the design

2/4/2014

28

Properties of a good Breakdown

Reflects the level of tests

Well differentiated and clear in scope

Balanced in size and amount

Modules mutually independent


Fitting the priorities and planning of the project

"Thou Shall Not Debug Tests..."

Large and complex test projects can be hard to "get to run"

If they are however, start with taking a good look again at your test design...

Rule of thumb: don't debug tests. If tests don't run smoothly, make sure:− lower level tests have been successfully executed first -> UI flow in


o e e e tests a e bee success u y e ecuted st U othe AUT is stable

− actions and interface definitions have been tested sufficiently with their own test modules -> automation can be trusted

− are you test modules not too long and complex?

2/4/2014

29

Breakdown examplesCRUD tests (Create, Read, Update, Delete) for all entity types in the app− like "order", "customer", "well", etc− for all: various types and situations

Forms, value entry− does each form work (try to test form by form, not entity by entity)− mandatory and optional fields valid and invalid values etcmandatory and optional fields, valid and invalid values, etc− UI elements and their properties and contents− function keys, tab keys, special keys, etc

Screen and transaction flows− like cancel an order, menu navigation, use a browser back and forward buttons, etc− is the data in the database correct after each flow

Business transactions, business rules− identify situations that the tests need to try

Function tests, do individual functions work− can I count orders, can I calculate a discount, etc

End-to-end tests


d to e d tests− like enter sale order, then check inventory and accounting

Tests with specific automation needs− like multi station tests

Tests of non-UI functionsHigh ambition tests (aggressive tests)− can I break the system under test

Identifying the modules

Step 1: top down: establish main structure:analyze what the business is and what the system does?how is it technically organized?do other “primary criteria” apply?use the list in the "breakdown examples" slide as a starting point− see it as "low hanging fruit": items that tend to apply well in many projects

also visit the “secondary criteria”− not always applicable, but can help to refine the design further

Step 2: bottom up: refine, complete: study individual functionalities and checks (like from exist test cases)


and identify test modules for them if neededidentify and discuss any additional criteria and needed testing situationsreview and discuss the resulting list(s) of test modulesif necessary try to create some early drafts of test modules and adjust the list

Repeat steps 1 and 2 if needed.

2/4/2014

30

Example of an application under test

Various item types− tests− actions− interface definitions

d t t− data sets− folders− ...

Various operations− open− cut, copy, paste− check out− ...

Various ways to initiate an operation− context menu, with or without accelerator key


, y− main menu, with or without accelerator key− toolbar− short cut key− function key− drag and drop− double click− ....

Defining some modules

Test modules for operations− primary and alternate paths− various values for fields like "comment" in check-in− paste in other projects

d t i− copy and paste various groups− not necessarily on each item type

Test modules for items− address all item types at least once− on each item type perform each operation− not necessarily each variant of each operation

UI handling− try for UI command if it starts the intended operation


y p− not necessarily on each item type or operation variant

Concurrency− try concurrency sensitive operations with multiple stations− in varying concurrency scenarios, with and without local "refreshes"− not necessarily each item type or operation variant− certainly not each UI command included

2/4/2014

31

What about existing tests?

Compare to moving house:− some effort can't be avoided− be selective, edit your stuff, look at the future, not the past

d id h t t h t th t it th− decide where to put what, then put it there− moving is an opportunity, you may not get such chance again soon

Adopt a module approach− define the modules and their scope as if from scratch− use the existing test cases in two ways:

• verify completeness• harvest and re-use them for tests and for actions

avoid porting over "step by step" in particular avoid over checking


− avoid porting over step by step , in particular avoid over-checking

Defining test runs using "test suites"

Build Acceptance Test

Smoke Test

System Test

© 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2011 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.

Functional Acceptance Test

Delivery Test

2/4/2014

32

Test Module Scope Prio Status

Model Life CyclesCreate, store, delete Models (=formula + data), as part of SYS sessions

1 pass

Result Life Cycles Create, store outputs. See them in the process store. 1 passFormula Life Cycles Create, edit, manage, remove formulas 2 passFormula Editor buttons, operations, undo 3 pass

Repositorydisplay of the Modeler repository, presence of user formulas, drag and drop usage. Effect of changing repository folder (environment variable)

1 failed

Model Store in Repository presence, re‐run, delete 1 passRepository UI example: selecting an item shows its description 2 errorsRepository UI example: selecting an item shows its description 2 errors

Formula EvaluationCorrectness of results, valid/invalid arguments, boundary analyses, special arguments

1 pass

Built‐in FormulasPresence, correctness, valid/invalid arguments, boundaries, special arguments, equivalence classes

1 pass

Data Table AssociationAssociate tabels view, change and remove associations, data applicability, for existing and defined formulas

2 pass

Quick Access buttonsLife cycle of Quick Access buttons, correctnes for the built‐in ones

3 dev

Formula argumentspresence, argument types, argument entry, parameters, defaults

2 pass

arguments for Built‐in l

arguments, argument types and defaults for each pre‐defined f l

2 failed


Formulas formula2 failed

Area Of Interest Relations defaulting, tree visibility, select/deselect, …. 1 pass

Model ExecutionModel times, start, stop (cancel), restart ("chunks", "timeboxes", ... needs more information)

3 pass

Graphics graphical representation of various data types and data sets 1 passGraphics Viewing zoom, select, drag and drop (no 3d now) 1 passAdministration users, projects, authorization 1 pass

Model results in central database

storing, removing, using, correctness, … (there are some other applications, mostly legacy, that can do the same Models to compare)

1 pass

Modeler UI various controls, panels, tabs 2 pass



1 pass



1 failed



1 pass


1 pass


2 pass


3 dev


2 pass



2 failed





3 pass




1 pass


2/4/2014

33



1 pass



1 failed



1 pass


1 pass


2 pass


3 dev


2 pass



2 failed





3 pass




1 pass




1 pass



1 failed



1 pass


1 pass


2 pass


3 dev


2 pass



2 failed





3 pass




1 pass


2/4/2014

34



1 pass



1 failed



1 pass


1 pass


2 pass


3 dev


2 pass



2 failed





3 pass




1 pass


Test Module Scope Prio Build 1 Build 2 Build 3


1 pass pass pass

Result Life Cycles Create, store outputs. See them in the process store. 1 pass pass passFormula Life Cycles Create, edit, manage, remove formulas 2 pass pass passFormula Editor buttons, operations, undo 3 pass pass pass


1 failed failed failed

Model Store in Repository presence, re‐run, delete 1 pass pass passRepository UI example: selecting an item shows its description 2 errors errors errorsRepository UI example: selecting an item shows its description 2 errors errors errors


1 pass pass pass


1 pass pass pass


2 pass pass pass


3 dev dev dev


2 pass pass pass



2 failed failed failed


Formulas formula2 failed failed failed

Area Of Interest Relations defaulting, tree visibility, select/deselect, …. 1 pass pass pass


3 pass pass pass

Graphics graphical representation of various data types and data sets 1 pass pass passGraphics Viewing zoom, select, drag and drop (no 3d now) 1 pass pass passAdministration users, projects, authorization 1 pass pass pass



1 pass pass pass

Modeler UI various controls, panels, tabs 2 pass pass pass

2/4/2014

35

Grail 2: Approach per Test Module

• Name the test modulesrecommendation: subject area only, leave words like "Verify", "Test", "Can" from titles of test modules (and test objectives and test cases)

• Plan the test module:• Plan the test module:when to develop: is enough specification availablewhen to execute: make sure the functionality at action level is well-tested and working already

• Process:analysis of requirementsformulation of "test objectives"create "test cases"

© 2009 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.© 2014 LogiGear Corporation. All rights reserved.

• Identify stakeholders and their involvement:users, subject matter expertsdevelopersauditors

• Choose testing techniques if applicable:boundary analysis, decision tables, transition diagrams, soap opera testing, ...

Eye on the ball, Scope

Always know the scope of the test module

Th h ld b biThe scope should be unambiguous

The scope determines many things:− what the test objectives are− which test cases to expect− what level of actions to use


− what the checks are about and which events should generate a warning or error (if a “lower” functionality is wrong)

2/4/2014

36

State your Objectives . . .

...TO-3.51 The exit date must be after the entry date...

test objective TO-3.51

name entry date exit dateenter employment Bill Goodfellow 2002-10-02 2002-10-01


enter employment Bill Goodfellow 2002-10-02 2002-10-01check error message The exit date must be after the entry date.

Grail 3: Specification Level, choosing actions

Scope of the test determines the specification level

As high level as appropriate, as little arguments as iblpossible

− Use default values for non-relevant arguments

Clear names (usually verb + noun usually works well)− to standardize action names: standardize both the verbs and the nouns, so

"check customer" versus "verify client" (or vice versa)− tests are not C++ code: avoid "technical habits", like mixed case and (worse)

underlines


Manage the Actions

Document the Actions

By-product of the test design

2/4/2014

37

Example of using actionsIn this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function• the "#" means an expression, in this case a variable• the ">>" assign to a variable for use later on in the test

key

key navigate F7key navigate 3y g

page tab

locate page tab Scan Criteriaw indow

wait for controls loaded searchtext

check breadcrumb general functions > searchw indow control value

select search scan direction Backward


w indow control value

enter value search business date match # bus datesource control

click search gow indow

wait for controls loaded search resultsw indow control variable

get search results sequence number >> seq num

Example of using actionsIn this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function• the "#" means an expression, in this case a variable• the ">>" assign to a variable for use later on in the test

variable

get sequence number >> seq num


2/4/2014

38

An example test for the Modeler

model name arguments formulamodel name arguments formula

create model vegas winner x 10*x

argument value

set argument x some money

model name expected

run model vegas winner a lot more money


run model vegas winner a lot more money

Too detailed?

Step Name Description Expected

step 16 Click the new formula button to start a new calculation.

The current formula is cleared. If it had not been save a message will show

step 17 Enter "vegas winner" in the name field The title will show "vegas winner"

step 18 Open the formula editor by clicking the '+' button for the panel "formula editor"

The formula editor will show with an empty formula (only comment lines)

step 19 Add some lines and enter "10*x;" The status bard will show "valid formula". There is a "*" marker in the title

step 20 Click the Save formula button The formula is saved, the "*" will disappear from the title

step 21 Open the panel with the arguments by clicking the '+' button

There two lines, for 'x' and 'y'


gstep 22 Click on the value type cell and select

"currency"A button to select a currency appears, with default USD

step 23 Click on the specify argument values link The argument specification dialog is shown

2/4/2014

39

Another example

TEST MODULE Order processing

start system

TEST CASE TC 01 Enter an order for two articles

user passwordlogin jdoe testdesign

windowcheck window exists welcome

order id cust id prioritycreate order AB123 W3454X normal


order id article amount priceadd to order AB123 tablet 3 219.95add to order AB123 case 3 3.19

id totalcheck order total AB123 669.42 . . .

Environments, configurations

Many factors can influence details of automation− language, localization− hardware

i f th t d t t− version of the system under test− system components, like OS or browser

Test design can reflect these− certain test modules are more general− others are specific, for example for a language


But for tests that do not care about the differences, the automation just needs to "deal" with them− shield them from the tests

2/4/2014

40

Capture variations of the system under test in the actions and interface definitions, rather than in the tests (unless relevant there).Can be a feature in a test playback tool, or something you do with a global variable or setting.

"Variations"

"Master Switch"

Actions, Interface Definitions

. . .


Variation Variation Variation

Possible set up of variations

linked variation

keyworded variation


Specify for example in a dialog when you start an execution:

2/4/2014

41

Non-UI Testing

Examples− application programming interfaces (API’s)− embedded software

protocols− protocols− files, batches− databases− command line interfaces (CLI’s)− multi-media− mobile devices

Impact is mainly on the automationtest design should in most cases be transparent towards the specific

testing devices


− test design should in most cases be transparent towards the specific interfaces

Often non-UI automation can speed up functional tests that do not address the UI

Multiple System Access

Test Modules, driving either one or multiple interfaces

Automation Scheme

protocolaccess

UIaccess

one or multiple interfaces


System (part)Under TestAPI

access

access access

databaseaccess

2/4/2014

42

Device Testing

Testing HostDevice

Software Under Test

AgentABTAutomation

InterfaceI f

Device

Android


Info

Multimedia: The "Play List" Approach

Approach applicable for graphics, videos, sound fragments, etc

The test includes "questions":− what the tester should see or hear− like "Are the matching areas blue?"− actions like "check picture"

The test tool keeps a "play list"− during the run items are captured and stored− after the run, the tester is presented with the items,

d h hi i


and the matching questions− the tester acknowledges/falsifies− the system stores those passed items− if during the next run the items are the same as

earlier passed ones, the tester is not asked again

2/4/2014

43

Performance Testing

The topic is complex, but to create tests can be quite straightforward− actions like "generate load <how much>" and "check response time <max

wait>"− use one tool to generate load (like JMeter), another to run the "normal"

functional test

Often performance testing isn't testing, but more close to research− analysis bottle necks and hot spots (for example discontinuities in response

times, means buffers are full)− application of statistical techniques like queuing theory


pp q q g y− how to realistically mimic large scale productions situations in smaller test

environments

The three controls you can/should address:hardware (equipment, infrastructure, data centers, etc)software (programs, database models, settings, etc)demands (1 second may cost 10 times more than 2 seconds)

See also: "Load Testing for Dummies", Scott Barber, gomez.com

Organization

Much of the success is gained or lost in how you organize the process− part of the teams− who does test designwho does test design− who does automation− what to outsource, what to keep in-house

Write a plan of approach for the test development and automation− scope, assumptions, risks, planning− methods, best practices− tools, technologies, architecture− stake holders, including roles and processes for input and approvals− team


team− . . .

Assemble the right resources− testers, lead testers− automation engineer(s)− managers, ambassadors, ...

Test design is a skill . . .Automation is a skill . . . Management is a skill . . .

. . . and those skills are different . . .

2/4/2014

44

Life Cycles

Product life cycles, rather than task life cyclesThe project planning and execution largely determines when the products are created

systemdevelopment

testd l t


development

testautomation

Typical Time Allocation

TEST DEVELOPMENT

effo

rts


AUTOMATION

time

2/4/2014

45

Keywords and ABT in Agile

Keywords are suitable for agile projects:− tests are easier to create and understand, in particular for non-

programmersth ll t t d l t ith t d f d t il th t h 't− they allow test development without a need for details that haven't been defined yet

− automated tests can quickly follow changes in the system under test

Action Based Testing in itself is quite agile− focused on products and cooperation− flexible in process, in fact each test module can have its own process− test modules are usually very suitable to drive system development

H ABT li hi h l l t t d i f b t


However, ABT relies on high level test design for best results− identifying test modules− in larger scale projects this may require at least some overall test

planning activities that are not necessarily easy to do in a single scrum team

Test Development and Automation in sprints

product owner team prod owner

& team

Agile life cycle

Test ModuleDefinition(optional)

Test Module Development

Interface Definition

Action Automation

Sprint ProductsProduct Backlog

Test re-use

Automation re-use


Test Execution

User storiesDocumentation

Domain understanding

Acceptance CriteriaPO Questions

SituationsRelations

Test development

Main Level Test Modules

Interaction Test Modules

Cross over Test Modules

2/4/2014

46

Automated testing in sprints

Try keep the main test modules at a similar level as the user stories and acceptance criteria− test modules can double as modeling device for the sprint

Aim for "sprint + zero", meaning: try to get test development and automation "done" in the same sprint, not the next one− next one means work clutters up, part of team is not working on the

same sprint, work is done double (manually and automated), ...Make sure you can do the interface mapping by hand (using developer provided identifications)


(using developer provided identifications)− can do earlier, before UI is finalized, and − recording of actions will go better

Make sure to have sufficient technical access to the AUT− for timing− for data, conditions etc

Fitting in sprints

Agreement on the approach:− questions like does "done" include tests developed and automated?− do we see testing and automation as distinguishable tasks and

kill tskillsets− is testability a requirement for the software

Understand subject matter and testing approaches

Automation technology should be readily available

Have a strategy for dealing with base and input data

M k t ti d t ti t f th l ti


Make testing and automation part of the evaluations

Address tests and automation also in hardening sprints

Just like for development, use discussions with the team and product owners to deepen understanding:− also to help identify negative, alternate and unexpected situations

2/4/2014

47

Summary

Keywords is one of the techniques for automated testing, in addition to record & playback and scriptingscripting

In itself keywords are not a silver bullet, it needs a good approach, careful planning and good organization to be successful


Keywords can work for GUI testing, but equally well for a variety of other purposes

1. Testing Computer Software, Cem Kaner, Hung Nguyen, Jack Falk, Wiley

2. Lessons Learned in Software Testing, Cem Kaner, James Bach, Bret Pettichord, Wiley

3. Experiences of Test Automation, Dorothy Graham, Mark Fewster, Addison Wesley, 2012

4. Automating Software Testing, Dorothy Graham, Mark Fewster, Addison Wesley

Some References

5. Action Based Testing (overview article), Hans Buwalda, Better Software, March 2011

6. Action Figures (on model-based testing), Hans Buwalda, Better Software, March 2003

7. Integrated Test Design & Automation, Hans Buwalda, Dennis Janssen and Iris Pinkster, Addison Wesley

8. Soap Opera Testing (article), Hans Buwalda, Better Software Magazine, February 2005

9. Testing with Action Words, Abandoning Record and Playback, Hans Buwalda, Eurostar 1996

10. QA All Stars, Building Your Dream Team, Hans Buwalda, Better Software, September


10. QA All Stars, Building Your Dream Team, Hans Buwalda, Better Software, September 2006

11. The 5% Solutions, Hans Buwalda, Software Test & Performance Magazine, September 2006

12. Happy About Global Software Test Automation, Hung Nguyen, Michael Hackett, e.a., Happy About

13. Misconceptions About Test Automation, Hans Buwalda, LogiGear Magazine, April 29th, 2013

Technology

Introducing Keyword-driven Test Automation