Exploring no man's landwith keyword driven testing

Martin Gijsen, M.Sc.Martin@DeAnalist.nl

Free Test ConferenceMarch 23, 2009

Presentation overview● Automated testing● When (not) to use Record & Playback● Automated testing as no man's land● Keyword driven testing● The process● Where to get the automator

Automated (functional) testing● A computer performs the test● Fast and consistent● Potential benefits:

● Time (to market) / cost / quality● More effective use of (test) resources● More challenged (= happier) testers● Accurate and up2date test status

● Continuity● Business goals are drivers

The Record & Playback approach● Record test actions, replay as needed● Checks are added to scripts● Fundamental problem is maintenance

● Software evolves until it is decommissioned● Maintainability requires structure and abstractions● Recorded test cases are long, unstructured and low

level programs● Maintenance sensitivity is high● Maintainability is low

So why is it still used?● Automated testing is often the job of a tester● Most testers are:

● willing to record, but● unwilling or unable to create good automation

solutions● Vendor sales pitch is not always clear on this

Result and conclusion

It is often initially unclear whateffective automated testing requires

2 out of every 3 TA projectsfails sooner or later

– Brian LaSuer

Use R&P only in case oflittle manual maintenance

What about the developers?● Test automation is software engineering● Developers have the skills to create a good

automation solution● Many developers are unwilling or unable to

create good test sets● … or automate test cases that testers wrote

Welcome to no man's land● Most testers cannot or will not automate well● Most developers cannot or will not test well● Automated testing is like a no man's land in

between testing and development

● Is automated testing doomed?

Exploring no man's land

Testing

Automated testing

Development

Testing

Automated testing= analysis (what to test)

+ automation (how to test)

Development

Exploring no man's land

What is in between?

Testing

?

Development

Test analysis

Test automation

No more no man's land?

Testing

Development

Test analysis

Test automation

Keywords – The DS(T)L● Domain Specific (Test) Language

● Used to express test cases● Defined by (or with) the testers● Specific to the application under test● No programming● High level

Bad example

select (“a box”);type (“some text”);click (“OK”);...

Better example

create account;1234567890;John;Doecreate account;2345678901;Jane;Doe

deposit;1234567890;12345,67

transfer;1234,56;1234567890;2345678901

check balance;1234567890;11111,11check balance;2345678901;1234,56

Good exampleAccount First name Last name

create account 1234567890 John Doe

create account 2345678901 Jane Doe

Account Amount

deposit John 12345,67

Amount From To

transfer 1234,56 John Jane

Account Amount

check balance John 11111,11

check balance Jane 1234,56

Keywords / Instructions● Often have arguments

● No (irrelevant) test execution details● No (irrelevant) interfacing details● No tooling details

● Only the essence of a test case remains● Easy to read, write and maintain test cases● Low maintenance sensitivity

Sample instruction documentationName and aliases 'begin test case' and 'begin testcase'

Description Indicates the beginning of a (new) test case.

Parameters Two parameters, both optional

1 The identification of the test case, printable characters, preferably unique within the test. Optional.

2 The test case description, printable characters. Optional.

Validity Anywhere except inside a procedure definition.

Pre-condition None.

Post-condition If the previous input line was part of a test case, that test case is closed. A new test case is opened and assigned the next sequence number for the report(s), starting at one.

Error condition If in a procedure definition.

Automating (basic)

AutomationTest Testreport

Application under test

Automating (more detailed)

EngineTest Testreport

Application under test

Instructions

Interfacing

The ETA Framework 'engine'● Can read from and report to a spreadsheet● Supports test cases, variables and procedures● Implement keywords in Java● Use any Java library (Abbot, WebDriver, MQ,

generated web service client, ...)● Documented, actively used and developed● Freeware

The process (waterfall)

Requirements

Keyword defs

Product development

Test cases

Automate Test

The process (agile)

Requirements

Keyword defs

Product development

Test cases

Automate TestRequirements

Keyword defs

Product development

Test cases

Automate Test

Requirements

Keyword defs

Product development

Test cases

Automate TestRequirements

Keyword defs

Product development

Test cases

Automate Test

Requirements

Keyword defs

Product development

Test cases

Automate TestRequirements

Keyword defs

Product development

Test cases

Automate Test

Where to get the automator● Main criteria (for continuity):

● Skills● Availability● Cost

● Educate someone from test team● Borrow from development team● Hire or reserve developer for test team● Hire external consultant

Summary● Automated testing can help achieve your goals● Avoid Record & Playback● Good keywords help:

● The tester can focus on the testing● The developer can focus on developing● Easy to write test cases and little maintenance

● A good test engine also helps● Use skilled automators● The automators must be available when needed

Questions

?

Automation design

EngineTest Testreport

Application under test

Instructions

Interfacing

Design for a telephony switch

Engine

Telephony switch

Instructions (C++)

Existing library (iTCL)

Design for a voice portalEngine

Voice portal

Instructions (C++)

Tool

Tool library

Design for web services (1)

Engine

Web services

Generic web serviceinstructions (Java)

Generatedweb service client (Java)

Design for web services (2)

Engine

Web services

Specific web serviceinstructions (Java)

Generatedweb service client (Java)

Design for a payments system

Engine

SEPA banking system

Iso20022 protocolinstructions Java)

SWIFT MT protocolinstructions (Java)

Web serviceinstructions (Java)

Web instructions(Java)

Messaging library (Java) Generated web serviceclient (Java)

Web driverlibrary (Java)