Embed Size (px)
Citation preview
Effective Test Automationà la Carte
Martin Gijsen
Test automation architectIndependent consultant
[email protected]© 2009
Overview
Effective test automation Domain Specific Test Languages Three example solutions (free tools only) Where to get the automator Conclusion Q & A
Test automation scenarios
2
1
time
total cost / effort
1. manual testing2. effective automated testing3. less effective automated testing
3
About 2/3 of all test automation projects fail sooner or later– Brian LeSuer (testing experience magazine,December 2008 issue, page 47)
Effective test automation
Record & Playback only works in case of (very) little manual maintenance to the testware
An automated test is a program Test automation requires software engineering For continuity of benefits, focus on:
3 R's: Tests must be easy to write, review and revise Maintainability and maintenance sensitivity
Domain Specific Test Languages
Used by testers to express automated tests Instructions defined by (or with) testers Only the essence of a test case should remain:
Natural, high abstraction level No irrelevant interfacing details No irrelevant tooling details The result is low maintenance
No programming Clear test report
A good DSTL based test caseaccount first name last name address
open account 1234567890 John Doe 11 Test street, Testburg
open account 1234567891 Jane Doe 27 Testability lane, Testville
account amount
deposit John 12.345,00
amount from account to account
transfer 1234 John Jane
account amount
check balance John 11.111,00
check balance Jane 1.234,00
Example instruction documentation
Name and aliases 'begin test case' and 'begin testcase'
Description Indicates the beginning of a (new) test case.
Parameters
1 The identification of the test case, preferably unique within the test. Optional.
2 The test case description. Optional.
Pre-condition None.
Post-conditionIf the previous input line was part of a test case, that test case is closed. A new test case is opened and assigned the next sequence number for the report(s), starting at one.
Error condition If in a procedure definition.
Example ...
Developer domain: Scripting Source code
Focus is on HOW to test itRequires software engineering skills
Tester domain: Instructions
Focus is on WHAT to testRequires analysis skills
How does it work?
Testautomation
solution
Testautomation
solution
TestTest Testreport
Testreport
SystemUnderTest
SystemUnderTest
Instructiondefinitions
Instructiondefinitions
Three examples
Real systems and test solutions:
1. Web services based billing system
2. Web application for batch payments
3. Generic solution for payment systems
Different systems, interfaces & tooling
Web services based billing system
Interface: web services (SOAP / XML) Interface type: synchronous messaging Tooling: ETA Framework, Axis, Java
Multiple implementations of instructionsfor phased test execution
A sample test caseurl
select end point http://12.34.56.78:1234/billing
name
select wholesale customer Joy inc.
cust nr first name last name ...
create end customer 12345 John Doe ...
cust nr service ... ...
add subscription 12345 X ... ...
item nr cust nr ... ...
check billing item X12345 12345 ... ...
The design
Test engineTest engine
TestTest Testreport
Testreport
Billingsystem
Billingsystem
Instruction logic
Instruction logic
Web serv. client
Web serv. clientWSDLWSDL AxisAxis
Web app for payment batches
Interface: web application (J2EE) Interface type: Graphical user interface Tooling: ETA Framework, WebDriver, Java
A sample test case
name
set screen item name order letter account accountNr
set screen item name order letter amount amount
set screen item name order letter date desiredDate
user name
log on JohnDoe
account amount date ...
enter order letter 123456789 12345,67 today ...
The design
Test engineTest engine
TestTest Testreport
Testreport
Webapplication
Webapplication
Instruction logic
Instruction logic
WebDriverWebDriver
Payment systems
Interfaces: Iso15022 / SWIFT MT messages Iso20022 / SWIFT MX / UNIFI messages
Interface type: asynchronous messaging Tooling: ETA Framework, Java
Low abstraction level is raised using procedures (not in example)
A sample (low level) test casename
define structure msgOut
name value
set msgOut.field1 value1
set msgOut.field2 value2
connection type name
send message toSut mt103 msgOut
connection name period
receive message fromSut msgIn within 10 seconds
value 1 value 2
compare ?msgIn.field1 value1
compare ?msgIn.field2 value2
The design
TestTest Testreport
Testreport
Payment systemPayment system
Test engineTest engine
Instruction logicInstruction logic
UNIFI protocolUNIFI protocol SWIFT protocolSWIFT protocol Incoming msgsIncoming msgs
Connection logicConnection logic
Test engineTest engine
WS logicWS logic Web app logicWeb app logic Payment logicPayment logic
WS clientWS client WebDriverWebDriver
Bringing it all together
System Under TestSystem Under Test
Protocol XProtocol X Protocol YProtocol Y Incoming msgsIncoming msgs
Connection logicConnection logic
The test automation menu
1. DSTL
– Define your own
2. Test engines
– ETA Framework
– Fit / FitNesse
– STAF
– xUnit
– ...
3. Interfacing components
– Web services
– Web app
– Messaging
– Java Swing
– ...
4. Vendor lock-in (optional)
Where to get the automator
Main criteria: Skills (programming ≠ software engineering) Availability Cost
Train someone from the test team Borrow from development Hire or reserve developer for test team Hire external consultant
Conclusions
Test automation is software engineering- apply the right skills to succeed
A DSTL enables testers and reduces maintenance Automated testing with free software is possible
for many systems and interfaces (even combined) While every system and DSTL is different,
the basic design of test solutions can be the same Non-programming testers and non-testing
developers can be effective at automated testing
Q & A
?
See also
“Exploring no man's land with keyword driven testing” - presentation at Free Test 2009
“Effective automated testing with a DSTL”- white paper
www.DeAnalist.nl for the ETA Framework and its documentation
