Upload
techwellpresentations
View
157
Download
4
Embed Size (px)
DESCRIPTION
When implementing test automation, many people encounter problems: where to start with automation, high maintenance costs for the automated tests, or unrealistic management expectations. The good news is that solutions to these problems exist and have been effectively used by many. A “pattern” is a general reusable solution to a commonly occurring problem. Patterns have been popular in software development for many years, but they are not commonly recognized in system-level test automation. Dorothy Graham shares a collection of common problems (issues) and their solutions (patterns) which she and others are now developing as a wiki. To help resolve typical issues, Dot gives you a brief guided tour of some patterns—from Maintainable Testware and Domain-Driven Testing to Fail Gracefully and Kill the Zombies. Dot helps you recognize test automation issues and shows you how to identify appropriate patterns to help solve them.
Citation preview
nt Session
Presented by:
Dorothy Indepen ultant
Brought to you by:
340 Corporate Way, Suite Orange Park, FL 32073 888‐2
W3 Concurre4/9/2014 10:30 AM
“Test Automation Patterns”
Graham
dent Testing Cons
300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Dorothy Graham Independent Test Consultant
In software testing for forty years, Dorothy Graham is coauthor of four books—Software Inspection, Software Test Automation, Foundations of Software Testing and Experiences of Test Automation—and is currently working with Seretta Gamba on a new book on test automation patterns. A popular and entertaining speaker at conferences and seminars worldwide, Dot has attended STAR conferences since the first one in 1992. She was a founding member of the ISEB Software Testing Board and a member of the working party that developed the ISTQB Foundation Syllabus. Dot was awarded the European Excellence Award in Software Testing in 1999 and the first ISTQB Excellence Award in 2012. Learn more about Dot at DorothyGraham.co.uk.
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
1
Test Automation Patterns
Prepared and presented by
© Seretta Gamba, Dorothy Graham 2014
Dorothy Graham
@DorothyGraham [email protected] www.DorothyGraham.co.uk
2
Contents Background
Issues and examples Test automation Patterns Wiki
Conclusion
Test Automation Patterns
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
3
Origin of patterns in software – 1970s: architect Christopher Alexander describes
patterns for buildings (e.g. light from 2 walls) – 1987: Kent Beck & Ward Cunningham
experimented & presented at OOPSLA – 1994: “Gang of Four” wrote a book applying
pattern principles to software design • “Design Patterns: elements of reusable object-oriented
software”, Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides
– now widely accepted in software development community
4
What is a pattern? – a generally reusable solution to a commonly
occurring problem within a given context – a description or template for how to solve a
problem that can be used in many different situations
– a pattern is NOT • a finished solution that can be ‘plugged in’ or
transformed directly into code • prescriptive • a set of steps
– a pattern needs to be implemented within context
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
5
Where did these automation patterns come from?
– Experiences book published 2012 – Seretta Gamba contributed Ch 21
• Automation through the back door, by supporting manual testing
– Seretta is a developer, used to using patterns
– when she read the other chapters, she thought: “Patterns! Patterns!”
– she wrote up test automation patterns as a book & asked me to join her
– Word didn’t work very well, so we made a wiki
6
Introduction to the wiki
• the Test Automation Patterns wiki – ISSUES – describe problems in automation – PATTERNS – proven ways to overcome problems – other things: acknowledgements, references, some
failure patterns • in this presentation
– I am using an off-line version of the wiki for demonstration purposes
– later I will tell you how you can gain access to the “real” wiki
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
7
Test automation patterns wiki
• purpose: share information, ideas and experiences about test automation, using issues and patterns
• mind-maps summarise issues and patterns (not clickable as links)
• issues and patterns are classified into sub-sections (e.g. management, process, etc.)
• “diagnostics” help you find your way into the most relevant issues for you (new)
8
Contents Background
Issues and examples Test automation Patterns Wiki
Conclusion
Test Automation Patterns
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
9
Issues (problems)
• what problems affect you in test automation? – what causes the most trouble? – what takes longer than it should? – what is holding you back in your automation? – what are you struggling with? – what would you most like to improve?
10
Test automation Issues: 4 examples • HIGH MAINTENANCE COST
– Maintenance of the test automation scripts takes a long time and costs more than it’s worth.
• NO PREVIOUS TEST AUTOMATION – Your first time automating, or automation has failed and
you are starting again. • STALLED AUTOMATION
– Automation has been tried, but it never “got off the ground” or has now fallen into disuse.
• UNREALISTIC EXPECTATIONS – Management has unrealistic expectations about what
test automation can & can’t do.
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
11
Issue: HIGH MAINTENANCE COST
• Maintenance of the test automation scripts takes a long time and costs more than it’s worth
• Questions to ask: – what cost (or time) would be acceptable? – how is the cost measured? – Note: answers may be different in different
contexts
12
Issue: HIGH MAINTENANCE COST 1
• What are your symptoms? – When software changes, e.g. window sequence is
different, tests fail. More effort to edit the existing tests than to start again. (Capture/replay used)
– Diagnostic question: Are the scripts structured and reusable? • 1) No, scripts are mainly “stand-alone” without much
reuse, automators are “re-inventing” • 2) Yes, structured and reusable scripts, but it needs
automators to add new tests, and we are short of time and/or automators
visit offline wiki
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
13
Issue: HIGH MAINTENANCE COST 2
• What are your symptoms?
– No time allowed to maintain the automated tests
– Tests are written in the tool’s language, but the tool has changed, or we have to change tools
14
HIGH MAINTENANCE COST (Tour) - 1
• Symptom: SUT changes -> large cost to update – MAINTAINABLE TESTWARE – Symptom: Scripts are mainly “stand-alone” without
much reuse, automators are re-inventing • GOOD PROGRAMMING PRACTICES
– DESIGN FOR REUSE, OBJECT MAP
– Symptom: Structured & reusable scripts, automator / time shortage • DOMAIN-DRIVEN TESTING
– ABSTRACTION LEVELS, KEYWORD-DRIVEN TESTING
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
15
HIGH MAINTENANCE COST (Tour) - 2
• Symptom: No time to maintain automated tests (we know we should) – MANAGEMENT SUPPORT
• Explain technical debt, SELL THE BENEFITS
• Symptom: Scripts in tool language, tool changed or having to change tools – TOOL INDEPENDENCE – ABSTRACTION LEVELS
16
DOMAIN-DRIVEN TESTING
• Uses a “Domain-specific test language” – keywords from the application domain
• insurance: New Policy or Make Claim • mobile: Make Call or Update Contacts
– each keyword is processed by a script that then processes any data for that keyword
– tests in spreadsheet or databases • make it easy for business testers to add new tests • needs technical support (to set up the TESTWARE
ARCHITECTURE and when things go wrong)
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
17
Testware architecture: abstraction levels abstraction here:
easier to write automated tests: widely used DOMAIN-DRIVEN
TESTING
abstraction here: easier to maintain, and change
tools: long life GOOD PROGRAMMING
PRACTICES, GOOD DEVELOPMENT
PROCESS
testware archite
cture
Testers
Test Execu/on Tool runs scripts
High Level Keywords
Structured Scripts
structured testware
Test Automator(s)
write tests (in DSTL)
18
Some other patterns
• FAIL GRACEFULLY – make sure a test “cleans up” after itself even if it
fails (so subsequent tests can run without worry) – alternative: FRESH SETUP, where each tests re-
sets its initial state, no matter what happened before, or SHARED SETUP
• KILL THE ZOMBIES – make sure tests are actually doing something;
delete or archive tests no longer in use
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
19
Contents Background
Issues and examples Test automation Patterns Wiki
Conclusion
Test Automation Patterns
20
About these patterns (and the wiki)
• the wiki has a limited set of patterns – and a limited “depth” for each pattern
• patterns use other patterns – it can get rather complex
• we are focusing on system-level automation rather than unit test automation – See Gerard Meszaros’ book “xUnit Test Patterns:
Refactoring test code” for unit test patterns • these are test automation patterns, not test
patterns
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
21
Diagnostics (new)
• modeled on a doctor’s questions to a new patient – ask questions, and ask more questions, depending on the answers
• First level – new to automation? Go to NO PREVIOUS TEST
AUTOMATION – completely happy? giving up (good riddance)?
• share your story – want to improve or revive your automation?
• see next level
22
Improve or revive automation
• Most pressing issue: – lack of support (from mgt, testers, dev) – lack of resources (staff, s/w, h/w, time) – lack of direction (what to aut, what architecture) – lack of specific knowledge (of SUT, tool,
maintainability) – unsatisfactory quality of automation (ROI,
maintenance, unreliable) • each takes you to more detail in the wiki, and
points you to an issue with resolving patterns
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
23
Lack of support
• different kinds of support may be lacking – managers pay “lip service” – don’t see the value – managers have UNREALISTIC EXPECTATIONS – testers don’t help the automators – developers don’t help the automators – specialists don’t help the automators (DB,
networks) – no one helps the newbies
24
Contents Background
Issues and examples Test automation Patterns Wiki
Conclusion
Test Automation Patterns
[email protected] [email protected]
© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk
25
Conclusion • we hope our wiki will help you identify and
understand problems (issues) and give you ideas (patterns) for dealing with them
• the wiki is free, by invitation – email me at [email protected] for an
invitation – if you already have a wikispaces account, also give
me your wiki user name – an invitation will be sent from the wiki giving you
instructions for how to set up your user name & password
26
Finally
• Thank you for coming today • Any questions? • Contact us:
– [email protected] – [email protected]
• We welcome your feedback • We welcome your stories & experiences • All the best with your test automation!