A Walk in the FieldsWhy Test Automation Is Really, Really Difficult, and Why It Usually Sucks

The Problem The product is ever-changing Developers do not communicate

changes proactively (or at all) 40% of our time was spent accounting

for these changes

We should be guided by features, not failures

The Solution Move testing further UP the chain, not

down Embed automation testers with regular

developers Use automation to GATE the release Apply some Principles of Real Software

Design to our little corner of the process

What’s in it for me? Managers – Increase your confidence

day-to-day and release-to-release Developers – no more whiny testers,

now they’re whiny PARTNERS Testers – worry about developing NEW

tests, not fixing broken ones

Add a User

Let’s Look at Some Code(Don’t worry about trying to read this, I’m trying to convey quantity, not quality)

Server-side input validation

Velocity Definition This is just for that one button!

DAO Code Hooking the fields to the database

Some SQL

What Was all That??! Recursive Velocity templates Java (with Hibernate annotations) In-depth Velocity macro definitions More Java SQL We didn’t even look at the CSS or

JavaScript

What Does QA Typically Know About This Stuff?

Some Numbers Some analysis and estimation around the

form: 150 lines of Velocity 600 lines of Java 200 lines of SQL 6,000+ lines of shared JavaScript libraries 1,000 lines of shared CSS

Whenever any of this changes, it can affect the functional, presentational, or performance aspects of this form

More Numbers This is one of 8 different forms

surrounding the creation/maintenance of users

User is one of ~200 top-level entities in the system

800 pages Thousands of forms, buttons, and

dynamic screen elements What happens if one thing changes?

Facts are facts Little of the code that does show up in

the browser was written, or would be easily understood, by the developer

It’s either auto-generated by the templates, or was designed by the UX team

What Makes it Complicated for Testing?

Why So Serious? 6 big fat lozenges of code that you

didn’t write Then there’s the “robot” you use to

control it, which you also didn’t write QTP SilkTest Watir Selenium/WebDriver

High Maintenance Tests Everything is in the test, in one big layer

Baby Steps Lower maintenance:

Utility or “Helper” methods Locators stored in xml or properties files

or something

Adolescence Even lower maintenance:

Page Object Model—tests are driven by the functionality of the site

Strongly-typed WebElements Database abstractions—an “API” for

testing Continuous Integration Screen shots upon failures

Adulthood No longer guided by failure:

User-modeled actions—tests are driven by use cases more than pure functionality

Domain-specific language, coupled with strong Page Objects

EventFiringWebDriver to give you pinpoint logging and error handling

TestNG TestListeners coupled with the above for fine-grained failure handling

Nirvana Virtually maintenance-free:

All of the above, plus… Page Object generation, created/updated

at build time whenever the developer makes an update

A Velocity Template

The generated code

Managers: You Want a Strategy? TEST AUTOMATION DEVELOPMENT IS

SOFTWARE DEVELOPMENT Treat it like any other project, but one in

which you *can* control the variables You already know 90% of what you need to

know Just make the connection and make the

commitment Insist on automation involvement from the

earliest point

Engineers: Now what? Make a connection with your automation

resources Find areas where your presentation

layer could be more consistent Prototype on a simple page or single

element type Figure out how you can communicate

change via continuous build instead of hallway conversations

Testers Now you don’t have to chase down

changes You understand much better how the

software gets built You can provide metrics and perspective

you couldn’t before Your code base is tied to theirs—sounds

dubious, but turns out to be awesome

Pros Maintenance goes way down Automation code is now tied to

developer code Including version-specific tests

QA now has skin in the development game

Development now has skin in the automation game

Cons Heavy up-front development time

But you should be quick-to-fail Need a higher level skill set than the

traditional QA role Difficult to implement on non-templated

systems Velocity, JSF, JSP, ASP, wicket OK PHP, JavaScript, Java-based HTML not so

easy

The Risk? “If they plugged up the leaks that allow

us to hack FireFox, this whole thing would go away.”

“Developers are a huge risk to test automation. They shouldn’t be allowed near the codebase.”

—Simon Stewart

Other ideas Could you somehow track the changes,

to get an idea of “churn rate” in your presentation layer?

Compile to some kind of meta-step, so the target, rather than being pure Java, could interface with anything?

Standard for “Object Repository”, maybe as an addition to Selenium?

Questions?