Visual Regression Testing

@liammcmurray

Change happens.We have 15 different layout templates, composed of dozens of design patterns, controlled by over 6,000 lines of css.

Every presentational change we make relies on our familiarity with the site to know where we expect changes to appear.

In addition, we use Browserstack to quickly check a representative sample set of our layout templates.

However, this requires a fair whack of time to run, and also needs a person to sit and look at each snapshot that's produced.

And they need to know what to look for.

And often, it’s the thing you’re not looking for that’s the thing you need to see.

Regression testing.In developmentland, this is the process of checking your new hotness doesn’t break what you already have (causing a bug or a regression).

Regression testing is:

● Automated

● Definitive

● Reproducible

● Scoped

Visual regression testing.Similarly, this is comparing the before and after layouts of a page once a change to the appearance has been made.

Visual regression testing needs to be:

● Automated

● Definitive

● Reproducible

● Scoped

This has nothing to do withBenjamin Button.

So, visual regression testing.All the technology exists, we just need to put it all together.

Fortunately, much cleverer people than me have written a guide on how to do just that. The tools we’ll use are:

Selenium - a standalone environment with tools specifically for automating web browsers

WebDriverIO - allows you to control your web browser via javascript

WebDriverCSS - enables WebDriverIO to save screenshots automatically

graphicsMagick - commandline controls for image manipulation

Node.js - allows us to write and execute automated tests

Running the tests.Once you’ve installed all the bits and bobs (we’ve made a repo with a full guide), actually getting results is pretty easy.

Using javascript, you just define the location of the page(s) you want to test, what bits of the page to check, any further commands you want the browser to do (e.g. click a link), and what message to relay if there is a difference.

Still running the tests.Some command line magic to get everything set up and ready...

Then off it goes...

Getting results.All these screenshots are compared against a baseline set of images. If any changes are detected, a file documenting these changes is produced.

Live technical demonstration emulation.

Baseline.

Change.

Highlighted differences.

Next steps.Currently, this was a proof-of-concept (discovery) task that only tests a few elements on a single page, and we run the tests manually.

Things we want to do next:

1. Test all elements, on a representative set of pages

2. Run the tests automatically, whenever a change is pushed to our staging servers