Why Automated Verification Matters

Kristian Karl, Spotify

@kristiankarl

1983 -Started programming1994 -Testing and automation1998 -Consultant2004 -Discovered Model-Based

Testing2010 -Test Manager at Spotify

TechnologyProduct

Design

Confidence

Why does automated verification matter?

● What can we check?● What is important to check?● How can we check it?● How to involve everyone?● How do we get ROI?

● Dev environments● Builds on pull requests and merge● Static code analysis● Code style checking● Dynamic code analysis● Verification● Archiving artefacts● Deployment

What can be automated?

BusinessLogic Coverage

ExecutionTime / Costs

CodeCoverage

TestType

UI

Integration

Unit Test

High

Medium

Low

Low

Medium

High

Long / High

Medium

Short /Low

A human using her intelligence and experience to critique and explore a system under test.

Testing

http://www.youtube.com/watch?v=IGQmdoK_ZfY

Agile Testing

Check forthe

expected

Findthe

unexpected

Business

Technology

● Functional testing● BDD● Specification by

example

● Unit tests● Component tests

● Exploratory testing● Scenarios● Usability testing

● Performance● Security

Our approach at SpotifyDisclaimer: There are 100+teams at Spotify, and theyall do things a bit different.Some teams do itvery differently...

Code change

Pre-merge build and

tests

Code and quality review

Risk analysis

Exploratory testing

Continuous Delivery

Pre-merge● Quick feedback loop● The total time for the build needs to

be time boxed● Style checking● Static code analysis● Unit tests● Stable functional tests● Stable integration tests● Packaging

Post-merge● All the steps from the pre-review

build● Longer running functional and

integration tests● System tests● Non-functional tests, like:

- resource leaks- performance

What has helped Spotify doing automated verification?

The Test Data Service

The Test The TestData Service

Some Production Service

Request:● Premium user● Not connected to Facebook● Using feature flag X

Query for user(s) matchingThe criterias

List a matching user(s)

Return selected userPick one user, and lock it,Prevent any other test from using that user.

Tim

e

Model-based Testing

à la GraphWalker

App not running

Authentication form displayed

Home page

Start app

Close

Exit

Valid credentials

Bad credentials

Logout

Our Test Interface(s)

Test automator role Developer role

Defines a testing API

Implements the testing API

Test Result Service

One test result index to rule them all - Many ways of visualizing

So what’s the lessons learned so far?

Backend teams

● Backend teams work with smallish micro services.

● High degree of automation on all parts of the pipeline.

● No one but the team touches the team’s code.● They have operational responsibility.● Few defects found after definition of done.

Frontend [mobile & desktop] teams

● Building monoliths● 200+ people involved● Low degree of automated verification● Low testability● High number of defects are

found after definition of done.That hurts productivity...

Tragedy of the commonsIndividuals acting independently and rationally according to each other's self-interest, behave contrary to the best interests of the whole group

What can you do to improve?

● Ways of working are more important than tools and frameworks.

● Zero bug policy, instead of huge backlog of bugs.● Pairing and embed● Everyone writes and maintain

automated verification.● Testability will require

non-functionality.● Failed verifications must be

fixed ASAP.

Automated verification matters, because it can

● Increase your productivity● Give you confidence to deliver anytime● Enable you work with

defect prevention, rather than defect detection

References

● https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/● https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/● https://dl.dropboxusercontent.com/u/1018963/Articles/SpotifyScaling.pdf● https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-

automation-pyramid● http://testautomationpatterns.wikispaces.com/● http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/● https://www.youtube.com/watch?v=IGQmdoK_ZfY● http://graphwalker.org/