+

Testing & ScrumHow to structure testing within the Scrum

Framework?

+Introduction

This presentation outlines my interpretation of how to work with

testing within the Scrum Framework

+Scrum

+ Generalizing Specialists (or Specializing

Generalists)[12]

+ Key Message: Everyone is a Tester

Everyone in the Scrum Team must contribute to the testing

effort

Each person has a different skillset that must be used efficiently

in the testing effort

So everyone in the team is a tester – but certain people have a

skillset that is optimal for finding those complex problems that

other people might miss

+Quality Definition

• “Quality is value to some person”[1]

• Working software is primary measure of progress [2]

• The Scrum Team owns the quality of the product [3]

• The Definition of Done should define a level of quality of the

output that the Development Team delivers

+Acceptance Criteria [4]

“Acceptance Criteria are the conditions that a software product must satisfy to

be accepted by a user, customer, or in the case of system level functionality,

the consuming system.”

“Acceptance Criteria are a set of statements, each with a clear pass/fail result,

that specify both functional and non-functional requirements, and are

applicable at the Epic, Feature, and Story Level. Acceptance criteria constitute

our “Definition of Done”, and by done I mean well done.”

The Given/When/Then format is helpful way to specify acceptance criteria:

Given some precondition When I do some action Then I expect some result

+Acceptance Criteria & Testers

Writing Acceptance Criteria has a lot in common with testing

It requires much of the same skill set to be able to write good

Acceptance Criteria as it does to perform testing

Even if the Product Owner is responsible for the Acceptance

Criteria, a tester can add value by contributing with his/her

expertise

+Agile Test Quadrants [7]

+ Key Message: Testing is infused into everything

Scrum is founded on Empirical Process Control [3]

Empiricism asserts that knowledge comes from experience and

making decisions based on what is known [3]

Testing is not something you just do at the end of a sprint – it is

infused into basically every activity

+Test Ownership

Scrum Team

Outside of Scrum Team

Isolated Tests

Contract/Collaboration Tests

Integration Tests

System Tests

Equipment & Competence Specific

Tests

We need to identify who is responsible for performing what types of tests

There are only two important parties for the Scrum Team

The Team

Outside of the Team

Clear ownership is Key

This is just my way of grouping different types of tests, it is not the only way – find a way that works for you

Test Type Definitions Contract / Collaboration / Integration tests

Integrated Tests are a Scam by J.B. Rainsberger [8]

Contract Test:

http://thecodewhisperer.tumblr.com/post/1325859246/in-brief-contract-tests

“Contract Tests explain how a class should extend a superclass or implement an interface, so that I don’t

have to read a bunch of prose to figure out how to do that.”

“…a test class that can run the same set of tests for two different implementations of the same

interface.”[11]

Collaboration Tests: Also known as interaction tests, as opposed to state-based tests

“Does the client talk to the next layer correctly?”

“Testing interactions means you're verifying that the code under test calls certain methods properly.”

http://googletesting.blogspot.se/2013/03/testing-on-toilet-testing-state-vs.html

Integration tests: http://en.wikipedia.org/wiki/Integration_testing

“An integrated test is any (low level) test where when it fails you cannot pinpoint exactly what went wrong.“

Individual software modules are combined and tested as a group

Test Type Definitions System Test

Tests that span across features and teams due to the complexity of the product

If product and/or feature complexity is very low, then integration tests may be enough to

sufficiently cover the test space

Equipment Specific Tests

Tests that require specific hardware or software that is not available to the development teams

If all tools are available to the Scrum Team(s), then this category disappears

Competence Specific Tests

Tests that require specific competence that is not available in the development teams

Examples: Localization, Tracking, Cross promotions, Cross missions, tests that require

understanding of multiple King games, etc.

If all competences are available to the Scrum Team(s), then this category disappears

Key Message: Complexity is the main separator

Using the Cynefin framework [9]

Simple tests can be done by anyone

Complicated tests are well suited for someone with a good understanding of the system

Complex tests are well suited for someone with a good testing skillset and a good understanding of the system

Chaotic tests are … difficult?

There is a place for someone with a strong testing skillset both in the Scrum Team, and outside of the Scrum Team

Key Message: You can place some testing outside of

the Scum Team if you have multiple teams

There is a point in placing some testing outside of the Scrum

Team

System test that spans over multiple features and teams

Tests that require equipment not readily available to all teams

Tests that require competence not readily available to all teams

However these test teams must work in parallel with all the

Scrum Teams, and this creates some interesting coordination

problems

Test Automation

This way of working obviously favors automated checks, but in

no way mandates it

The Scrum Team should make the decision what they want to

automate and what they want to test manually

Even if you have automated checks that cover isolated

components, communication between components, and how

multiple components work together in a group, you will still

need to perform manual testing if you have a user interface

+Testing & Checking

“Checking is the process of making evaluations by applying algorithmic decision rules

to specific observations of a product.” [5]

Preferably all manual testing should be actual testing and not

checking

Checking detailed Acceptance Criteria that are hard to automate could

be an exception

Designing and prototyping automated checks is a test activity

Running automated regression checks is a check activity

+(Exploratory) Testing [6]

“Testing is the process of evaluating a product by learning

about it through exploration and experimentation, which

includes: questioning, study, modeling, observation and

inference, output checking, etc.”

All testing is exploratory, even scripted testing, if you are doing

it responsibly

+Regression Testing

Everyone is responsible for covering their own test areas during regression test activities

This motivates programmers and testers to design the system properly with testability in mind, and to automate efficiently

During all test activities you must always have a risk-based approach [10]

You cannot cover the entire test space every time and still be efficient

This drives the need to reduce system complexity and design the system with this in mind

+Conclusion

Testing is an integral part of the Scrum Framework

Everyone should contribute

But there is still a place for a strong testing skillset, driven by

the complexity of the product

Test Ownership should be clear and it is possible to place some

testing outside of the Scrum Team

References

[1] Definition of QualityWeinberg, Gerald M. (1992), Quality Software Management: Volume 1, Systems Thinking, New York, NY: Dorset House Publishing, p. 7[2]Agile Manifesto Principleshttp://agilemanifesto.org/principles.html[3] The Scrum Guidehttp://www.scrumguides.org/docs/scrumguide/v1/scrum-guide-us.pdf[4] Acceptance Criteriahttp://www.leadingagile.com/2014/09/acceptance-criteria/[5] Testing and Checkinghttp://www.satisfice.com/blog/archives/856[6] Exploratory Testing 3.0http://www.satisfice.com/blog/archives/1509[7] Agile Testing Quadrantshttp://www.developsense.com/presentations/2014-06-Dublin-RSTAgileTesting.pdf[8]Integration Tests are a Scamhttps://vimeo.com/80533536[9]Cynefinhttp://en.wikipedia.org/wiki/Cynefin[10] Heuristic Risk-Based Testinghttp://www.satisfice.com/articles/hrbt.pdf[11]Contract Tests: An Examplehttp://blog.thecodewhisperer.com/2011/07/07/contract-tests-an-example/[12]To combine … or nothttp://angryweasel.com/blog/to-combine-or-not/