+
Testing & ScrumHow to structure testing within the Scrum
Framework?
+Introduction
This presentation outlines my interpretation of how to work with
testing within the Scrum Framework
+Scrum
+ Generalizing Specialists (or Specializing
Generalists)[12]
+ Key Message: Everyone is a Tester
Everyone in the Scrum Team must contribute to the testing
effort
Each person has a different skillset that must be used efficiently
in the testing effort
So everyone in the team is a tester – but certain people have a
skillset that is optimal for finding those complex problems that
other people might miss
+Quality Definition
• “Quality is value to some person”[1]
• Working software is primary measure of progress [2]
• The Scrum Team owns the quality of the product [3]
• The Definition of Done should define a level of quality of the
output that the Development Team delivers
+Acceptance Criteria [4]
“Acceptance Criteria are the conditions that a software product must satisfy to
be accepted by a user, customer, or in the case of system level functionality,
the consuming system.”
“Acceptance Criteria are a set of statements, each with a clear pass/fail result,
that specify both functional and non-functional requirements, and are
applicable at the Epic, Feature, and Story Level. Acceptance criteria constitute
our “Definition of Done”, and by done I mean well done.”
The Given/When/Then format is helpful way to specify acceptance criteria:
Given some precondition When I do some action Then I expect some result
+Acceptance Criteria & Testers
Writing Acceptance Criteria has a lot in common with testing
It requires much of the same skill set to be able to write good
Acceptance Criteria as it does to perform testing
Even if the Product Owner is responsible for the Acceptance
Criteria, a tester can add value by contributing with his/her
expertise
+Agile Test Quadrants [7]
+ Key Message: Testing is infused into everything
Scrum is founded on Empirical Process Control [3]
Empiricism asserts that knowledge comes from experience and
making decisions based on what is known [3]
Testing is not something you just do at the end of a sprint – it is
infused into basically every activity
+Test Ownership
Scrum Team
Outside of Scrum Team
Isolated Tests
Contract/Collaboration Tests
Integration Tests
System Tests
Equipment & Competence Specific
Tests
We need to identify who is responsible for performing what types of tests
There are only two important parties for the Scrum Team
The Team
Outside of the Team
Clear ownership is Key
This is just my way of grouping different types of tests, it is not the only way – find a way that works for you
Test Type Definitions Contract / Collaboration / Integration tests
Integrated Tests are a Scam by J.B. Rainsberger [8]
Contract Test:
http://thecodewhisperer.tumblr.com/post/1325859246/in-brief-contract-tests
“Contract Tests explain how a class should extend a superclass or implement an interface, so that I don’t
have to read a bunch of prose to figure out how to do that.”
“…a test class that can run the same set of tests for two different implementations of the same
interface.”[11]
Collaboration Tests: Also known as interaction tests, as opposed to state-based tests
“Does the client talk to the next layer correctly?”
“Testing interactions means you're verifying that the code under test calls certain methods properly.”
http://googletesting.blogspot.se/2013/03/testing-on-toilet-testing-state-vs.html
Integration tests: http://en.wikipedia.org/wiki/Integration_testing
“An integrated test is any (low level) test where when it fails you cannot pinpoint exactly what went wrong.“
Individual software modules are combined and tested as a group
Test Type Definitions System Test
Tests that span across features and teams due to the complexity of the product
If product and/or feature complexity is very low, then integration tests may be enough to
sufficiently cover the test space
Equipment Specific Tests
Tests that require specific hardware or software that is not available to the development teams
If all tools are available to the Scrum Team(s), then this category disappears
Competence Specific Tests
Tests that require specific competence that is not available in the development teams
Examples: Localization, Tracking, Cross promotions, Cross missions, tests that require
understanding of multiple King games, etc.
If all competences are available to the Scrum Team(s), then this category disappears
Key Message: Complexity is the main separator
Using the Cynefin framework [9]
Simple tests can be done by anyone
Complicated tests are well suited for someone with a good understanding of the system
Complex tests are well suited for someone with a good testing skillset and a good understanding of the system
Chaotic tests are … difficult?
There is a place for someone with a strong testing skillset both in the Scrum Team, and outside of the Scrum Team
Key Message: You can place some testing outside of
the Scum Team if you have multiple teams
There is a point in placing some testing outside of the Scrum
Team
System test that spans over multiple features and teams
Tests that require equipment not readily available to all teams
Tests that require competence not readily available to all teams
However these test teams must work in parallel with all the
Scrum Teams, and this creates some interesting coordination
problems
Test Automation
This way of working obviously favors automated checks, but in
no way mandates it
The Scrum Team should make the decision what they want to
automate and what they want to test manually
Even if you have automated checks that cover isolated
components, communication between components, and how
multiple components work together in a group, you will still
need to perform manual testing if you have a user interface
+Testing & Checking
“Checking is the process of making evaluations by applying algorithmic decision rules
to specific observations of a product.” [5]
Preferably all manual testing should be actual testing and not
checking
Checking detailed Acceptance Criteria that are hard to automate could
be an exception
Designing and prototyping automated checks is a test activity
Running automated regression checks is a check activity
+(Exploratory) Testing [6]
“Testing is the process of evaluating a product by learning
about it through exploration and experimentation, which
includes: questioning, study, modeling, observation and
inference, output checking, etc.”
All testing is exploratory, even scripted testing, if you are doing
it responsibly
+Regression Testing
Everyone is responsible for covering their own test areas during regression test activities
This motivates programmers and testers to design the system properly with testability in mind, and to automate efficiently
During all test activities you must always have a risk-based approach [10]
You cannot cover the entire test space every time and still be efficient
This drives the need to reduce system complexity and design the system with this in mind
+Conclusion
Testing is an integral part of the Scrum Framework
Everyone should contribute
But there is still a place for a strong testing skillset, driven by
the complexity of the product
Test Ownership should be clear and it is possible to place some
testing outside of the Scrum Team
References
[1] Definition of QualityWeinberg, Gerald M. (1992), Quality Software Management: Volume 1, Systems Thinking, New York, NY: Dorset House Publishing, p. 7[2]Agile Manifesto Principleshttp://agilemanifesto.org/principles.html[3] The Scrum Guidehttp://www.scrumguides.org/docs/scrumguide/v1/scrum-guide-us.pdf[4] Acceptance Criteriahttp://www.leadingagile.com/2014/09/acceptance-criteria/[5] Testing and Checkinghttp://www.satisfice.com/blog/archives/856[6] Exploratory Testing 3.0http://www.satisfice.com/blog/archives/1509[7] Agile Testing Quadrantshttp://www.developsense.com/presentations/2014-06-Dublin-RSTAgileTesting.pdf[8]Integration Tests are a Scamhttps://vimeo.com/80533536[9]Cynefinhttp://en.wikipedia.org/wiki/Cynefin[10] Heuristic Risk-Based Testinghttp://www.satisfice.com/articles/hrbt.pdf[11]Contract Tests: An Examplehttp://blog.thecodewhisperer.com/2011/07/07/contract-tests-an-example/[12]To combine … or nothttp://angryweasel.com/blog/to-combine-or-not/