Embed Size (px)
Citation preview
ConfidentialPA12013-03-261 ConfidentialPA12013-03-261
Testing vs. Checking Rephrased
ConfidentialPA12013-03-262
Introduction
• This presentation is based on– Michael Bolton
• http://www.developsense.com/blog/2009/08/testing-vs-checking/
– Cem Kaner• http://context-driven-testing.com/?p=69
• And in many ways just a rephrasing of – Marcus Gärtner
• http://www.shino.de/2013/03/21/automated-vs-exploratory-we-got-it-wrong/
ConfidentialPA12013-03-263
Introduction
• These are just thoughts and ideas written down to further stimulate discussions
• Much of what is written here have probably already been stated by someone else
• This presentation helped me to think about testing vs. checking and develop my own thoughts around the subject
ConfidentialPA12013-03-264
Overview
• The main difference between testing and checking is not whether it requires sapience or not
• It is not whether the goal of the activity is to find new errors or to just avoid regressions
• The main difference between testing and checking is whether you are further detailing the specifications or not
ConfidentialPA12013-03-265
Detailing the Specification
Specification Designing Unit TestsArchitectural Design
and Coding Guidelines
Developing Code Code Review
Designing API Tests Designing Manual Tests
Designing UI Automated Tests Exploratory Test
Detailing the SpecificationThis is not en exhaustive list of activities to detail the specification, and they don’t have to be executed in the order listed here, and can be executed in parallel
ConfidentialPA12013-03-266
Confirming the Specification
Specification Static Analysis of Code
Executing Automatic Tests
Executing Scripted Manual Tests
Confirming the Specification
This is not en exhaustive list of activities to confirm the specification, and they don’t have to be executed in the order listed here, and can be executed in parallel
ConfidentialPA12013-03-267
Testing vs. Checking
• If you want to further detail your specification, you perform testing
• If you want to confirm an already existing specification you perform checking
• As seen in the pictures you can do other activities than testing/checking to detail/confirm the specifications
ConfidentialPA12013-03-268
System Complexity
• For a simple system it is possible to do all the specification detailing upfront, by just designing architecture, developing code and writing unit tests
• For complex system, an upfront approach is impossible, since we cannot perceive exactly how the system will operate in every detail – we need feedback from the system to be able to further detail the specification – exploring the system provides this feedback
ConfidentialPA12013-03-269
Automated vs. Exploratory
• If you want to detail your specification further, and also have the possibility to confirm the specification on several occasions later on, automation (or scripted tests) is suitable
• If you want to detail your specification further, but will not need to confirm the specification later on, exploratory testing is more cost effective
• If you need feedback from the system to be able to detail it further, exploring the system in some way is necessary
ConfidentialPA12013-03-2610
How exploring details the specification
• When performing exploratory testing, some of what is discovered is written down, and then immediately details the specification
• That which is not written down during exploratory testing still details the specification, but only in the tester’s own perception, as tacit knowledge
ConfidentialPA12013-03-2611
Feedback from the System
• In complex systems, it is impossible to list how the system should behave in all different states, since the number of states is just to high
• When exploring the system, you enter states which have previously not been specified, and by deciding whether or not the behaviour is acceptable, you detail the specification in one way or the other
ConfidentialPA12013-03-2612
Summary
• Testing details the specification, Checking confirms the specification
• It is necessary to explore complex systems, and get feedback from that system, to be able to detail them past a certain point
