Embed Size (px)
DESCRIPTION
Millions of test cases executed thousands of times mean nothing when a catastrophic defect surfaces and threatens the value of the product. In software testing, what is not tested is more important than what has been tested. However, with continuous adoption of agile and automation, the focus has shifted to what has been tested. In an agile environment, where results of continuous integration are visible to everyone, it’s easy to get fooled. In this talk, I will explore how validations we seek with testing are affected by fallacies and biases and why green reported by continuous integration might not be good enough. I will try to demonstrate limitations of the acceptance criteria by showing examples of defects in open-source projects. I will conclude with the discussion on how a context-driven approach can safe-guard us from the inferences we may make from the green build. If teams are not context-driven, all the benefits we see from adopting agile, automation and continuous integration can be short-lived. Projects are not judged by the presence of automation or green build; projects are judged by how they work in production. Remember the case of Ariane 501, where the project was progressing well and the team celebrated the first 36 seconds of the launch. However, the spacecraft crashed in the 37th second and the project became unsuccessful. This paper is based on an excellent book by Nassim Nicholas Tale – “The Black Swan” and relates limitations of validation to software testing.
Citation preview
Misleading validations: Be aware of greenAnand Ramdeo
Agile Testing Days2013
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
1
Disclaimer - Say no to
Test Automation
ATDD / BDD
Continuous Integration
Continuous [email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
2
EmbraceTest Automation
ATDD / BDD
Continuous Integration
Continuous Delivery
“To exist is to change,to change is to mature, to mature is to go on
creating oneself endlessly”
Henri Bergson
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
3
Embrace
Test Automation
ATDD / BDD
Continuous Integration
Continuous Delivery
WithCaution!!
One line can change it all.
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
4
The Black Swan - Nassim Nicholas Taleb (NNT)
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
5
MediocristanNumber of test cases
Defects found
Defects fixed
Time between failed builds
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
6
ExtrimistanOne defect can change everything
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
7
Software Testing
Problem to Discover?or
Problem to Prove?
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
8
Right way to developSoftware
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
9
Men are disturbed not by things, but by the views which
they take of them.Epictetus
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
10
ATDD + TDD = Happy Customer?
TDDRight wayto developSoftware.
ATDD/BDDBuild the thing PO wants.
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
11
Information or Theory?
FromRed is bad, green is good.
ToIt’s Green, so it must be good.
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
12
Be aware of confirmation bias, understand what greenreally means.
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
13
So I have a theory (Green is Good)
andI am suffering from confirmation bias
(Pre-defined tests are passing in CI)
butWhat about other members in the team?
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
14
Tunnel VisionWhat are we delivering in this
Sprint?
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
15
System viewis extremelyImportant.
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
16
Lucid Fallacy100% code coverage Continuous Integration in
placeMandatory code reviewsAcceptance tests are part of
the build
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
17
First Valentine Second Valentine Third Valentine
With every passing year, we are getting closer :-)
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
18
So I have a theory (Green is Good)
andI am suffering from confirmation bias
(Pre-defined tests are passing in CI)
andTeam members have the same tunnel vision
(Focus is on current sprint)
andWe are affected by lucid fallacy
(No red build from past seven days!!)
butSurely, we can learn from failures, when it fails.
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
20
Project was progressing well and delivered on time, however customer is not happy.
OR
Project was progressing well and delivered on time, however customer is not happy because upgraded version
corrupted their existing data.
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
21
So I have a theory (Green is Good)
andI am suffering from confirmation bias
(Pre-defined tests are passing in CI)
andTeam members have the same tunnel vision
(Focus is on current sprint)
andWe are affected by lucid fallacy
(No red build from past seven days!!)
andWe can’t learn from failure because of narrative fallacy
(Something failed in production, investigate, fix, root cause analysis etc.)
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
22
Be awareOur job is to prevent teams from taking un-
informed risks.
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
23
So does green work?
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
24
The value of any practice depends on its
context.
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
25
GivenThat time runs out or participants are boredWhenSpeaker have nothing more to say or can not answerThanSpeaker say thank you for your time and attentions.
Questions / Discussions?
[email protected] | Blog: TestingGeek.com | @testinggeek | Play: TestSpicer.com
“Please evaluate my presentation and use for this the AgileTD Mobile App which you can find atwww.touchmyconference.com/ATD2013. I would appreciate your feedbacks. Thank you very much!”
26
