Embed Size (px)
Citation preview
Noah Sussmannsnoahsussmancomnoahsussman
How Continuous Delivery is changing Quality AssuranceContinuous ImprovementGroupOnPalo AltoJanuary 15 2013
film still from The Lord of the Rings
The canonical Agile release cycle
Cocento Tecnologia on Flickr
Sprints of two or more weeks in length
TheMasonDixon on Etsy
Start deployment once the sprint is over
SisterDimension on Flickr
QA is part of the release process
film still from The Lord of the Rings
QA sign-off is required before going live
evoo73 on Flickr
The Continuous release cycle
Travis S on Flickr
Minimum viable feature set
Releasing a feature is decoupled from deploying code
David E Smith on Flickr
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
film still from The Lord of the Rings
The canonical Agile release cycle
Cocento Tecnologia on Flickr
Sprints of two or more weeks in length
TheMasonDixon on Etsy
Start deployment once the sprint is over
SisterDimension on Flickr
QA is part of the release process
film still from The Lord of the Rings
QA sign-off is required before going live
evoo73 on Flickr
The Continuous release cycle
Travis S on Flickr
Minimum viable feature set
Releasing a feature is decoupled from deploying code
David E Smith on Flickr
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Cocento Tecnologia on Flickr
Sprints of two or more weeks in length
TheMasonDixon on Etsy
Start deployment once the sprint is over
SisterDimension on Flickr
QA is part of the release process
film still from The Lord of the Rings
QA sign-off is required before going live
evoo73 on Flickr
The Continuous release cycle
Travis S on Flickr
Minimum viable feature set
Releasing a feature is decoupled from deploying code
David E Smith on Flickr
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
TheMasonDixon on Etsy
Start deployment once the sprint is over
SisterDimension on Flickr
QA is part of the release process
film still from The Lord of the Rings
QA sign-off is required before going live
evoo73 on Flickr
The Continuous release cycle
Travis S on Flickr
Minimum viable feature set
Releasing a feature is decoupled from deploying code
David E Smith on Flickr
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
SisterDimension on Flickr
QA is part of the release process
film still from The Lord of the Rings
QA sign-off is required before going live
evoo73 on Flickr
The Continuous release cycle
Travis S on Flickr
Minimum viable feature set
Releasing a feature is decoupled from deploying code
David E Smith on Flickr
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
film still from The Lord of the Rings
QA sign-off is required before going live
evoo73 on Flickr
The Continuous release cycle
Travis S on Flickr
Minimum viable feature set
Releasing a feature is decoupled from deploying code
David E Smith on Flickr
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
evoo73 on Flickr
The Continuous release cycle
Travis S on Flickr
Minimum viable feature set
Releasing a feature is decoupled from deploying code
David E Smith on Flickr
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Travis S on Flickr
Minimum viable feature set
Releasing a feature is decoupled from deploying code
David E Smith on Flickr
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Releasing a feature is decoupled from deploying code
David E Smith on Flickr
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
An airport without an air traffic controller
mdashChad Dickerson
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Etsy
Real-time data on how releases impact revenue
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Default to open access
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Constant tweaks to live features
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
dogpose on Flickr
Large features are deployed piecemeal over time
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Every feature is part of an AB campaign
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Joy and Jon
Dark launches
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Opt-in experiments
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Partial rollouts
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Wikipedia
Config Flags
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Joe Thomissen on Flickr
Wire-Offs
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
if ($cfg[new_search]) new hotness$resp = search_solr()
else old busted$resp = search_grep()
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
$cfg = array( checkout =gt true homepage =gt true profiles =gt true new_search =gt false)
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
There is no ldquodone donerdquo
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Observed Behavior Of Complex Systems
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Emergent behaviors require unplanned responses
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Improvements are discovered rather than designed
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Users of the system have complex expectations
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Complex systems are never ldquocompleterdquo
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
QA Happens When
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
First of all what is ldquoQuality Assurancerdquo
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
QA assuring that there are no defects
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
It is impossible to prove the absence of defects
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Lukjonis
There will always be bugs in production
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Testing is everyonersquos job
Library of Congress
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
The Jargon File
Myths About Bug Detection
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Myth there are a finite number of bugs
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
niscratz on Flickr
Myth here are a finite number of detectable bugs
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Myth all severity one bugs can be found before release
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Fred Brooks at Etsy
Myth software is built to specifications
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Myth at some point software is finished
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Myth most bugs have complex unpredictable causes
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking disguises that the error is the programmers own creation
mdash Edsger Dijkstra
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
The whole time Irsquom programming Irsquom constantly checking my assumptions
mdashRasmus Lerdorf
loriabys
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
As youre about to add a comment ask yourself ldquoHow can I improve the code so that this comment isnt neededrdquo Improve the code and then document it to make it even clearer
mdash Steve McConnell
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible you are by definition not smart enough to debug it
mdashBrian Kernighan
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
No blame
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Many Small Anomalies Combined
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
An organizations defenses against failure are a series of barriers represented as slices of swiss cheese The holes in the cheese represent weaknesses in individual parts of the system Failures occur when a hazard passes through all of the holes in all of the defenses
mdash Wikipedia
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
John Allspaw
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Prioritize the elimination of small errors
John Allspaw
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Focus less on mitigation of large catastrophic failures
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Optimize for recovery rather than failure prevention
Failure is inevitable
Richard Avedon
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Unit testing is great for preventing small errors
John Allspaw
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Resilience Not ldquoQualityrdquo
John Allspaw
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Readable code
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Reasonable test coverage
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Sane architecture
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Good debugging tools
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
An engineering culture that values refactoring
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Measurable goals
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Manual TestingBut probably not the kind yoursquore thinking of
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Real-Time Monitoring is the new face of testing
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Etsy
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Etsy
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Etsy
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Anomaly detection is hard
Greg and Tim Hildebrandt
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Watching the graphs
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
As of 2012 Etsy collected well over a quarter million real-time metrics
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Deciding which metrics matter is a human problem
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Everyone watches some subset of the graphs
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Human vision is an excellent tool for anomaly detection
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
QA happens when
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Exploratory testing can be performed at any time
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Rigorous scientific approach
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Focus on customer satisfaction
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Less focus on product specifications
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Exploratory Testing is equally useful before or after a release
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Just Quality
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
ldquoAssurancerdquo is a terrible word Letrsquos discard it
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Quality exists but itrsquos tricky to assure or prove that
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Therersquos no such thing as a formal proof of quality
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Most of us would agree that quality exists
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
ldquoCustomer Experiencerdquo is a better term of art than ldquoQualityrdquo
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Customer ExperienceThough therersquos no formal proof for that either
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Exploratory Testing addresses areas that Developer Testing doesnrsquot
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Developer Testing validates assumptions
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
The Independent Testerrsquos job is to invalidate assumptions
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Technology Informs Customer Experience
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Exploratory Testing requires an understanding of the whole system
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Exploratory Testing requires understanding how the system serves a community of users
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Customer Experience is as much about technology as it is about product requirements
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
mdash Eric S Raymond
Most bugs most of the time are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Source diffs logsIf your QA Analysts donrsquot look at these mdash teach them
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Customer Support
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Your customer support operators spend more time talking to your users than anyone else
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Customer Support interface with users as individuals rather than as aggregate data
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Keep the feedback loop short
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Manage Your Culture
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Effeciency To Thoroughness Trade-Off
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Rapid release cycles have different risks thanslower release cycles
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Continuous Delivery does not alter the fundamental nature of risk
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Test in both dev and prod
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Detectable errors should be caught in dev
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Undetectable errors must be worked out in production
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Software exists in context
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Networks services and people are always in flux
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Small changesets are easier to debug
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
An SCM revert is a changeset
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Large changesets are riskier and harder to debug
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Fail Forward
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
scrapnow on Etsy
Always deploy the HEAD revision of trunk
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Never roll back to an earlier state Always roll forward When its desireable to revert a previous change do that as part of a new commit
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Instead of rolling back fix the problem and move on
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Let go of the idea of ldquolast stable releaserdquo
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Scott Holloway
Focus less on satisfying the requirements
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Watch the graphs
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
NASA
Listen to your customers
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Kirsten Dunst on the set of Marie Antoinette
Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Kirsten Dunst on the set of Marie Antoinette
Low-Ceremony Process
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
WSHS Science blog
Iteratively improve your product
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Further ReadingldquoHow Google Tests Softwarerdquo James Whittaker
ldquoLook At Your Datardquo John Rausser
ldquoOptimizing For Developer Happinessrdquo Chad Dickerson
ldquoOutages Postmortems and Human Errorrdquo John Allspaw
httpenwikipediaorgwikiSwiss_cheese_model
ldquoWhat Is Exploratory Testingrdquo James Bach
Questions
noahsussmannsnoahsussmancominfiniteundocom
Questions
noahsussmannsnoahsussmancominfiniteundocom
