Viktor Slavchev &The Testing trollSiteground
The “worst” practices of software testing
About me
• Senior QA @ Siteground Hosting• Wooden philosopher at Pragmatic
(part time)• Blogger @ mrslavchev.com(long
time ago in a galaxy far far away)• True believer that testing is a
human centric activity
• A mythical testing creature• Graduated the technical
university of Trolandia• Certified as SHGMCNGTT -
Super Hyper Giga Mega Certified Ninja Guru Testing troll
• Book author
About The testing troll
And the book:
• Absolute bestseller• Sold more copies than “50
shades of gray” in Trolandia• “Absolutely mind-blowing”
New York Times• “The troll is strong with this
one”Darth Vader
Best practice#1 - Rely on requirements and documentation
Worst practice #1: Learn about the real testing oracles instead
Testing oracles can benefit from:
Feelings and mental models
• It starts as a feeling• Not sufficient for a
bug report
Artifacts: documents, designs, requirements, etc.
Experience in:
• The domain• General testing• Software industry
Conference
• Developers• Other testers• Clients• Anyone who
matters
Example 1: Reference oracle
• External application• Other platform• Correctness presumed
Example 2: Self-verifying data as an oracle
Desired response: Credit card #:
000: Approved 4470330769941000
010: Partially Approved 4658512425423010
110: Insufficient Funds 4488282659650110
Example 3: Constraints oracle
Example: VAT number
BG123456789• Length• Country index• Spaces
Source:The Oracle Problem and the Teaching of Software Testing
Best practice#2 - Do regression testing by executing your old tests
Regression testing != repetitive testing
Repetitive testing is the reason why “others” see testing like this...
Worst practice #2: Regression checks and focus on tests that reveal new information
Regress - verb• to move backward; go back.• to revert to an earlier or less advanced
state or form
Regression bug:• Bug causing regress in
quality
Regression testing:• Testing to expose
regression bugs
Regression check:• Building part of a test• Definite answer• Small and easy to
manage
Focus on information
Best practice#3 - Automationion testing is the best, “manual” testing sucks
Quote from J. Kohl - Tap into mobile application testing
Ideas that will produce bug-free software:• C++ STL• OOP• Agile• TDD• Test automation
Quote from J. Kohl - Tap into mobile application testing
“Each of these were also used to tell me that testers would no longer be needed on software projects. So far, I am fifteen years in and I haven’t seen anything bug free yet.”
Can a testing tool produce better testers? Answer:
How to make efficient automation that provides value to testing?The answer is:
Worst practice #3: Use automation as a tool
• To extend your abilities • Not to replace you
completely• To ask the right
questions
Automation is great for:
• Confirmation / Verification
• Low level checking• Precision• Functional correctness• Solving machine
problems
But automation sucks when dealing with:• Exploration• Unanticipated risks• Qualitative evaluation• Open ended questions• Solving human
problemsSource:Things Could Get Worse- Michael Bolton
Source: The non-manual, unautomated tester
Best practice#4 - Assure quality
Quality is not a resourceQuality != testing
Worst practice #4: Provide information about risks
• Risk based testing - focus testing to explore potential risks
“Nightmare newspaper headings”
• You missed a critical bug in production
• It caused your company a huge financial loss
• What would it be?
• Pre-mortem - imagining a failure and working backwards in order to determine the possible risks.
Benefits:• Evaluation of risks• Business perspective• Prioritization
Best practice#5 - Follow best practices
Worst practice #5: Be always alert and aware of your context• What would be your
approach in 3 different contexts?
Context #1: Weather app
• You’re an outsourced testing expert
• Working for a start-up• Low on budget• Behind schedule
Context #2: Medical software
• High precision is a must• Human lives might
depend on it• Regulatory compliance
of testing
Context #3: Court case
• Bug caused an injury• You are independant
expert• On a case for negligence
in testing
Questions:
• How exhaustive will be your testing in each context?
• Will bug-hunting be your highest priority?
• Will you have the same strategy of testing?
Conclusion
• No best practices!• “We are not into the
belief business” J.Bach• In order to be a good
tester, you have to be a professional sceptic
Thank you very much!
… And don’t forget to Rock ‘n Troll
Image credits:
Too lazy to post them all. Image credits go to the almighty internet, I don’t own any of the images included in this presentation, except the ones created by me, linked to my blog.
STIONS?
ANY