Exploratory Testing

Or how to find bugs faster…

What is Exploratory Testing? Why are we doing black box testing? Session Based Testing Where do we test? Types of testing How outlines can help How flow charts could help

Summary

Exploratory testing is an approach to software testing that is concisely

described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1983,[1] now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."[2]

While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run. Exploratory testing is often thought of as a black box testing technique. Instead, those who have studied it consider it a test approach that can be applied to any test technique, at any stage in the development process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement of the tester, and the tester's responsibility for managing his or her time.[3]

- http://en.wikipedia.org/wiki/Exploratory_testing

What is Exploratory Testing?

is not just adhoc and saying adhoc is a turn off

to people who don't understand exploratory testing. Why? True adhoc is literally monkey pressing buttons.

… is a learning process. The code will change as the project

progresses. Doing the same thing over and over again has diminishing returns

Not a replacement for doing/running test cases

Exploratory Testing

Is not a replacement for automation testing It’s a supplement to reduce the risk of the

unknown quantity of bugs

Comment : I would implore various strategies of depth and breadth type testing in trying to reduce the risk of an unknown quality and severity of bugs to make them known.

Exploratory Testing

Because human thoughts and behavior

changes from individuals as well as over time. We don’t necessarily do the same thing over and over again like automation testing does

Bottom line : people think. And people think differently.

Why do we do black box testing?

Be aware of yourself and why you may miss

bugs: http://viscog.beckman.illinois.edu/media/

mistakes.html

http://viscog.beckman.illinois.edu/flashmovie/15.php

Human Cognitive Behavior

Be a scientist! Poke around! Try to figure things out Go beyond just doing the test case… Talk to each other about it, learn from each other If you don’t understand the bug, then how do you expect other

people to? It’s like chess, try to think in steps ahead Test cases aren’t the rule book; they are a guideline

The more you test the more experience you will gain. You can learn from any source; I often learn from people I have mentored as much as I’ve learned from the people who have taught me!

Testing

Session-based testing is a software test method

that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. The method can also be used in conjunction with scenario testing. Session-based testing was developed in 2000 by Jonathan and James Bach.

Highly recommend looking at this… this is what top blackbox testers do in finding bugs as well as other things discussed later on.

Session Based Testing

New Code Change

Looking at code change Trends in bugs Talking to Dev; what might they be worried about? Talking to other QA; what seems fragile? Risk areas

What’s dangerous if it fails for human safety Embarrassment (functionality) Complexity untested

Where to test?

MVC = Model View Controller

UI Testing Functional/Business Logic testing Data/Input Testing

UI Testing Does user flow make sense? (Usability) Culturally effective? Which culture? Does it look cluttered?

Types of testing

Functional Testing (Positive Testing) Negative Testing Boundary Testing Race Conditions Environment conditions Device specific Localization Initialization Load/Endurance 508 Compliance

Types of testing

Functional Testing (Positive Testing)

Use Cases and specifications Negative Testing

Error handling, like bad email and incorrect website address

Boundary Testing Max characters, or with integers -1, 0, 1, max -1, max,

max + 1 Race Conditions

Running music app, video app, and getting a phone call at the same time

Examples

Environment conditions

Rural area testing of telephony vs urban Device specific

Battery running low, graphics issues Localization

Date format, starting day, fitting in ui, punctuations

Initialization Start up of device, FTE, start up of apps versus

OTA and continual usage

Examples

Load/Endurance

Loading tons of facebook, versus keeping music app running for days on end

508 Compliance Handicap ie : blind, deaf user interfaces

Examples

From 1.0

https://wiki.mozilla.org/B2G/QA/Gaia_Test_Plan/Browser

https://wiki.mozilla.org/Releases/Firefox_25/Test_Plan

https://wiki.mozilla.org/QA/Fennec/Milestones/2.0/TestPlan:Virtual_Keyboard/Android_VKB

https://wiki.mozilla.org/QA/Fennec/24/Beta/8

How outlines/test plans can help

https://docs.google.com/spreadsheet/ccc?key

=0Arku3jleCA0UdF9ZZWtyYm9iV1NLOGRTWkQzWHFsS2c#gid=5

B2G Specific Example

Mapping out user flow

Demo

How flowcharts can help

QA conferences

They can be pricey and advertisey QA newsgroups/forums/websites QA magazines/books/articles

http://en.wikipedia.org/wiki/Software_testing http://www.amazon.com/exec/obidos/ASIN/047

1358460/softwarqaandtest

Resources

https://wiki.mozilla.org/B2G/Architecture https://github.com/sotaroikeda/firefox-diagrams/wiki/Firefox-

Diagrams

Git Hub: https://github.com/mozilla-b2g/gaia/commits/master Compare to sources.xml file :

https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-central-unagi/latest/sources.xml

Tool for searching code https://mxr.mozilla.org/

B2G Architecture