Upload
karen-n-johnson
View
436
Download
1
Tags:
Embed Size (px)
Citation preview
Do you find yourself needing to “check” something in production immediately after go live? Do you
need to investigate how a page or a campaign looks in multiple browsers or on a mobile device?
Without realizing it, you may be performing the role of a software tester in addition to your primary
role. What is a software tester? In this session Karen offers 1) a brief overview of the software
testing profession, 2) a summary of different types of testing, 3) a list of resources to learning more
about testing. If testing is part of what you or your team need to juggle to “make it happen” come
attend this session.
About this talk …
Testing Explained © Karen N. Johnson 2014
• Software Test Consultant • Published Author (Beautiful Testing) • Co-founder of WREST: Workshop on Regulated Software Testing • Website: www.karennicolejohnson.com or www.karennjohnson.com • Twitter: @karennjohnson
Karen N. Johnson
Testing Explained © Karen N. Johnson 2014
Can you make sure all the parts to the campaign “look good?”
Testing Explained © Karen N. Johnson 2014
When you already have a fulltime job and a profession?
But how do you immerse yourself in another world?
Testing Explained © Karen N. Johnson 2014
Waterfall & Agile Development
Testing Explained © Karen N. Johnson 2014
¨ There are many different ways to develop and test software.
¨ In past years, the Waterfall method was the most common software development methodology.
¨ In current times, an Agile development process is now more common.
Agile Development
Testing Explained © Karen N. Johnson 2014
The Agile Manifesto ¨ We are uncovering better ways of developing software by
doing it and helping others do it. Through this work we have come to value: ¤ Individuals and interactions over Processes and tools ¤ Working software over Comprehensive documentation ¤ Customer collaboration over Contract negotiation ¤ Responding to change over Following a plan
Acronyms: we love them! ¨ ATDD: Acceptance Test-Driven Development ¨ BDD: Behavior-driven development ¨ TDD: Test-driven development
Testing Explained © Karen N. Johnson 2014
Checking vs. Testing
Testing Explained © Karen N. Johnson 2014
¨ Within the profession of testing, there is a long-time discussion known as “checking vs. testing.”
¨ Often less experienced people in testing and/or people who are not professional testers “test” in such a manner that they are “checking” software works.
¨ In these situations, people are not exploring, they are not looking for issues, they are checking to confirm software works versus looking to expose or discover issues.
Checking vs. Testing
Testing Explained © Karen N. Johnson 2014
¨ If you never read an article on testing, if you never read a blog post on checking vs. testing, learn this one concept:
Know when you are checking to confirm software is working vs. testing with the intention of discovering possible issues.
¨ Do not confuse the two approaches (checking vs. testing) as meaning the same thing.
¨ There is nothing wrong with “checking” software or “confirming” that software works but this is not the same as intentionally looking to discover defects.
Tested = Checked + Explored
Testing Explained © Karen N. Johnson 2014
In Elisabeth Hendrickson’s book, Explore It!, she states that neither checking nor
exploring is enough.
“You’re not done testing until you’ve checked that the software meets expectations
and you’ve explored whether there are additional risks. A comprehensive test
strategy incorporates both approaches.”
Exploratory vs. Scripted Testing
Testing Explained © Karen N. Johnson 2014
¨ Within the profession of testing, there is a long-time discussion on the use of scripted vs. exploratory testing.
¨ Let’s look at each format.
Scripted testing: a definition
Testing Explained © Karen N. Johnson 2014
Scripted testing follows a path that is written by the tester
themselves or someone else. The script includes test steps that are
documented. It is expected that the tester will follow the steps as
detailed. The tester’s job in a scripted environment is to follow
each step and report findings (if any).
Exploratory testing: a definition
Testing Explained © Karen N. Johnson 2014
James Bach’s definition: “Exploratory testing is simultaneous learning, test design, and test execution.” This style of testing requires that your brain be fully engaged at all times. Cem Kaner’s definition: “Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”
Exploratory & Scripted Testing
Testing Explained © Karen N. Johnson 2014
¨ As it turns out, you don’t have to choose one form over another. You could use a blended approach.
Schools of Software Testing
Testing Explained © Karen N. Johnson 2014
¨ Around 2003, several people in the testing field constructed a way to explain different core approaches to software testing. The construct was referred to as the schools of testing.
¨ The schools of testing have been greatly debated over the years. Some people have been so opposed to the construct of the schools concept, they have refused to even discuss the construct or acknowledge the schools as a way of discussing differing views.
Schools of Software Testing
Testing Explained © Karen N. Johnson 2014
¨ In 2003, the schools were defined by Bret Pettichord as:
¤ Analytical School ¤ Factory School ¤ Quality Assurance ¤ Context-Driven School
Schools of Software Testing
Testing Explained © Karen N. Johnson 2014
¨ You don’t have to be involved in the debate.
¨ You don’t have to choose a school.
¨ But it can be helpful to know this “argument” exists in the testing profession.
¨ It can be helpful to know what school a tester identifies with.
The Seven Basic Principles of the Context-Driven School
1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.
3. People, working together, are the most important part of any project’s context.
4. Projects unfold over time in ways that are often not predictable.
5. The product is a solution. If the problem isn’t solved, the product doesn’t work.
6. Good software testing is a challenging intellectual process.
7. Only through judgment and skill, exercised cooperatively throughout the entire
project, are we able to do the right things at the right times to effectively test
our products.
The Original Quadrants of Testing
Testing Explained © Karen N. Johnson 2014
From Brian Marick’s blog
The Original Quadrants of Testing
Testing Explained © Karen N. Johnson 2014
Support programming vs. critique product
¤ “When it comes to supporting programming, tests are mainly about preparing and reassuring.”
¤ “On the other side of the division, tests are about uncovering prior mistakes and omissions.”
Agile Testing Quadrants
Testing Explained © Karen N. Johnson 2014
From the book, Agile Testing by Lisa Crispin and Janet Gregory
The Quadrants are being reconsidered
Testing Explained © Karen N. Johnson 2014
¨ Gojko Adzic http://gojko.net/2013/10/21/lets-break-the-agile-testing-quadrants/
¨ Markus Gärtner http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/
¨ Elisabeth Hendrickson http://www.slideshare.net/ehendrickson/the-thinking-tester-evolved?ref=http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/
Certification & Standards
Testing Explained © Karen N. Johnson 2014
¨ There are several certification programs available. ¨ There are several boards such as IEEE and ISO, that
devise standards around testing. ¨ And there are testers who are in agreement with
certifications and/or standards. ¨ And there are testers who are strongly opposed.
Types of Testing
Testing Explained © Karen N. Johnson 2014
¨ There are numerous types of testing. Each type has a focal point and serves a different purpose. For example, user acceptance testing vs. performance testing.
¨ In some cases, a type of testing will be more likely to occur at different times during the development process. For example, unit testing vs. system testing.
Black Box Testing
Testing Explained © Karen N. Johnson 2014
¨ Black box testing is one of the most common terms used in testing.
¨ The basic premise is the person testing does not know how the code was written and does not have insight inside the “box” or the system.
¨ Black box testing is testing without a technical understanding of how a program was written. This is why the “box” (or the item being tested) is “black” (because we don’t know what is inside.)
White Box or Glass Box Testing
Testing Explained © Karen N. Johnson 2014
¨ The opposite of black box testing. White box testing means the tester has an inner understanding of how the program was written, this form of testing allows for more targeted technical testing.
¨ Much of the time, black box testing is the type of testing being used.
Testing Explained © Karen N. Johnson 2014
An example … Suppose I am testing the credit card entry part of an ecommerce website. Black box testing: • Try using an invalid credit card White box testing: • Review the code • Learn after three invalid credit card entries, the account is locked • Enter three invalid attempts check error handling and messaging With black box testing, I might try an invalid credit card once, see an error message and not continue to test because I don’t know about the three attempt condition.
Unit Testing
Testing Explained © Karen N. Johnson 2014
Martin Fowler’s description:
“The common properties of unit tests — small scope, done by the programmer
herself, and fast — mean that they can be run very frequently when
programming.”
Integration Testing
Testing Explained © Karen N. Johnson 2014
Paul Jorgensen’s definition:
Integration is the phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing.
Lee Copeland’s definition:
In integration we assemble units together into subsystems and finally into systems. It is possible for units to function perfectly in isolation but to fail when integrated.
System Testing
Testing Explained © Karen N. Johnson 2014
Lee Copeland’s definition:
A system consists of all of the software (and possibly hardware, user manuals,
training materials, etc.) that make up the product delivered to the customer.
System testing focuses on defects that arise at this highest level of integration.
Typically system testing includes many types of testing: functionality, usability,
security, internationalization and localization, reliability and availability,
capacity, performance, backup and recovery, portability, and many more.
UAT or User Acceptance Testing
Testing Explained © Karen N. Johnson 2014
Lee Copeland’s definition:
Acceptance testing is defined as that testing, which when completed
successfully, will result in the customer accepting the software and giving us
their money.
Testing Explained © Karen N. Johnson 2014
An example … Suppose I am testing the purchasing process of an ecommerce website. Unit testing: Build a unit test for credit card expiration date. Integration testing: When the code for credit card processing is integrated with the code that handlings shipping information, I can build an integration test. At the system level, I might build testing from the start of product selection through the entire place an order, credit card and shipping process; a test that brings all the pieces together.
Regression Testing
Testing Explained © Karen N. Johnson 2014
¨ A common form of testing; used to retest to ensure changes in one part of a program have not introduced issues in another part of a program.
The “ilities” of testing ¨ Software testers commonly refer to a series of terms as the
“ilities” of testing.
¨ The “ilities” are often a challenge to test but are often the same items that customers are often most concerned about.
Testing Explained © Karen N. Johnson 2014
The “ilities” of testing ¨ Capability ¨ Reliability ¨ Usability ¨ Security ¨ Scalability ¨ Performance ¨ Installability
¨ Compatibility ¨ Supportability ¨ Testability ¨ Maintainability ¨ Portability ¨ Localizability
Testing Explained © Karen N. Johnson 2014 Source: James Bach, Rapid Software Testing
More types of testing ¨ Stress ¨ Multi-user ¨ Cookies ¨ Compatibility ¨ Developer ¨ Unit ¨ Integration ¨ System ¨ User Acceptance ¨ Installation ¨ API/web services
¨ Black box ¨ White or glass box ¨ Gray box ¨ Functional ¨ Automated ¨ Regression ¨ Security ¨ Data ¨ Exploratory ¨ Scripted ¨ Performance
Testing Explained © Karen N. Johnson 2014
But if you only learn one question to ask, let the question be …
What if?
Testing Explained © Karen N. Johnson 2014
• What if? Is a challenge question.
• What if? Is the type of question that begs for investigation and
discovery.
• What if? Does not require technical training or knowledge.
Testing Explained © Karen N. Johnson 2014
During storyboarding and
design is a great time to
use: What if?
Testing Explained © Karen N. Johnson 2014
“State transitions provide a
Bermuda-Triangle-like moment in time
where “things” can go wrong for a user
or system. And once a user or system
is in a bad state, they can be awkwardly
trapped or lost.”
Testing Explained © Karen N. Johnson 2014
State transition
“When a system must remember something about what has happened
before or when valid and invalid orders of operations exist, state-
transition diagrams are excellent tools to record this information.”
Source: Lee Copeland. A Practitioner's Guide to Software Test Design
Testing Explained © Karen N. Johnson 2014
1. You will never find everything.
Testing Explained © Karen N. Johnson 2014
The number of combinations and possibilities is greater than what you can test.
There are more ways to test than there are hours in the day.
2. You will never have enough time.
Testing Explained © Karen N. Johnson 2014
If you have more ideas for testing than time, choose what you will test wisely and get help from your team.
3. You cannot do it all alone.
Testing Explained © Karen N. Johnson 2014
Build testing into the job regardless of who in your agency gets it done. Do not allow testing to be an optional activity.
4. Don’t allow testing to be an extra bill to the client.
Testing Explained © Karen N. Johnson 2014
Note-taking, mind mapping, voice recording: whatever it takes, don’t lose a great idea.
5. Capture your test ideas.
Testing Explained © Karen N. Johnson 2014
6. To find issues, you have to test beyond the simple path.
Testing Explained © Karen N. Johnson 2014
Email: [email protected] Twitter: @karennjohnson
Karen N. Johnson
Testing Explained © Karen N. Johnson 2014
Checking vs. Testing
Testing Explained © Karen N. Johnson 2014
¨ James Bach, Blog: Testing and Checking Refined, http://www.satisfice.com/blog/archives/856
¨ Michael Bolton, Blog: Testing vs. Checking
¨ http://www.developsense.com/blog/2009/08/testing-vs-checking/
¨ Elisabeth Hendrickson, Explore It!
¨ Ministry of Testing, http://www.ministryoftesting.com/2012/07/mindmaptesting-and-checking/
Exploratory vs. Scripted Testing
Testing Explained © Karen N. Johnson 2014
¨ James Bach, paper: Exploratory Testing Explained (10pp),http://www.satisfice.com/articles/et-article.pdf#page=1&zoom=auto,-55,792
¨ James Bach, video, Understanding exploratory testing (6 minute video), https://www.youtube.com/watch?v=I-ItEKt_N_s
¨ James Bach, Session-Based Test Management (1 page), http://www.satisfice.com/sbtm/
¨ James Bach, What is Exploratory Testing? And How it Differs from Scripted Testing
¨ http://www.satisfice.com/articles/what_is_et.shtml
¨ Cem Kaner, slide set (180pp) A Tutorial in Exploratory Testing, http://www.kaner.com/pdfs/QAIExploring.pdf
¨ Michael Bolton, Resources on Exploratory Testing, Metrics, and Other Stuff, http://www.developsense.com/resources.html
¨ Gregory Mooney, Exploratory vs. Scripted Testing: One or the Other, or Both?, http://blog.smartbear.com/testing/exploratory-vs-scripted-testing-one-or-the-other-or-both/
Schools of Testing
Testing Explained © Karen N. Johnson 2014
¨ Bret Pettichord, Schools of Software Testinghttps://www.prismnet.com/~wazmo/papers/four_schools.pdf
¨ Bret Pettichord, http://www.testingeducation.org/conference/wtst_pettichord_FSofST2.pdf
¨ Cem Kaner, Blog: Schools of software testing, http://kaner.com/?p=15
¨ Context-driven testing, http://context-driven-testing.com/
¨ James Bach, http://www.satisfice.com/blog/archives/134
Quadrants of Testing
Testing Explained © Karen N. Johnson 2014
¨ Brian Marick blog post: http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2
¨ Lisa Crispin, Janet Gregory, Agile Testing (book) ¨ Gojko Adzic http://gojko.net/2013/10/21/lets-break-the-agile-testing-quadrants/ ¨ Markus Gärtner http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/ ¨ Elisabeth Hendrickson http://www.slideshare.net/ehendrickson/the-thinking-tester-evolved?ref=http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/
Rapid Testing ¨ Rapid testing is a method, a series of techniques and a class,
all in one taught by James Bach.
¨ James is arguably the most known tester across the globe.
¨ James posts the slides to his Rapid Software Testing class on his website. www.satisfice.com
Testing Explained © Karen N. Johnson 2014