Before You Dive Into Testingby Anna Lucovsky

About Your Seminar Host – Anna Lucovsky

• QA Manager with 20 years of experience in building of successful and

methodological QA teams and management of automatic and manual testing

in diverse fields of software industry.

• I specialize in – Building of QA teams from scratch

– Automatic testing (C# CodedUI, QTP, TestComplete)

– Implementation of testing methodologies and test management tools

– Improvement of QA processes

– Management of large QA teams

• I am also the Asia Champion and World Silver Medalist in Software Testing

World Cup 2016

• LinkedIn: https://www.linkedin.com/in/anna-lucovsky-3b94084/

First Things First: Assess Your Testing Project

What kind of project is this:

• Long/Short Term? What is the project scope?

• What condition is the product currently in?

• What technologies /environments are involved, is the team at ease

with them?

• What resources does the team have: testers / tools / time / existing

tests / automation ?

• Do you directly manage the testers? Is a team distributed?

• What are your interfaces – developers, system analysts, customers,

DevOps

• What is the development model: Waterfall / Agile / FDD, any mix of

those?

Test Basis – The Basis of Your Assessment

• ‘Test Basis’ - all project documents that could serve as the basis for

test design:

– system design document

– user requirements

– change requests

– user presentations

– ERD document

– infrastructure diagram

– release documents

Take into consideration that these documents are subject to change, so test cases

would need to be adjusted subsequently

• Interviews with system analysts / developers / customers / users

• Studying the system via free usage / exploratory testing

Last But Not Least - Your Test Project Stakeholders

• Who are your stakeholders – your boss / another team / customers / your own

team ?

• What is the scope of your influence – what can you realistically change,

• who listens to you, specifically the stakeholders?

QA Team – Your main resource• Is this project:

– Just more of the same

– Something different

• Do the team skills match the project requirements?

– Junior / Experienced team

– Test design abilities

– Quality of test execution

– Automation / NFT

– Is the team mix correct?

• Is the team motivated?

• Is any change management required?

– Team skillset

– Team members

– Working processes / tools

Assess Your Product Quality Quickly – Use Case

• Quick assessment is useful when you are

– Starting with an existing company product that is new for you

– Starting a new job

– Participating in a testing contest

• Let’s say you have assessed your project, but you still have to

understand the SUT current condition because you need to know your

starting point:

– Use a mind map Small: Large:

– Spend 2-3 days of exploratory testing going over all major aspects of the product

– Prepare a result map / table marking each aspect quality status

– Convey the information to the stakeholders in a visual way

– this might get you some buy-in in case the results speak

– for themselves and it’s your job to make them to do so

Especially if the result is like this

Smart Test Executionby Anna Lucovsky

Building the Testing Tree

• It is highly recommended to use a test management tool that allows

building testing trees for convenient execution

• Trees can be build with

– division according to functionality sections in the system design document

– division according to the system screens

– division according to the requirements tree and change requests

– division according to versions and features comprising each version

• Do not forget to add to the tree:

– Non-functional tests

– Sanity tests

– Regression tests

• So now you have a good test suite (of 10000 test cases), but what about

the test execution?

• The answer is => Risk Management

Risk Management – What problems does it address?

• What problems does risk management address?

– Careless tester’s problem – you have all your tests, you start

testing and stop once the release date is reached => important

tests might not be executed

– Thorough tester’s problem – you introduce a test management tool,

design tests with decent coverage, add tests for every new version

and then don’t have enough time to finish executing them, since

your team does not grow exponentially. The management expect

the tests to take X time per version (the less the better), but your

tests amount only increases over time.

Risk and Its Components

• RISK => The possibility that something unpleasant will happen• Our life is full of risks and we manage them every day, starting with choosing a Waze route in the

morning and ending with going to the gym late in the evening

• Risk Components:

Probability (low-high) – how probable it is that the failure will occur - at every customer site, at

50% of the customer sites, at 2% of the sites.

vs

Impact (low-high) – functionality impaired, money loss, e.g. customers leave or stop paying

maintenance, employees need to be sent onsite, reputation damages, costs of fixes. Specific

product market is of importance here, consider a credit line financial software vs WhatsApp

• Good enough software is built on risk management assumptions and it has

– enough benefits

– no critical problems

– more benefits than problems

– And improving it will cause more harm than good

Risk Management – Benefits

• Risk management is the key to getting budget, time and resources (remember the stakeholders!)

– There is never enough time for all the tests

– We need a quality product, on schedule, within budget with a given staff

– When risks are clear to the stakeholders, they will more likely support the

team advocating for quality

• With risk management the responsibility is shared

– If testers are defined as gatekeepers of quality, they take on the

responsibility of external risks. Testers are like a thermometer => they give

information about what was tested, what risks were addressed and what

risks are still there without “feelings”- based evaluations of the SUT

forehead

– After all, it is the management that is to make the decision of Go/No Go

Tips for Risk-Based Test Project Management• There must be the ‘Risk Classification’ section in your STP

• Risk categories: Contractual, Functional, Technical, Technological, Customer Hardware, Customer

Software, Human Resources, Third Party

• Risks table would include Risk Description | Consequences | Mitigation plan (preventive actions)

• Risk description- What could go wrong, generate problems, be expensive from the

shareholders’ point of view:

– “QA do not have time to test” vs “Implementation failure with a critical customer if X is not tested”

– “We have run 800 tests, 200 still not run” vs “Registration and booking are ok, messaging not yet

tested, 20% of issues found in calling” – maybe messaging is the critical one for the customer?

• STP has to be signed upon before the tests start => responsibility division and

delegation

• STR should refer to the same sections and be prepared BEFORE the last test cycle

– Post-mortems don’t help this release, maybe the next one.

– The management has to make a fact-based decision with risks clear and have time to fix whatever

needs to be fixed before the release date.

– Alternatively, delay the release, but not in the last minute. If the current release doesn’t have enough

benefits and its problems are too serious, testing and fixing should continue.

Tips for Risk-Based Test Execution

• Report bugs on specs – risks are born there

• Look for modules impacted between the latest versions

• The more developers needed to fix a specific issue – the more risky it is

• Evaluate severity, priority, module and amount of issues previously found when choosing

your test suite

• Analyze bugs reported by the customers and concentrate your efforts there

Tools Usage Tips

– Use test priorities for every test scenario in regression tests. After each release adjust those priorities.

– Change module/component once the issue is fixed to follow up on the real problematic area afterwards

– Tests for new features = critical priority. BUT no risk – no test, an effective test case has to do with high

risk

– Use priorities for requirements. Cover the requirements with tests in the test management tool. Use the

requirements priorities as guidelines to choosing your tests suite.

– Define priorities to be run for every type of release (major / minor / update)

Exploratory Adventure – Risky or Not?• At times it is tempting to default to exploratory testing

– When there is no time for test design

– When the test basis is problematic

– When there is no time for junior testers training

– When there is no time in general

• Disadvantages:

– Renting vs Buying – you invest your resource, but are left with no asset in the long-run

– Important areas may be overlooked as testing is ad-hoc

• Exploratory testing is an art and should be used with caution by experienced testers who know

the system under test very well. Still, change favors the prepared mind.

• Exploratory tests should not replace existing tests covering the risky areas, but be performed

in addition to those or used for quickly assessing the product quality

• Exploratory testing is also used upon SUT with unstable and changing functionality

• In order to minimize the disadvantages, it is important to roughly document steps that were

performed

• Scenarios that expose problems in the SUT are to be added to the existing test suite even at

the expense of similar cases that have not found problems over time

Automation Will Help, But How?by Anna Lucovsky

It was decided to automate our manual tests…Before you write a single code line, consider several factors:

• There are more failed automation projects than the successful ones

due to many reasons:

– Using record-and-play

– GUI Automation is slow

– Automation is unstable

– Automation has low ROI

– Automation is often non-scalable and hard to maintain due to architecture problems

• What are the stakeholders expectations:

– Reducing the manual QA staff

– Using automation as a part of continuous integration flow

– Eliminating human mistakes

– Creating incentive for the team

– Automating the repetitive and boring parts in order to employ the QA staff in more

creative areas

If the decision to automate is for the right reason…

• Evaluate your scope:

– GUI Tests / Non-GUI tests (API)

– Quality of the existing test cases – they have to be detailed

• Evaluate your team for programming skills:

– Design and architecture vs Using building blocks

– OOP knowledge

– Knowledge of your SUT programming language

• Evaluate your SUT

– Technology

– Existing tools supporting this technology (GUI, API)

– Availability of the SUT code for your possible needs

• All of the above are important factors for choosing the

automation tool (along with its price, of course )

Critical Part - Choosing the RIGHT tests to automate

• Even if a test is boring and repetitive, it does not mean it

has to be automated right away, it also has to be relatively

easy to automate and show some payoff immediately (for

the stakeholders!)

• Tests that are executed the most are the immediate

candidates in case they are not too technically tricky for

automation

• API tests are excellent candidates once their interfaces are

finalized

• Usually the order to start with would be as follows:

– API tests

– Smoke tests

– Critical sanity tests

Questions?