QA CLUB KIEV #18TEST MANAGEMENT AND APPROACHES

B Y O L E K S A N D R M A I D A N I U K

BACKGROUND AND EXPERIENCE

Head of Quality Assurance Solutions at Ciklum

Co-founder at QA Club Kiev

Advisory Board Member at BrainBasket Foundation

Consultant at GoIT

Co-founder at QAExperts.pro

Co-founder at TestathonUA

AGENDA

1. Test Management Dependencies

2. Test Management and Agile

3. Agile alternative to Test Management

4. Test Reports

5. Test Management: When to use what?

6. Test Management comparison

7. Q&A

TEST MANAGEMENT DEPENDENCIES

Domain-dependent (e-Commence, Medicine, Nuclear Energy)

SDLC-dependent (Waterfall, Agile) TeamStructure-dependent (2-5, 6-15.. 16-30 size

of the team) Phase-dependent (Proof of Concept, Active

Development, Maintenance) Environment-dependent (1 environment for

testing or 10, or 25?) CorporatePolicy-dependent (Microsoft, Google,

licenses purchased) Services-dependent (Manual, Automation,

Performance, Security etc.)

TEST MANAGEMENT AND AGILE

Stop use a heavy weight specialized tool for test management

Just stop.

THE AGILE ALTERNATIVE TO TEST MANAGEMENT

You need to manage the test effort for the:BacklogSource Control ManagementContinuous IntegrationAutomated Regression Tests

WHERE DO THE TEST LIVES?

High-level acceptance criteria, test ideas, Exploratory testing charters belong to the Backlog with the associated Story;

Technical artifacts including test automation and manual regression test scripts belong to the Source Control System versioned with the associated code.

WHERE DO WE CAPTURE TESTING ESTIMATES?

In Agile, we ultimately care about Done Stories.

Coded but not Tested means Not Done.

We don’t need a separate place to put estimates.

HOW DO I PRIORITIZE TESTS?

Agile teams work with a prioritized backlog. Instead of prioritizing tests, they prioritize Stories. And Stories are either Done or not.

Given that context, it does not make sense to talk about prioritizing the tests in isolation.

THERE IS NEVER ENOUGH TIME TO TEST

If the Story is important enough to code, it’s important enough to test.

If you’re working in an Agile context it is absolutely critical that everyone on the team understands this.

TRADITIONAL: WHAT ABOUT THE TEST REPORTS?

Traditional test management systems provide all kinds of reports: pass/fail statistics, execution time actuals vs estimated, planned vs executed tests, etc. Much of this information is irrelevant in an Agile context.

Continuous Integration results

95%

5%

PROPOSED: WHAT ABOUT THE TEST REPORTS?

The CI system provides the information that remains relevant: the automated test execution results. And those results should be 100% Green (passed) most of the time.

Traditional test management

80%

20%

WHAT ABOUT HISTORICAL TEST RESULTS DATA?Most teams find that the current CI reports are more interesting than the historic results. If the CI build goes Red for any reason, Agile teams stop and fix it. Thus Agile teams don’t have the same kind of progression of pass/fail ratios that traditional teams see during a synch and stabilize phase. And that means historic trends usually are not all that interesting.

However, if the team really wants to keep historic test execution results (or are compelled to do so as a matter of regulatory compliance), the test results can be stored in the source control system with the code.

REGULATORY COMPLIANCE AND TEST MANAGEMENTIn that context, specialized test management solutions may be the defacto standard, but they’re not the best answer. If I’m working on a system where we have to be clear, concrete, and explicit about requirements, tests, and execution results, then I would much rather do Acceptance Test Driven Development.

ATDD provides the added value of executable requirements. Instead of the tests and requirements just saying what the system should do, they can be executed to demonstrate that it does.

ATDD REQUIRES EFFORT

Certainly, doing ATDD requires effort. But so does maintaining a separate test management system and all the corresponding traceability matrices and overhead documentation.

TEST MANAGEMENT SYSTEMS: COMPARISON

GuRock TestRail

HP Quality Center

MicroFocus QA Director

TestLodge PassMark TestLog

Zephyr Oracle Test Manager

IBM Rational Quality Manager

QASymphony qTest

Tricentis Tosca Testsuite

GoogleDocs TechExcel DevTest

SmartBear QaComplete

ApTest Manager

Bugzilla Testopia

SpiraTest MicroFocus SilkCentral

Testuff QaTraq Professional

TestLink

TEST MANAGEMENT SHOULD HELP YOU

Store test scripts in one place and CRUDs

Requirements Management

Easy and quick access to the test scripts

Calculate and show test coverage

Easy create test suites (e.g. Smoke, Regression)

Support Versioning

Release Management

Dashboards and Reporting

TEST MANAGEMENT: WHAT USE WHEN?

GoogleDocs – 1-5 QA team size projects, Agile, Active Development, No Security and Corporate Policy, Any non-critical domain for life

TestRail – 5+ QA team size, Agile, Active Development and Maintenance, Security Policy, Automation is implemented in the projectDomain-dependent (e-Commence, Medicine, Nuclear Energy)

SDLC-dependent (Waterfall, Agile)TeamStructure-dependent (2-5, 6-15.., 16-30 size of the team)Phase-dependent (Proof of Concept, Active Development, Maintenance) Environment-dependent (1 environment for testing or 10 or 25?)CorporatePolicy-dependent (Microsoft, Google, licenses purchased)Services-dependent (Manual, Automation, Performance, Security etc)

TEST MANAGEMENT SYSTEM AS A RULE

Send them the URL to this presentation. Ask them to read it.

Then ask them what additional value they’re getting out a test management system that they wouldn’t get from leveraging SCM, CI, the Backlog, and the automated regression tests.

SO, HAVE I CONVINCED YOU?

If not, please tell me why