Product QA – A Test engineering perspective

Ensuring products are tested the way they will be used

Product QA: Why it is Tough

1. Product is more than software• Software application• Platform and variations• Documentation: online help, manuals, tech briefs, examples, training• Add-ins and third party product integrations• Installation experience• Handling upgrades and updates

2. Quality is more than lack of bugs• Does it solve the problem when it works?• Is it practical to use? Is it designed for efficient use?• Does it keep working or works only under ideal conditions?• Is it economical to build, maintain and respond rapidly to

customer/user needs?

Role of QA in Product Company

• QA is everything you do to minimize risk of failure and promote excellence

• Help producing products that satisfy target customer base

• Broad functions of QA– Risk management: release planning, release approval– Bring customer involvement: early release programs– Act as skillful developers: mimic tough customer!– Process definition and improvement– Inspection and testing– Experience-based improvement: contribute to product

definition, many finer points• Should play an important role throughout The Life of a

Product

Hawk-Eye Methodology-Precision Delivered

5 4

3

21

6

Planning Design

Coding

TestingRelease

Post-Release

Process Definition

Process Improvement

QA Planning

Risk Assessment

QA Certification

Release Inspection

Test Development

Test Suite Design

Product DefinitionInputs

Early ReleaseProgram

Test Execution

Code Review

Design Review

Our offerings

Services

Business landscape

Non-testing Functions

QA Planning

• What QA is required to do• QA procedures, expectations from other

teams• Effort estimation and team planning• Resource planning, tools and software

procurement

Test Suite Design

• Design the tests in parallel to software design• Tests need organization into independent test

suites• Build the tests before you build the software• If you find a bug, write a test that proves the

bug exists, fix the bug and then rerun the test to prove it is fixed

Test Suite Design Review

• Goal: Clarification and accept/reject design decisions

• Starts with a presentation of the design for the component/module to a selected panel of developers

• Peer review is commonly used, with most relevant engineers from other parts of the product included

• Discussion and coordination of comments incorporation

Check-in Review & Code Walkthrough

• Check-in review done with peers of the dev team– Including new features and bug fixes

• Code walkthrough by dev across streams, including QA– review meeting with prepared participants– Sample source files are selected for review

• Focus is on finding defects, before they slip through to testing phases– Lower costs by catching them early

Early Release Programs

• Need to ensure smooth porting of current customers

• Test ISV applications available internally• Public program to increase the test base– Alpha and Beta testing

Release Readiness

• QA evaluates quality, independent of developers– Developers should not determine how much

quality is sufficient

• QA acts as internal customers and verifies release readiness

Our Test ClassificationBased on Class Description

How a test is written

Black box Test does not use knowledge of product code and design

White box Test uses product internals, even calls unexposed functions

When a test is being run

Sanity testing Basic set of tests run after every nightly build, to ensure that nothing is broken

Alpha testing Testing done on code release to QA

Beta testing Testing done with the help of external partners, early release customers, etc.

Final testing Final set of testing done by QA to approve GA release

What purpose the test is serving

Functional Verify that functional requirements of the product are as per specifications

Compliance Test suite for compliance to industry standards

Load Verify performance and scalability of the product; also produce data on installation requirements

Reliability Negative testing to ensure that the product functions gracefully under unexpected conditions

Usability Testing the product for practicality, efficiency of use and ease of learning of the product interface

Functional Testing

• Key aspects to cover (they are endless!)– Correctness– Interoperability– Compatibility– Compliance– Security– …

• Think of permutations– Functions vs. input data vs. states– Product vs. platforms– System vs. external factors– Testing vs. versions of product

Compliance Testing

• Emphasis on when product is expected to be declared “compliant” with Industry Standard (such as J2EE™)

• Compliance test suite (TCK for J2EE) might be made available

• Compliance to standards does not mean the product is functionally tested for use

Load Testing

• Performance testing– Performance that can be achieved on a configuration setup– Deterioration with load– Variations across platforms– Writing a sizing guide with suggested deployment

configurations– Generating a near real-world load

• Scalability testing– How does the system scale with increase in platform capacity– Fail over and load balancing under clustered configurations

• Benchmarking– Comparative performance testing and analysis with competitor

products– Public benchmarking, such as SPECjAppserver™ or ECperf™

Performance Reports

Reliability Testing

• Aspects to test– Maturity: how the product expects erroneous user handling

– Fault-tolerance: Ability to perform under faulty hardware/platform conditions with degraded performance but near full functionality

– Integrity: How the system cannot be broken by malicious users

– Recoverability: ability of the product to recover to a consistent state when power outage, network fault or other disaster happens

– Safety: protection of user data, backup, logging, etc.

Usability Testing

• Aspects to cover– Understandability: clarity of interface, intuitiveness– Learnability: ease of learning, matches user’s current

understanding of other products– Consistency: across versions, all across the product

• Its not only about GUI, its also about file formats, XML, alternatives such as command line tools

• We normally work with external consultants who are specialized in usability engineering

Other Aspects for Testing

• Some of these aspects may not be important to end-customers, but very important for engineering

• Key aspects include– Efficiency

• Storage• Processing

– Maintainability• Code organization, branching• Modularity for ease of changes, maintenance, refactoring

– Portability• Platform support• Reusability of code

Making Testing Easier

• Document the design

• Use internal error checking

• Test units before integrating them

• Tell the tester what’s new or flaky

• Test the build yourself, first

• Evolve product in functional layers

Tools and Infrastructure

Test Automation

• Test automation helps repeatability of testing• Useful test automation is a major software project• Engineers are able to detect more problems, more accurately

Industry standard skills

In-House Tools

BrighTest Automation tool developed in-house Leverages Selenium and Java API based approach Easily customizable to suite specific project

requirements Ability to store reports in our own format

for future use. Easy to install. Easily redefined parameters Customized Alerts and Reports. Weekly and Nightly build integrations Applicable to testing of web based

applications and products as well as Social Platforms

Pilot usage on for a couple of happy customer engagements.

Extensively used to test Pramati’s Enterprise 2.0 product Qontext.

Test Case Management System

J2EE™ Compliance Testing• J2EE CTS Kit

– From Sun

• Standards Verification• Automated Harness

Test Management Framework• Augment CTS

– For Complex scenarios

• Built on Studio Framework– Homegrown

• Ease of writing new tests• Automated Discovery of new tests• Extensive tests metadata

– Compose new plans/suites

• Integrated into the Build System– For automated post-build testing

Reliability Testing

Blade- Distributed Stress Testing System

Thank you.

info@imaginea.com