Testing 29/Apr/2009

© 2008 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

Testing 29/Apr/2009Petr Panuš[email protected] Manager & Offshore Team Manager, SOA Center HP Software R&D

Agenda1. Why is SW testing necessary2. Testing principles3. Testing design techniques

Why is SW testing necessary• SW does not always work as we wish• Defects in SW can cause harm

−Not all defects in SW result in a failure• Testing reduces the probability of

undiscovered defects remaining in the software

• Testing gives us the confidence about the tested SW

Testing Principles

Testing Principles• ISTQB defines 7 Testing Principles

1. Testing shows presence of defects2. Exhaustive testing is impossible3. Early testing4. Defect clustering5. Pesticide paradox6. Testing is context dependent7. Absence-of-errors fallacy

Testing shows presence of defects• Tests show the presence not the absence

of defects.• Testing can’t prove that SW is defect free• Testing reduces the probability of

undiscovered defects remaining in the software

• However, the fact that no defects were found, is not a proof of correctness.

Exhaustive testing is impossible• It is not cost-effective

−Defects are not of an equal risk• We need to prioritize our tests

−Use risk analysis and priorities to focus testing efforts

−Use appropriate testing techniques to meet the risk, giving us confidence in what testing has been performed

• Exhaustive testing is equivalent to Halting problem−There is no algorithm that can for an arbitrary

program and its input validates if the program works (with the given input) fine or not

Early testing• In SW development, what can be tested

−Requirement specification−Use-case Analysis document−Technical Analysis document−Design document−Functional Implementation−Performance−Usability

• What can be also tested−Test Description−Documentation

Price of potential defect fix increases

Example

Example – Early testing• Defect in H310: JBoss 4.3.0 not supported

on Win 2008 (although Hermes 310 PRD requires this combination to be supported)−7 other duplicates reported−Involving 4 people from QA−9 people from DEV

Defect clustering• Defects get clustered since different

reasons−A component might be more complex than

others−A component might be developed by a less

experienced developer−A component might be developed by a less

careful developer−A component might have a poorer specification−A component might need more refactoring

(introducing more other defects)• Another explanation:

http://parlezuml.com/blog/?postid=242

http://parlezuml.com/blog/?postid=242

Pesticide paradox• Old tests will eventually stop finding new

defects−Defects create immunity to these tests

• To find defects, new tests need to be introduced−Or old tests refactored

• Conclusion: Regression tests do not find majority of new defects

Testing is context dependent• Different kinds of tests are run in different

periods−Defect testing

• To discover faults or defects in the software where its behavior is incorrect or not in conformance with its specification;

• A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system.

−Validation testing• To demonstrate to the developer and the system

customer that the software meets its requirements;• A successful test shows that the system operates as

intended.−We start with defect testing and perform

validation testing later in the SW development process.

Testing is context dependent II.• We can also test differently because of

−The type of industry (safety-critical, business, nuclear)

−Number of customers (impact the SW makes)• One customer patch• New version of SW for potentially many customers

Absence-of-errors fallacy

Absence-of-errors fallacy• We test a system to see if it meets the

documented requirements• We find and fix defects to demonstrate that

the system meets these specifications• Finding and fixing defects does not help if

the system is unusable and does not fulfill the user’s needs and expectations

Testing Techniques

Testing design techniques• Black-box techniques

−Equivalence partitioning−Boundary value analysis−Decision tables−State transitions

• White-box techniques−Statement testing and coverage−Decision testing and coverage−Other structural-based coverage

No details about the product implementation are known

Tester knows how the tested requirement is implemented

Equivalence partitioning• Tax Calculator based on annual income• Input – integral number

• Partitions of equivalenceInvalid part Valid part

#1Valid part #2 Valid part #3

… -2, -1 0, 1, … 19999 20000, … 50000

50001, …

Income Tax< $20000 No tax$20000 - $50000 15%> $50000 20%

Equivalence partitioning II• Split input into partitions

−Valid partitions−Invalid partitions

• Have one test-case for each partition−Even invalid ones

• One more example: Enter number of a month

Invalid partition 1

Valid partition Invalid partition 2

… -2, -1, 0 1, 2, … 12 13, 14, …

Boundary value analysis• Similar as Equivalence partitioning but

−Test boundary values• N-BVA (BVA = 0-BVA)

−Test N boundary values −BV, BV+1, BV-1, … BV+N, BV-N

Decision tables

Decision tables - Example• Example – HP SOA Systinet Licensing

Framework• HP SOA Systinet consists of

−Core, Reporting, Lifecycle, etc.−Policy Manager (optional)−Contract Manager (optional)

• HP SOA Systinet License can limit−Number of users−Number of days

• Default license included in the installer −Policy & Contract Manager included−Limited to 60 days, unlimited to number of users

Decision tables• Example – Testing license application

Conditions

Actions


Conditions

Default PM on CM on # days limited # users limited

Actions

Try PM

Try CM

Login after expirationLogin more users than allowed


Conditions

Default Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N N N N N N N N N N N N N N N NPM on Y Y Y Y Y Y Y Y N N N N N N N N Y Y Y Y Y Y Y Y N N N N N N N NCM on Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N# days limited Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N# users limited

Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N

Actions

Try PM

Try CM

Login after expirationLogin more users than allowed


Conditions



Actions

Try PM X

Try CM X

Login after expiration

X

Login more users than allowed

Default license


Conditions



Actions

Try PM X

Try CM X


X


Default license


Conditions



Actions

Try PM X X

Try CM X X


X X


X Default license Visibilit

y Edition


Conditions



Actions

Try PM X X X

Try CM X X X


X X


X Default license

UnlimitedStandardEdition

Visibility

Edition


Conditions



Actions

Try PM X X X X

Try CM X X X X


X X X


X X Default license

UnlimitedStandardEdition

LimitedStandardEdition

Visibility

Edition

Decision tables II• Catches all possible combinations• Helps to analyze the situation and to decide

−Which use-cases must be tested−Which use-cases does not make sense−Requires knowledge of the business

environment• Helps to prioritize the use-cases

State transitions• Example – Testing ‘Contract Request

Lifecycle’Create Request

Accept Request

Reject Request Revoke

Request

Delete Request

State transitions II• Transition diagram shows valid transitions

−Does not show invalid (that should be also tested)

• Good for testing use-cases that are possible to be described by transitions between states

• The scenarios may contain test-cases for each−State−Transition−Event that triggers state change (transition)−Action that may result from those transitions

White-box techniques

Statement testing and coverage

Decision testing and coverage

Other structure-based techniques• Condition Coverage

−All single boolean conditions in a single statement must be evaluated to true and false

−Full condition coverage does not imply full decision one!

• Condition/Decision Coverage −Hybrid metric composed by the union of

condition coverage and decision coverage.

Other structure-based techniques II• Multiple Condition Coverage

−All possible combinations of all boolean conditions in a single statement must be evaluated to true and false

• Path Coverage−Whether each of the possible paths in each

function have been followed. A path is a unique sequence of branches from the function entry to the exit.

• Function, Call, LCSAJ, Loop, Race, etc. coverages

• FMI, see http://www.bullseye.com/coverage.html

http://www.bullseye.com/coverage.html

Example• RTCA published DO-178B that requires

minimal coverage for aeronautics SW systems based on their criticality:

Effect of System Failure Example Required Code Coverage

Catastrophic Crash100% condition/decision coverage 100% statement coverage

Hazardous Passenger fatality100% decision coverage100% statement coverage

Major Passenger injury 100% statement coverage

Minor Flight plan change No code coverage requirement

No effectEntertainment system failure No code coverage requirement

Summary

Summary• Testing Principles

−Testing shows presence of defects−Exhaustive testing is impossible−Early testing−Defect clustering−Pesticide paradox−Testing is context dependent−Absence-of-errors fallacy

• Testing Techniques−Black-box techniques−White-box techniques

Q&A

Documents

Testing 29/Apr/2009