Software testing

Software Testing Strategies

based onChapter 13

1

Software Testing

2

Testing is the process of exercising a program with the Testing is the process of exercising a program with the specific intent of finding errors prior to delivery specific intent of finding errors prior to delivery to the end user.to the end user.

errorserrors

requirements conformancerequirements conformance

performanceperformance

an indicationan indicationof qualityof quality

Who Tests the Software?

3

developerdeveloper independent testerindependent tester

Understands the system Understands the system

but, will test "gently"but, will test "gently"

and, is driven by "and, is driven by "deliverydelivery""

Must learn about the system,Must learn about the system,but, will attempt to but, will attempt to breakbreak it it

and, is driven by and, is driven by qualityquality

Software TestingGoalsTypes of testsLevels of testsTest measuresTest plan

GoalsVerification: Have we built the software

right?

Validation: Have we built the right software?

Bug-free, meets specs

Meets customers’ needs

Are Verification and Validation different or variation of the same ideaMy argument: “meeting specs” should equal “meeting customer needs”... not generally true (my kitchen, satellite systems)

Software TestingGoalsTypes of testsLevels of testsTest measuresTest plan

most of this discussion focuses on verification(more specifically bug testing)

Types of testsBlack boxWhite box

Black box tests

input output

interface

1. Does it perform the specified functions?2.Does it handle obvious errors in input?3.Ariane5 – lousy error handling4.Classic ints vs floats, yards vs meters Black box should catch these if there is adequate “test coverage”

Example: ordered list of ints

class OrdInts

create

getFirst

getNext

insert

delete

print

L=create()L.insert(5)L.insert(-1)L.insert(-1)p=L.getFirst()print (p)L.delete(p)p=L.getFirst()print(p)p=L.getNext(p)print(p)p=L.getNext(p)

-1-15error

Black box testsAdvantage: black box tester≠developer is

unbiased by implementation details. e.g., Use Case testing, just work through all the Use Cases

Disadvantage: black box tester is uninformed about implementation detailsunnecessary tests – test same thing in different

wayinsufficient tests – can miss the extremes,

especially if actual use follows a different pattern

black box tests

Input Code

choose good distribution of input – hope good distribution of code tested

x

x

x x

unnecessary tests

Input Code

large range of input may exercise a small part of codee.g., operator test of satellite control stations, run through eachinput and output light/key option. Testing same functions, whereas no one had a test for my map function.

insufficient tests

Input Code

a small range of input may exercise a large range of codebut can you ‘know’ this without knowing the code? Did we miss the 20%

sufficient tests

Input Code

complexcode

a small range of input may exercise a small but important/error-prone region of code

White box tests Based on code

test 1

test 2

Example: ordered list of intsclass ordInts {

public: …

private: int vals[1000];

int maxElements=1000;…

}

Example: ordered list of ints

bool testMax(){ L=create(); num=maxElements; for (int i=0; i<=num; i++)

print iL.insert(i)

print maxElements;}

White box testsAdvantage:

design tests to achieve good code coverage and avoid duplication

can stress complicated, error-prone codecan stress boundary values (fault injection)

Disadvantage:tester=developer may have biasif code changes, tests may have to be

redesigned (is this bad?)

Example: ordered list of ints

class OrdInts

create

getFirst

getNext

insert

delete

print

L=create()L.insert(1)L.insert(2)L.insert(3)

L.insert(1001)p=L.getFirst()print(p)p=L.getNext(p)print p

123

…

……

Types of testsBlack box: test based on interface, through

interfaceWhite box: test based on code, through code

Testing strategy can/should include all approaches!

My experience: Black Box = non developer, outside testing idiots in your case who?White Box = developer, part of development process in your case who?Gray Box = hated, non developer within developer organization in your case who?

Software Testing

21

White-Box TestingWhite-Box Testing

... our goal is to ensure that ... our goal is to ensure that allall statements and conditionsstatements and conditions have have

been executed at least been executed at least onceonce ... ...

Black-BoxBlack-Box Testing Testing

requirementsrequirements

eventseventsinputinput

outputoutput

Basis Path Testing

22

First, we compute the cyclomatic complexity:

number of simple decisions + 1

or

number of enclosed areas + 1

In this case, V(G) = 4

White-Box TestingWhite-Box Testing

Basis Path Testing

23

Next, we derive the Next, we derive the independent paths:independent paths:

Since V(G) = 4,Since V(G) = 4,there are four pathsthere are four paths

Path 3: 1,2,3,6,7,8Path 3: 1,2,3,6,7,8Path 2: 1,2,3,5,7,8Path 2: 1,2,3,5,7,8Path 1: 1,2,4,7,8Path 1: 1,2,4,7,8

Path 4: 1,2,4,7,2,4,...7,8Path 4: 1,2,4,7,2,4,...7,8

Finally, we derive testFinally, we derive testcases to exercise these cases to exercise these paths.paths.

11

22

3344

55 66

77

88

White-Box TestingWhite-Box Testing

A number of industry studies have indicated A number of industry studies have indicated that the higher V(G), the higher the probability that the higher V(G), the higher the probability or errors.or errors.

Loop Testing

24

Nested Nested LoopsLoops

ConcatenatedConcatenated Loops Loops Unstructured Unstructured

LoopsLoops

Simple Simple looploop

White-Box TestingWhite-Box Testing

Why is loop testing important?

Equivalence Partitioning & Boundary Value Analysis

25

If x = 5 then …

What would be the equivalence classes?

If x > -5 and x < 5 then …

White-Box TestingWhite-Box TestingBlack-Box Black-Box

TestingTesting

Comparison Testing

26

Used only in situations in which the reliability of software is absolutely critical (e.g., human-rated systems)Separate software engineering teams develop

independent versions of an application using the same specification

Each version can be tested with the same test data to ensure that all provide identical output

Then all versions are executed in parallel with real-time comparison of results to ensure consistency

Black-BoxBlack-Box Testing Testing

Levels of tests

UnitIntegrationSystemSystem integration

white

black

Testing measures (white box)Code coverage – individual modulesPath coverage – sequence diagramsCode coverage based on complexity – test

of the risks, tricky part of code (e.g., Unix “you are not expected to understand this” code)

Levels of Testing

29

Unit testing Integration testing Validation testing

Focus is on software requirements System testing

Focus is on system integration Alpha/Beta testing

Focus is on customer usage Recovery testing

forces the software to fail in a variety of ways and verifies that recovery is properly performed Security testing

verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration

Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume

Performance Testing test the run-time performance of software within the context of an integrated system

Unit TestingTests the smallest individually executable code

units.Usually done by programmers. Test cases might be selected based on code, specification, intuition,

etc.

Tools:Test driver/harnessCode coverage analyzerAutomatic test case generator

Integration TestingTests interactions between two or more units or components. Usually done by programmers. Emphasizes interfaces.

Issues: In what order are units combined?How do you assure the compatibility and

correctness of externally-supplied components?

Integration TestingHow are units integrated? What are the

implications of this order?

Top-down => need stubs; top-level tested repeatedly.

Bottom-up => need drivers; bottom-levels tested repeatedly.

Critical units first => stubs & drivers needed; critical units tested repeatedly.

Integration TestingPotential Problems:Inadequate unit testing.Inadequate planning & organization for

integration testing.Inadequate documentation and testing of

externally-supplied components.

Stages of Testing

Testing in the LargeSystem TestingEnd-to-End Testing Operations Readiness TestingBeta Testing Load TestingStress TestingPerformance TestingReliability Testing Regression Testing

System Testing

Test the functionality of the entire system.

Usually done by professional testers.

Realities of System Testing

Not all problems will be found no matter how thorough or systematic the testing.

Testing resources (staff, time, tools, labs) are limited.

Specifications are frequently unclear/ambiguous and changing (and not necessarily complete and up-to-date).

Systems are almost always too large to permit test cases to be selected based on code characteristics.