University of Sunderland

CSE100: Introductory Software Engineering

Lecture 12: Testing

Requirements

Analysis

Design

Development

Testing

Evaluation

Software Development Processes/Lifecycle

Sp

ecification

Development

Val

idat

ion

Evolution

Also, for SDLC see http://uk.youtube.com/watch?v=OfgfnZZdMlI

We are here

We are also here

Requirements

Analysis

Design

Development

Testing

Evaluation

Overlapping/In Parallel?

Testing, Debugging, Verification & Validation:

Explanations & clarifications!

Testing is not debugging!

• Testing is– Seeing if a given set of inputs causes an acceptable or unacceptable behaviour in a program

• Debugging is– Seeing why a given set of inputs causes an unacceptable behaviour and what must be changed to cause the behaviour to be acceptable

Verification

Testing is one (widely used) part of verification

Verification also includes:• Inspections/reviews (formal)

– Small teams inspecting the documents and work products in stages throughout the development lifecycle

• Walkthroughs (informal)– As above but less formal/no documentation

What is verified?– Requirements specification– functional design– component design– Code

You saw this in the SimSE game!

Requirements

Analysis

Design

Development

Testing

Evaluation

Software Development Processes/Lifecycle

Sp

ecification

Development

Val

idat

ion

Evolution

But the interplay of verification & validation willdepend on your chosen development methodology

Example verification/inspection stages

• E.g. in the requirements specification– Is there ambiguity? Is it complete?

• E.g. in the component/module– Are there errors in the code logic? (error of commission)

– Are some parts of the spec ignored in the code? (error of omission)

• E.g. boundary value errors– (see later examples)

‘V’ Model

Requirements

Design

Code

Designmatches

Requirements

Structural

Functional

Usability

Codematches Design

Verification – build it the right way

Verification – build it the right way (fault-free)

(fault-free)

Valid

atio

n -

build

the

right

thin

g

Valid

atio

n -

build

the

right

thin

g

Or, in more detail….

Question

• When I ‘tested’ the bridges you built in the 2nd week, was I doing verification or validation?

• What would be appropriate methods of verifying and validating your bridge builds?

Discuss with neighbour

Software Validation

• To ensure that the software:– Conforms to the specification– Meets the needs of the user– Performs effectively and efficiently

– Performs safely

Validation

• Done after verification stages, towards or at end of coding– depending on methodology e.g. waterfall, e.g. agile or XP or test-driven, this might be at end of development or throughout

In easy to understand terms, validation is, kind of, where ‘testing’ meets ‘evaluation’

Purpose of Testing?

• 2 different purposes:1. Test the Structure of the software2. Test the Function of the software

1. Structural (White Box or Glass Box testing)– Looks at the code and its structure– Test cases examine decision points in the code– Includes unit testing, integration testing

2. Functional testing (Black Box testing)– Doesn’t look at the code– Examines whether a program accepts the correct

inputs and produces correct outputs– Includes UAT, Alpha & Beta Testing

Structural Testing

Types of structural testing

1. Unit testing

2. Integration (system) testing – Do the units coordinate properly? Are there

any problems with component interfaces?

Remember, problem solving = breaking into chunks and working on the chunks, what passes between them, and how they combine into one large chunk (unit and integration)

Structural 1) Unit Testing

• Verifies that a module works correctly in isolation– Driver component makes method calls on the tested component

– Any methods used are simulated as stubs – e.g. prints ‘hello’ to tell you it’s been successfully tested

Unit testing

Driver Component

Stub

Stub

Stub

Drivers and Stubs Example

Book Hand Over

Read BookBar Code

Update Book Details

Return Book

Module Under TestStub (supplies BarCode Value)

Driver: Calls Stub then module under test, finally outputs Date of Return

BarCode

BarCode

DateofReturn

Structural 2) Integration Testing

• Big Bang– Bring everything together then test– Bad idea

• Improved Big Bang– Bring everything together having done unit testing first

– Better idea than Big Bang• Incremental

– Better approach– Faults are localised

Structural Testing: General Description

• Test all paths correctly and fully– This can be difficult if code is complex

• Follow a process to create test data– What is ‘test data’?

• Can be done by the code writer or programmers may swap modules, – or in teams e.g. Quality Assurance teams, adversaries, bounty hunters

Process for Structural Testing

1. Draw up a Test Run Chart (rectangles & circles) to identify all paths through the module

2. Identify the type of test for a particular path– The type of path determines the number

of tests needed for each path

3. Draw up a Test Case Table showing • all tests and • expected results• actual results

3 stages

Step 1) Test Run Chart

• Shows logical flow of a program/module• Identifies all paths to be tested

The rectangle indicates normal processing

The circle indicates a location in the module where the processingpath splits.

Path 1 (No) Path 2 (Yes)

SwitchOff?

Calculatetemperature

Testing Each Path

There are five There are five paths in the main, left loop: in the main, left loop:

1 2 3 4 5

Conditions: • If heater not on (1,2,3,4) or is on (5);• If office hours (1,2) or not (3,4)

• If temperature < 65F (2) or not (1)• If temperature < 45F (4) or not (3)

DecisionAction

How exhaustively do you test?

• Exhaustive – means what?• It means: every possible scenario

• What are the implications of this?– Let’s look at an example…

Exhaustive Testing

0 loop < 20

How many possible paths are there through this program?

Discuss this with your neighbour

Exhaustive Testing

0 loop < 20

How many possible paths are there through this program?

2,432,902,008,176,640,000

Exhaustive vs Effective Testing

• Exhaustive testing = unrealistic• In the example just given:

– loop executed 20 times• but also every possible variation in the loop • And every possible sequence of variations

When combined gives millions of tests!

• So, have to run a set of tests that is Effective not Exhaustive– E.g. 99.99% correct (exhaustive) and 99.5% correct (effective)

Step 2) Identify Type of Test

Conditional Tests: Individual Values

• Make sure TRUE and FALSE sides of each condition are tested

• Testing individual values:for every valid value test for an invalid value

E.g. test for gender (‘m’ or ‘f’)– Would test for (‘m’ , ‘f’, + 2 invalids) = 4 tests

• E.g. expect ‘m’ and ‘m’ is entered OR expect ‘m’ and ‘ ’ is entered

– Assumes another letter has not been coded as valid, would not be easy to find other than:• Every character in the character set is tested (exhaustive testing)

Conditional Tests: Value Ranges

• Requires Boundary Testing:• Every bound should be tested with 2 tests:

– E.g. If the range of values is from 0 to 10:•Test 0 (valid) and -1 (invalid)•Test 10 (valid) and 11 (invalid)•Test a value randomly between 1 and 9

– 5 tests in this example– Exhaustive would be -99 to +99 (inclusive) i.e. every possible 2-digit integer

Loop Testing

• Focus exclusively on the validity of loops

• The following tests are applied to simple loops where n is the maximum number of allowable passes through the loop, try to: 1. skip the loop entirely

2. only pass once through the loop

3. m passes through the loop where m < n

4. attempt n - 1, n, n + 1 passes through the loop

Identifying the Tests

• Good idea to number the tests

• Example test for value Range– 0 to 10 gives 5 tests (boundary)

• If this test is test path 2 in the module then number tests as follows:

Test Id.

Value Pass/

Fail

2/1 -1 F

2/2 0 P

2/3 4 P

2/4 10 P

2/5 11 F

Test Data for Multiple Tests

• The same data can be used for multiple tests.

• Take an example of a record with 2 values: A & B, both are in the range 0-5 to be valid, B must be less than A.

• One test data record can carry out 3 tests for 3 different paths

• Tests 3/1, 4/1 and 5/1. Use the same data

Test id

Values Pass Fail

3/1 Test A=3

Pass

4/1 Test B=5

Pass

5/1 B < A Fail

Value A Value B

3 5

Test Cases

A set of test inputs, execution conditions, & expected results for a particular objective:1. Guarantee all independent paths in a

program have been exercised at least once2. Exercise True & False for all logical

decisions 3. Execute all loops and conditions at their

boundaries and within their operational bounds

4. Exercise internal data structures (e.g. arrays, record structures) to ensure their validity

Test Cases

• Derived during all phases of the development cycle

• Determine what are your expected results before you run a test-case– Then, running a test case provides the actual results

• Never throw away test-cases and results

• Test cases are presented as a table.

Step 3) Test Case Table

Description Input ExpectedResults

ActualResults

       

     

       

Pass/Fail

 

 

 

 

Test id.(path/no.)

 

 

 

 

Functional Testing

Types of Functional Testing

UAT, Alpha Testing, Beta Testing1. User-acceptance testing (UAT) – before final

delivery:– Product is handed over to user to test it in a real

time scenario– See if it works according to the system specifications

and satisfies all the user requirements. – As the user uses the software, some unknown bugs might

be found – these are communicated to the developers to be fixed, and this helps improve the final product.

When testing is done by paid testers, or the developers – it’s done in 2 stages:

2. Alpha-testing – in development environment3. Beta-testing – in end-users’ environment (by

developer and/or end user)

User Acceptance Testing

• Define acceptance criteria with client during phases of SDLC

• Well defined acceptance plan helps development by identifying users’ needs during software development

• The UAT must be created or reviewed by client & Software Engineer together

• Identifies:– interim and final products for acceptance, – acceptance criteria (and who will do the acceptance

activities) – Schedule (including time for users to do the activities)

• When are you ready to do UAT? – System Testing is completed and defects identified are either

fixed or documented– Acceptance plan is prepared and resources have been identified– Test environment for the acceptance testing is available

• When is it finished?– Acceptance decision is made for the software – In case of any caveats, development team is notified

Beta-Testing

• Undertaken after UAT• Timed ahead of the official release date

• System released to a small (select) number of customers

• Customers use the system as though already released and are ‘encouraged’ to find bugs

Functional Testing: General Description

• Tests the specification (or data created from the spec.) not the code– ‘what system does’ not ‘how it does it’

• i.e. tests data produced by the designer not the programmer

Functional Testing

• Black box attempts to derive sets of inputs that will fully exercise all the functional requirements of a system.

• Aims to find errors in the following categories: • interface errors• errors in data structures or internal

database access• performance errors• initialization and termination errors

An Example Black Box Test

BookHanded Over

UpdateBook Details

ReadBook Record

Bar Code

Book Code

Date of Return

MembershipId

After testing these 3 modules individually, all 3 modules are combined in a test. For the 3-module together test, a single argument the – ‘MembershipId’ producing a single reply ‘Date of Return’. Book Barcode isRead as part of the processing.

Book Code

Date ofReturn

• Earlier we said functional testing uses a ‘black box’ approach i.e. looks at input and output but ignores ‘inside workings’

• So – how to decide on your test input?

• We can use Equivalence Partitioning to guide black-box testing

Deciding on functionaltest input data

Equivalence Partitioning

Input domain

12

3

4

Input domain partitioned into four sub-domains.

Too manytest inputs.

Four test inputs, one selected from each sub-domainor equivalence class.

How to partition

• Inputs to a program provide clues to

partitioning:– If a program takes an input X, where X

is an integer

– Where X<0 the program must perform one

task and another task where X>=0

– The input domain is way too big because

X can be a huge number of values

…how to partition

In partition testing our goal is to cover all equivalence classes

– We expect the program to behave the same way for every input where X < 0

– We expect the program to perform the same way for all values of X >= 0

– An equivalence class is considered covered when at least one test has been selected from it.

Two sub-domains

One test case:X = -3

Another test case:X = 15

X<0 X>=0Equivalence class

Equivalence class

Non-overlapping (disjoint) partitions

• In the previous example, the two equivalence classes are non-overlapping. In other words the two sub-domains are disjoint

• In this case we can pick just one test input from each equivalence class to test the program

Overlapping partitions

• Suppose a program takes three integers X, Y and Z, and it is known that:

•X<Y •Z>Y

Overlapping partitions

X<Y

X>=Y

Z>Y Z<=Y

X<Y, Z>YX=3, Y=4, Z=7

X<Y, Z<=YX=2, Y=3, Z=1

X>=Y, Z>YX=15, Y=4, Z=7

X>=Y, Z<=YX=15, Y=4, Z=1

Guidelines for defining equivalence classes for black

box testing: If an input condition specifies a range (e.g. 1-

12), one valid and two invalid equivalence classes are defined (e.g. 5, 0 and 13).

If an input condition requires a specific value (e.g. ‘m’) then one valid and two invalid equivalence classes are defined (e.g. ‘m’, ‘f’, and ‘_’).

If an input condition specifies a member of a set (e.g. ‘change pin’, ‘check balance’, ‘withdraw cash’), then one valid and one invalid equivalence classes are defined.

If an input condition is Boolean (may or may not be present), then one valid and one invalid class is defined.

Example equivalence class (set of valid or invalid states

for input conditions)

Example: • a function which accept input parameter "month" of a

date. • valid range for month is 1(January)to 12(December).

This valid range is called a partition. • In this example there are two further partitions of

invalid ranges. – The first invalid partition would be <= 0 – the second invalid partition would be >= 13. – You can also have invalid partition for non-digit input as well

       .... -2 -1  0 1 .............. 12 13  14  15 .....        --------------|-------------------|--------------------- invalid partition 1     valid partition    invalid partition 2

Final comments on testing

Testing good practice:1. Begins at component level & works out

towards integration of whole system2. Different testing techniques are

appropriate at different times3. Testing is done by the developers and (in

large projects) an independent test group4. Testing & debugging aren’t the same – but

any test strategy has debugging in it

• The last part of the course will look at:– user evaluation strategies and at – evolution of software projects

Before tomorrow, you must:

• Look over the course notes so far• Tomorrow you will get the assignment

– There’ll be time to do it in class in the next 2 weeks, with tutor support

– There’ll be an interim marking next Tuesday

• The assignment tests ALL the module learning outcomes

• This means we’ll be asking you to give a full account of what you’ve learned– All aspects over the full SDLC