Testing Versus Inspection

7/30/2019 Testing Versus Inspection

1/40

Test versus inspection

Tor Stlhane


2/40

Content

Part 1

Introduction

Inspection processes Testing processes

Part 2

Tests and inspections some data Inspection as a social process two experiments

and some conclusions


3/40

Adams data - 1

Mean Time to Problem Occurrence

yearsProduct 1.6 5 16 50 160 500 1600 5000

1 0.7 1.2 2.1 5.0 10.3 17.8 28.8 34.22 0.7 1.5 3.2 4.3 9.7 18.2 28.0 34.33 0.4 1.4 2.8 6.5 8.7 18.0 28.5 33.74 0.1 0.3 2.0 4.4 11.9 18.7 28.5 34.25 0.7 1.4 2.9 4.4 9.4 18.4 28.5 34.26 0.3 0.8 2.1 5.0 11.5 20.1 28.2 32.07 0.6 1.4 2.7 4.5 9.9 18.5 28.5 34.08 1.1 1.4 2.7 6.5 11.1 18.4 27.1 31.99 0.0 0.5 1.9 5.6 12.8 20.4 27.6 31.2


4/40

Adams data 2

The main information that you get from the

table on the previous slide is that

Some defects are important because they will

happen quite often.

Most defects are not important since they will

happen seldom.

How can we tell the difference?


5/40

Testing and inspection the V model


6/40

Testing and inspection

The important message here is that testingcannot always be done.

In the first, important phases, we have nothing

to execute and will thus always have to dosome type of inspection.

This might be considered one of the weaknesses

of traditional software engineering over Agiledevelopment.


7/40

Testing and inspection 2

In order to understand the main differences

between testing and inspection, we should

consider Fits list.

Based on this, we will give a short discussion of

the relative merits of testing and inspection.


8/40

Area of

competence

Man Machine

Understanding Good at handling variations in

written material

Bad at handling variations in

written materialObserve General observations,

multifunctional

Specialized, good at observing

quantitative data, bad at

pattern recognition

Reasoning Inductive, slow, imprecise but

good at error correction

Deductive, fast, precise but

bad error correction

Memory Innovative, several access

mechanisms

Copying, formal access

Information

handling

Single channel, less than 10

bits per second

Multi channel, several

Megabits per second

Consistency Unreliable, get tired, depends

on learning

Consistent repetition of several

actions

Power Low level, maximum ca. 150

watt

High level over long periods

of time

Speed Slow seconds Fast


9/40

Man vs. machine 1

Good when we need the ability to Handle variation

Be innovative and inductive

Recognize and handle patterns

Not so good when we need the ability to

Do the same things over and over again in aconsistent manner

Handle large amount of data


10/40

Man vs. machine 2

In order to do the best job possible we needprocesses where we let each part

Do what they are best at:

Man is innovative Machine handles large amounts of data

Support the other with their specialties.

Machine supports man by making large amountsof information available

Man support machine by providing it with

innovative input


11/40

General considerations - documents

Architecture, system, sub-system and

component design plus pseudo code. Here we

can only use inspections.

Man will use experience and knowledge to

identify possible problems

Machine can support by identifying information

e.g. find all occurrences of a string.


12/40

General considerations code (1)

For executable code, we can use inspection,testing or a combination of both.

The size and complexity degree of dynamism

of the code will, to a large degree, decide ourchoice.

Other important factors are the degree of

experience with The programming language

The algorithms used


13/40

General considerations code (2)

Simple code Start with inspection all code

Design and run tests

Complex code Start with inspection focus on algorithm and

logic

Decide test completeness criteria we cannottest everything

Design and run tests


14/40

Inspection processes


15/40

Inspections 1

The term inspection is often used in a rather

imprecise manner. We will look at three typesof inspection:

Walkthrough

Informal inspection also called informalreview

Formal inspection also called formal review

or just inspectionThe first two types are usually project internal

while the last one is used as a final acceptanceactivity for a document.


16/40

Inspections 2

For all types of inspections:

The quality of the results depends on the

experience and knowledge of the participants.

Garbage in Garbage out

It might be a good idea to involve customer

representatives.


17/40

The walkthrough process

Walkthrough is a simple process mostly used

for early decisions for an activity. The

document owner:

1. Makes a rough sketch of the solution

architecture, algorithm etc.

2. Presents explain the sketch to whoever

shows up.

3. Registers feedback improvements.


18/40

Walkthrough pros and cons

Pros: Easy and inexpensive. Needs no extra

preparation.

Collect ideas at an early stage of development.Cons:

No commitment from the participants

May collect many loose or irrelevant ideas


19/40

The informal inspection process

Individual

checking

Planning

Productdocument

Changerequests

Rules,checklists,

procedures

Logging

meeting


20/40

Informal inspections pros and cons

Pros: Is simple and inexpensive to perform.

Can be used at all stages of development

Usually has a good cost / benefit ratio

Needs a minimum of planning

Cons:

No participant commitment

No process improvement


21/40

The formal inspection process

The formal inspection process described below

is with small variations the most

commonly used. The version shown on the

following slides stem from T. Gilb and D.Graham.

We recommend this process as the final

acceptance process for all importantdocuments


22/40

Formal inspection process overview

Walk-

through

Kick-off Individual

checking

Edit

and

follow-

up

Planning

Process

improvements

Product

document

Changerequests

Rules,

checklists,

procedures

Logging

meeting


23/40

Distribution of resources

Activity Range %Typical

value %

Planning 3 5 4

Kick-off 4 7 6

Individual checking 20 30 25

Logging 20 30 25

Editing 15 30 20

Process brainstorming 15 30 16

Leader overhead, follow up, entry,

exit

3 5 4


24/40

Initiating the inspection process

The inspection process starts with a request

for inspection from the author to the QA

responsible.

The QA responsible appoints an inspection

leader.

First step is always to check that the

document is fit for inspection.


25/40

Planning

Important planning points are:

Who should participate in the inspections

Who is interested?

Who have time available for preparation and

meetings?

Who has the necessary knowledge concerning

application, language, tools, methods?


26/40

Kick-off

Important activities here are:

Distribution of necessary documents:

Documents that shall be inspected

Requirements

Applicable standards and checklists

Assignment of roles and jobs

Setting targets for resources, deadlines etc.


27/40

Individual checking

This is the main activity of the inspection. Each

participant read the document to look for

Potential errors - inconsistencies with

requirements or common application

experience

Lack of adherence to company standards or

good workmanship


28/40

Logging meeting

The logging meeting has three purposes:

Log issues already discovered by inspection

participants

Discover new issues based on discussions

and new information that arises during the

logging meeting.

Identify possible improvement to the

inspection or development process.


29/40

Improve the product - 1

The author receives the log from the inspection

meeting. All items - issues - in the log are

categorised as one of the following:

Errors in the authors document.

Errors in someone elses document.

Misunderstandings in the inspection team.


30/40

Improve the product - 2

Errors in own document:

Make appropriate corrections

Errors in someone elses documents:

Inform the owner of this document.

Misunderstandings in the inspection team:

Improve document to avoid further

misunderstandings.


31/40

Checking the changes

This is the responsibility of the inspectionleader. He must assure that all issues raised

in the log are disposed of in a satisfactory

manner: The documents that have been inspected

Related documents - including standards

and checklists Suggested process improvements


32/40

Formal inspection pros and cons

Pros:

Can be used to formally accept documents

Includes process improvement

Cons:

Is time consuming and expensive

Needs extensive planning in order to succeed


33/40

Testing processes


34/40

Testing

We will look at three types of testing:

Unit testing does the code behave as

intended. Usually done by the developer

Function verification testing also calledsystems test. Does the component or system

provide the required functionality?

System verification testing also called

acceptance test. Does the hardware and

software work together to give the user the

intended functionality?


35/40

The unit testing process

Unit testing is done by the developer one or

more times during development. It is a rather

informal process which mostly run as follows:

1. Implement (part of) a component.

2. Define one or more tests to activate the code

3. Check the results against expectations and

current understanding of the component


36/40

Unit testing pros and cons

Pros:

Simple way to check that the code works.

Can be used together with coding in an

iterative manner.

Cons:

Will only test the developers understandingof the spec.

May need stubs or drivers in order to test


37/40

The system test process

A systems test has the following steps:

1. Based on the requirements, identify

Test for each requirement, including error handling

Initial state, expected result and final state

2. Identify dependencies between tests

3. Identify acceptance criteria for test suite

4. Run tests and check results against

Acceptance criteria for each test Acceptance criteria for the test suite


38/40

Systems test pros and cons

Pros:

Tests systems behavior against customer

requirements.

Cons:

It is a black box test. If we find an error, the

systems test must be followed by extensive

debugging


39/40

The acceptance test process

The acceptance test usually has three actvities

both involving the customer or his

representatives:

Rerun the systems test at the customers site.

Use the system to solve a set of real-world

tasks.

Try to break the system by stressing it or by

feeding it large amounts of illegal input


40/40

Acceptance test pros and cons

Pros:

Creates confidence that the system will beuseful for the customer

Shows the systems ability to operate in thecustomers environment

Cons:

Might force the system to handle input that itwas not designed for, thus creating anunfavorable impression.