Testing: improving quality of software, projects and processeslore.ua.ac.be/Teaching/STestenMaster/08TestImproving... · · 2009-05-13content of the presentation introduction

Testing: improving quality of software, projects and processes

Copyright © CTG, Inc. 1

Bert Jagers: [email protected]

Pre-Sales Consultant Application Solutions & Testing

Company Overview (www.ctg.com)

� Organization:

� HQ CTG Inc.: Buffalo, New York (Founded in 1966)� 26 additional offices in USA

� Turnover 2008: 353 million $ (= € 267 million)

Copyright © CTG, Inc.

� Turnover 2008: 353 million $ (= € 267 million)

� Headcount : 3.100 employees

� HQ CTG Europe: Diegem, Belgium (Founded in 1976)� Offices: Belgium, Luxemburg, UK, Germany

� Turnover 2008: € 59 million

� Headcount : 550 employees

� Quality certification ISO 9001, Tick IT

� Listed on the Nasdaq

2

CTG Testing Services

Collaboration Services

Business Enterprise

Test Automation ServicesTest Mgt Services

Test Improvement Services


Testing competence

Business Process

Performance Services

Business Alignment Services

EnterpriseArchitecture

Services

Security Testing Services

Performance Testing Services

CTG Testing Market PositionCTG Testing Market Position

� Market Leader in Belgium� Specialized Test Solution provider for over 10 years� Number of test specialists: 160� Preferred test partner� Test automation : partnerships with


� Test automation : partnerships with

&

� Active participation� International: EuroStar, ICS Test, Professional Tester, …� Belgium: TI-KVIV, SAI, UAMS, Datanews, Industry

Magazine, Smart Business, Network & Telecom, Vacature, etc.

Some of our References in Testing

Test Management

Set up of Test Practices /Test Process Improvement


Implementing /Using Test Tools

Test Management& Test Execution

content of the presentation

� introduction

� STBox™� Process

� People

� Technology


� Technology

� Testing @ a customer

� Test Improvement

� Conclusion

6

What is structured testing?

� CTG definition (ISTQB-2006)� The process consisting of all life cycle activities,

� both static and dynamic,

� concerned with planning, preparation and evaluationof software products and related work products todetermine that they satisfy specified requirements,


determine that they satisfy specified requirements,

� to demonstrate that they are fit for purpose,

� to detect defects,

� and to advice about risks and quality. (CTG)

structured testing: why?

DeploymentTestDevelopmentAnalysis/DesignPlanning DeploymentTestDevelopmentAnalysis/DesignPlanning

Deadline



X 100 !!


X 50 !!

Source: Compuware


Distribution of Bugs


Source: Bender RBT Inc.


� this is what it should be like…

TestDevelopment DeploymentAnalysis/DesignPlanning


Not Not a phase after a phase after development !development !

structured testing: why? added value?

� I ran some tests− but I can’t remember what happened or how I could

repeat those tests− was this testing valuable?

� I ran 100 tests and the system failed 23 times− I can’t remember how I did it or where the system


− I can’t remember how I did it or where the system failed

− was this test valuable?

� I found 56 defects− so what?

� the PROCESS of testing only has VALUE if it generates INFORMATION (this is the deliverable)

structured testing: why? added value?

• Project Intelligence

• Risks/Mitigations


• Critical for management

V-Model

Business Requirements

System Requirements

Unit

System Testing

System Integration

Testing

Acceptance Testing

Development

Testing

Business


� Test Levels

� Test Types

� Regression Testing

� Retesting

� White Box Testing

� Black Box Testing

Global Design

Detailed Design

Code/Build

Unit Testing

Unit Integration

Testing

Development

Quality Attributes (ISO9126)Accuracy

Suitability

Interoperability

Security

Maturity

Fault Tolerance

Recoverability

Understandability

Learnability

Operability

Functionality

Reliability

Useability


Operability

Attractiveness

Analyzability

Changeability

Stability

Testability

Adaptability

Co-existence

Installability

Replaceability

Time Behavior

Resource Utilization

15

Useability

Efficiency

Maintainability

Portability


Test Methodology

why “STBox™”?

� a Software Testing Method Based on CTG Experience

� “Box” refers to CTG’s general

project framework

Test Project Preparation

Detailed Test Planning

Test Build

Test Execution

For every Iteration/Test Level/Test Type:

Test Management

Quality

Management


� STBox™ is a software testing method that covers 3 dimensions:� process

� people

� technology

Test Project Closure

why “STBox™”?

� simplify communication, provide consistency, uniformity and continuity

� ISEB and ISTQB certification programs are used

� cover only certain aspects (e.g. terminology, templates) –STBox™ provides the “glue”

� also aspects where STBox™ does not follow the


� also aspects where STBox™ does not follow the standards (e.g. test procedure template, definitions black and white box testing, “test basis” concept, etc.)

18


the different phases



Test Build

For every Iteration/Test Level/Test Type

Test Management

Quality

Management


Test Build

Test Execution


Test Management

Quality

Management

STBox™in detail

1. Determine

Test Scope


2. Determine

High-Level

Test Strategy

Detailed Test

Planning4. Determine

Detailed

Test Strategy

Test Build

6. Design

Tests

7. Organize

Test

Execution

Test

Management

17. Monitor

and Adjust

Test Plans

18. Follow

Up

and Report

Status

3. Plan

Test Project

Additional Test Design

5. Plan

Iteration /

Test Level /

Test Type

FOR EACH ITERATION / TEST LEVEL / TEST TYPE :

9. Install

Test Object 23. Review

Test

22. Review

Test Basis

8. Set Up

Test

Infrastructure

16. Staff and

Manage

Test Team

Quality

Management

Test Strategy

Matrix

Project

Test Plan

FTT Tree

Detailed

Test Plan

Test Cases

Checklists

Test

Execution

Schedule

Test Scripts

FTT Tree

Test

Procedures

Test Status

Reports

Review

Report

Defect

s


11. Execute

Tests

12.

Summarize

Test

Execution

Test Project Closure13.

Consolidate

Test

Deliverables

15. Release

Test Staff

from Test

Assignment

Test infrastructure or test

object needs modification

Test object needs modificationTest repository needs

modification

10. Verify Test Infrastructure

& Test Object

14. Evaluate

Test Project

19. Manage

Defects

20. Manage

Issues &

Changes

21. Facilitate

Delivery

Test

Deliverables

and Organize

Approval

Test repository needs

modification

Test object needs modificationTest repository needs

modification

Test ResultsDefect

s

Test ResultsDefect

s

Test

Summary

Report

Test

Evaluation

Report

Test Execution

21

STBox™Essentials


the different phases related to V-Model

Acc Tst

Planning

Project Preparation

User Requirements


System

Requirements

Global

Design

Acc

Test

Build

Sys Int Tst

Planning

System

Int

Sys Tst

Planning

System

Test

Comp Int

Tst Planning

CompComp Tst

Test Management

Quality

Management

Project M

anagement


Component Test Execution

Project Closure

Detailed

Design

Code / Build

Build Int

Test

Build

Test

BuildComp

Int

Test

Build

Comp Tst

Planning

Comp Tst

Build

Component Integration Test Execution

System Test Execution

System Integration Test Execution

Acceptance Test ExecutionDeployment


Test Management

Quality

Management

Quality

Management

Project M

anagement



Test Build


Test Management

Quality

Management


Test Execution


Test Management

Quality

Management

1. Determine

Test Scope

TEST PROJECT PREPARATIONTEST MANAGER

test project preparation

1. determine test scope



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


Test Scope

2. Determine High-

Level Test

Strategy

3. Plan

Test Project

TO 8 TO 5 TO 4

TO QUALITY MANAGEMENTTO TEST MANAGEMENT

Test Strategy

Matrix

Project Test

Plan

25

TEST MANAGER

TEST MANAGER

1. determine test scope

ACCOUNTABLE

RESPONSIBLE

TO 2

TO 3

Meetings,

Interviews

Context

Information

Testing

Information

Checklist

Project

Plan

Inventory and

Assess

Available

Information

Define Test

Project

Scope

and

Objectives

1.1 1.2



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


Test

Manager

Project

Test Plan

Project

Manager

CONSULTED

Stakeholders

IT

Stakeholders

Business

26

2. determine high level test strategy

TEST MANAGER

TEST MANAGER



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


TEST MANAGER


What to test? / How deep to test?



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


objective: finding the most importantdefects as soon and as cheap as possible.




Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


Product Risk Management

Product Risk = Impact * Likelihood

� What is the impact on business?


� What is the impact on business?

� What is the likelihood defects will occur?

� Use risk factors

30

Risk factors

Likelihood

� Complexity

� New development? (level of reuse)

� Interrelations (#

Impact

� Business criticality

� Financial (or other) consequences (or damage)


� Interrelations (# interfaces)

� Size

� Technology

� Experience

damage)

� User intensity

� External visibility

31

The Product Risk Matrix

I II

LIK

EL

IHO

OD

25

HighMedium Focus of Development Level Testing

X

X

X

XX


III IV

IMPACT

LIK

EL

IHO

OD

5

15

3 9 15

MediumLow

Focus of System Level Testing

X XX

X

X

X

X X

X

32

The Product Risk Matrix

I II

LIK

EL

IHO

OD

25

Medium Focus of Development Level Testing

X

X

X

XX

Full code Inspection

Boundary Value Analysis

Statement Coverage 90%

Pair Inspection

Statement Coverage 70%

High


III IV

IMPACT

LIK

EL

IHO

OD

5

15

3 9 15

MediumLow

Focus of System Level Testing

X XX

X

X

X

X X

X

33

Statement Coverage 70%Exploratory Testing

Importance of Risk Management

� More efficiënt (and cheaper) testing

� First test in large risk areas� And test more thorough there

� Zekerheid dat het aantal technische risico’s tot een


� Zekerheid dat het aantal technische risico’s tot een minimum gereduceerd is tijdens developer testen.

� Necessary: � Developers know the risks

(Technical analyse!)

� Interesting for regression testing

34




Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

Requirements

Open documents

Typing new text

Editing existing text

Printing

Saving documents

English

French

Dutch

German

Impact Likelihood Priority

2.1. identify high level risks and requirements


German

Notepad causes instability of Windows

Response times of user actions in Notepad takes too long

Notepad for German contains translation errors in menus, pop-up windows, error messages

Unexpected crash of Notepad due to unnatural behaviour of user

Notepad is not installed correctly together with Windows installation

Notepad user interface is not user-friendly and intuitive

Notepad allows security exploits (e.g. Access to windows kernel through Notepad)

Test team is not experienced in test automation

Versions for system testing are not stable enough to proceed testing

Delay in development

Existing test procedures are not re-used to full extent

Test manager leaves the project (sickness, resignation, etc.)

Test team is not familiar with the test methodology


Test Project Risks


Product Risks




Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

Preventive Detective Corrective (Contingency )

FunctionalitySuitability

Open documents Functional Testing

Reviewing System requirements

Typing new text Functional Testing

Editing existing text Functional Testing

Printing Hire developer specialized in

printing

Functional Testing


Saving documents Functional Testing

English Functional Testing


French Functional Testing

Dutch Functional Testing

German Outsourcing of translations to

professional German translator

Functional Testing


SecurityNotepad allows security exploits (e.g. Access to windows kernel through Notepad) Security audit on design Security Testing

ReliabilityFault tolerance

Risk Response

L L L

M H H

M L LM L L

H L MH M H

H L MH H H

H M H

H L M

FTTs


2.3. define risk response


Fault toleranceUnexpected crash of Notepad due to unnatural behaviour of user Functional Testing


UsabilityUnderstandability

Notepad user interface is not user-friendly and intuitive Usability Testing

EfficiencyTime Behaviour

Response times of user actions in Notepad takes too long Performance Testing

PortabilityCo-existence

Notepad causes instability of Windows Portability Testing

InstallabilityNotepad is not installed correctly together with Windows installation Installability Testing

Reviewing Installation guide

Preventive Detective Corrective (Contingency )

ProcessExisting test procedures are not re-used to full extent Organize meeting between owner

of existing test procedures and test

manager

Delay in development - Close monitoring of the progress

by the project manager

- Start with development of high-

priority features

Demand overwork and/or engage

extra capacity

PeopleTest team is not experienced in test automation Assign dedicated test automators

and send them to training

Test manager leaves the project (sickness, resignation, etc.) Assign test lead and involve test

lead in test management activities

Test lead acts as back-up test

manager

Test team is not familiar with the test methodology Organize briefing on test

methodology

TechnologyVersions for system testing are not stable enough to proceed testing Define clear entry criteria for

system testing

Send test object back to

development team and apply

exploratory testing to the unstable

test object

Risk Response

H H

M H H

L H M

H M H

H

L M M

H H H


Test Project Risks

H M H

M L L

M L L

L L L

H M H


2.4. create test strategy matrix



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

ID

FUNCTIONAL TESTING

NOTEPAD ENGLISH

H OPEN

M SAVE

H PRINT

GENERAL FEATURES

M TYPING

Priority Features To Test

X

X

Component

Testing

X

X

Component

Integration

Testing

X

X

X

X

System

Testing

X

X

X

X

System

Integration

Testing

Acceptance

Testing

X

X

X

X

Test Build Info


M TYPING

M EDITING

H UNNATURAL BEHAVIOUR

L NOTEPAD FRENCH

L NOTEPAD DUTCH

H NOTEPAD GERMAN

L SECURITY TESTING

USABILITY TESTING

L USER INTERFACE

PERFORMANCE TESTING

L USER ACTIONS

PORTABILITY TESTING

L INSTABILITY OF WINDOWS OS

INSTALLABILITY TESTING

H AUTOMATIC INSTALL TOGETHER WITH WINDOWS XX X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X


2.5. assign test techniques



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

ID

FUNCTIONAL TESTING

NOTEPAD ENGLISH

H OPEN

M SAVE

H PRINT

GENERAL FEATURES

M TYPING

M EDITING

Part of E2E Test

Tbd Tbd Exploratory Testing Checklist

J-Unit Tst based on

decision table

Experienced

developer

Program Interface

Test

Decision TablesSyntactical Tst

Inspection System

Requirements

Part of E2E Test

Inspection System

Requirements

J-Unit Tst based

ondecision table

Program

InterfaceTest

Syntactical Tst Part of E2E Test

J-Unit Tst based

on

decision table

Experienced developer

Program

Interface

Test

Syntactical Tst

System

Integration

Testing

Acceptance

Testing

Test Build Info Component

Testing

Component

Integration

Testing

System Testing

Priority Features To Test


M EDITING

H UNNATURAL BEHAVIOUR

L NOTEPAD FRENCH

L NOTEPAD DUTCH

H NOTEPAD GERMAN

L SECURITY TESTING

USABILITY TESTING

L USER INTERFACE

PERFORMANCE TESTING

L USER ACTIONS

PORTABILITY TESTING

L INSTABILITY OF WINDOWS OS

INSTALLABILITY TESTING

H AUTOMATIC INSTALL TOGETHER WITH WINDOWS Install from CDCode ReviewInstall from

network

Walkthrough

Installation Guide

Tbd

Exploratory TestingUse list of pre-

defined

integration tests

Manual responsetime checking

Response timechecking with tool

Checklist

Penetration testPenetration test

Exploratory Testing

Extra Beta Test

German speaking

community

Code Review Exploratory

Testing

Inspection

Translationfiles

Tbd

Code Review Review translation

files

Tbd

Code Review Review translation

files

Idiot Proofing

Destructive Tst

Idiot Proofing

Destructive Tst

Idiot Proofing

Destructive TstInspection System

Requirements

1. Determine

Test Scope

TEST PROJECT PREPARATION

2. Determine High-

Level Test

Strategy

Test Strategy

MatrixTEST MANAGER

TEST MANAGER



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

3. plan test project


3. Plan

Test Project

TO 8 TO 5 TO 4

TO QUALITY MANAGEMENTTO TEST MANAGEMENT

Project Test

Plan

39

TEST MANAGER




Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management



Purpose: the C-words

� CONSOLIDATION

� COMMUNICATION



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


� COORDINATION

� CONSENSUS

� COMMITMENT

� CLAIM

� CONTROL (EVALUATION)

41


Contents: the W-words

� WHY

� WHO



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


� WHAT

� WHEN

� WHERE

� WHICH WAY (HOW)

� WHICH MEANS

42


IEEE 829 (1)

� content – 16 points � test plan information� introduction



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


� test Items� features to be tested� features not to be tested� approach� item pass / fail criteria� suspension and resumption criteria

43


IEEE 829 (2)

� content – 16 points� test deliverables� testing tasks� environmental needs



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


� environmental needs� responsibilities� staffing and training needs� schedule� risks and contingencies� approvals

44



Test Build


Test Management

Quality

Management


Test Execution


Test Management

Quality

Management



Test Build

Test Execution

For every Iteration/Test Level/Test

Type:


Test Management

Quality

Management



Test Build

Test Execution


Type:


Test Management

Quality

Management

detailed test planning

4. determine detailed test strategy


TEST MANAGER

TEST MANAGER

4. determine detailed test strategy



Test Build

Test Execution


Type:


Test Management

Quality

Management



Test Build

Test Execution


Type:


Test Management

Quality

Management




Test Build

Test Execution


Type:


Test Management

Quality

Management



Test Build

Test Execution


Type:


Test Management

Quality

Management

detailed test planning

5. plan iteration / test level / test type


TEST MANAGER

TEST MANAGER

5. plan iteration/test level/test type



Test Build

Test Execution


Type:


Test Management

Quality

Management



Test Build

Test Execution


Type:


Test Management

Quality

Management




Test Build


Test Management

Quality

Management


Test Execution


Test Management

Quality

Management

6. design tests

TEST BUILD

6. Design

Tests


FOR EACH ITERATION / TEST LEVEL / TEST TYPE :FROM 5FROM 4FROM 3

FTT TreeTESTERTESTER

TEST SUPPORT



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


7. Organize

Test

Execution

9. Install

Test Object

8. Set Up

Test

Infrastructure

FROM 12 TO 8FROM 11FROM 10 FROM 12FROM 11FROM 10

Test Cases

Test Scripts

Checklists

Test

Execution

Schedule

Test

Procedures

51

TEST MANAGER

TEST SUPPORT

TEST SUPPORT



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

6. design tests


6. design tests

a test case is a description of a well considered combination of input data, the processing of that input and the there from resulting predicted output.

test case



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


input and the there from resulting predicted output.

frame

INPUT

PROCESS

OUTPUT

TESTER

TESTER



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

test build

7. organize test execution


TEST MANAGER

TEST SUPPORT

TEST SUPPORT

TEST BUILD


FOR EACH ITERATION / TEST LEVEL / TEST TYPE :FROM 5FROM 4FROM 3

TESTER



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

test build

8. set up test infrastructure

9. install test object


6. Design

Tests

7. Organize

Test

Execution

9. Install

Test Object

8. Set Up

Test

Infrastructure

FROM 12 TO 8FROM 11FROM 10 FROM 12FROM 11FROM 10

FTT Tree

Test Cases

Test Scripts

Checklists

Test

Execution

Schedule

Test

Procedures

55

TESTER

TEST SUPPORT

TEST SUPPORT

TEST MANAGER



Test Build


Test Management

Quality

Management


Test Execution


Test Management

Quality

Management

TEST EXECUTION

Test infrastructure or


& Test Object


TO 8 FROM 7 TO 6FROM 8 FROM 9TO 9

Test ResultsDefect

s

TESTER

TEST SUPPORT



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

test execution

10. Verify test infrastructure & test object


11. Execute

Tests

12.

Summarize

Test

Execution


test object needs

modification

Test object needs

modification


modification


modification

Test object needs

modification


modification

TO 14

Test Results

Test

Summary

Report

Defect

s

57

TEST MANAGER

TESTER

TESTER

TEST EXECUTION


test object needs


& Test Object




Test ResultsDefect

s

TESTER

TEST SUPPORT



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

test execution

11. execute tests


11. Execute

Tests

12.

Summarize

Test

Execution

test object needs

modification

Test object needs

modification


modification


modification

Test object needs

modification


modification

TO 14

Test Results

Test

Summary

Report

Defect

s

58

TEST MANAGER

TESTER

TESTER

11. execute tests – record results



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management





Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management





Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


TEST EXECUTION


test object needs


& Test Object




Test ResultsDefect

s

TESTER

TEST SUPPORT



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

test execution

12. create test summary report


11. Execute

Tests

12.

Summarize

Test

Execution

test object needs

modification

Test object needs

modification


modification


modification

Test object needs

modification


modification

TO 14

Test Results

Test

Summary

Report

Defect

s

62

TEST MANAGER

TESTER

TESTER

12. create test summary report

� defects per test hour

2,5

3 Desirable

Low

Severity



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


0

0,5

1

1,5

2

2,5

Week 2 Week 4 Week 6 Week 8 Week 10

Low

Medium

High



Test Build


Test Management

Quality

Management


Test Execution


Test Management

Quality

Management

TEST MANAGER



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

test project closure

13. consolidate test deliverables


TEST MANAGER

TEST MANAGER

TEST MANAGER

TEST MANAGER



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


14. evaluate test project


TEST MANAGER

TEST MANAGER

TEST MANAGER

14. evaluate test project

purpose?

� lessons learned

� gather historical data

� consolidated test summary reports

� transfer test deliverables (reusability)



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


� transfer test deliverables (reusability)

how?

� according to test plan

� test process (efficiency)� are we doing the things right?

� test product (effectiveness)� are we doing the right things?

� advice about quality (risks/alternatives)

TEST MANAGER



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


15. release test staff from test assignment


TEST MANAGER

TEST MANAGER

TEST MANAGER



Test Build


Test Management

Quality

Management


Test Execution


Test Management

Quality

Management

test management

TEST

MANAGEMENT

16. Staff and

Manage

Test Team

FROM 3

FROM 5

19. Manage

Defects

20. Manage

Issues &

Changes



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


17. Monitor

and Adjust

Test Plans

18. Follow

Up

and Report

Status

Test Status

Reports

21. Facilitate

Delivery

TO 14

70

TEST MANAGER

16. staff & manage test team

Testing in total project effort

40%

20%

5%



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

test budgeting & planning


40%

15%

Acceptance Testing

Comp & Comp Int Testing

System & System Int Testing

But ….

These ratios are industry standards!

HOWEVER: No project is “standard”


-> correction factor needs to be applied

- depending on how much you know (risk involved) the ingredients of the correction factor may vary

Test Budget Influencers

Test Test EffortEffortEstimateEstimate

ProductivityProductivity

HowHow goodgood are are mymy teams?teams?

StrategyStrategy

WhatWhat??HowHow thoroughthorough??

InfraInfra & Tools& Tools


Learn from historical Data!

SizeSize

HowHow ““muchmuch”??”??Scope?Scope?

InfraInfra & Tools& Tools

HowHow ““fastfast” ” cancanwe test?we test?

ExpectedExpectedQualityQuality

Test Object?Test Object?Test Basis?Test Basis?

73

Correction factor Business Criticality Major Low=2

Medium=4

High=6

Technology Minor Low=1

Medium=2

High=4

Complexity Major Low=2

Medium=4


Medium=4

High=6

Development team Major Low=2

Medium=4

High=6

Test Team Minor Low=1

Medium=2

High=4

74

20% ≥ Test Budget ≥ 65 %

Correction factor

Priority Answer Factor Comment

Some business people are impacted between 30% and 100%

Based on existing and well known technology

Very complex, many interfaces to existing applications, high risk for some projects.

Experienced development team but problems with quality of analysis documents

Minor L 1 Very experienced Test Team. Good business knowledge.

4

6

1

6H

L

H

MMajor

Major

Minor

Major

5. Test Team

4. Developement Team

3. Complexity

2. Technology

1. Business Risk

Questions

Example 1


Minor L 1 Very experienced Test Team. Good business knowledge.

Factors 18

1,125

Industry standard 40

Test Budget 45 %

Risk Indicator (= Factors/16)

5. Test Team

16. staff & manage test team

test budgeting & planning

relation between:

� time (milestones/deadlines)

� budget (resources)

� quality (test coverage)



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


� quality (test coverage)

� scope

76

18. follow-up & report status

status report

purpose:� New defects� Open defects� Defects Solutions



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


� Defects Solutions� Trends & Metrics� Resource Tracking� Lead Time Tracking� Work Product Tracking

77

0

10

20

30

40

50

60

Week 1Week 3Week 5Week 7Week 9 Week 11

New

Closed

Open(cat A + B)


� Performance Indicators:

1. Define the test performance indicators

2. Align them with:� The Test Policy


� Objectives for Test Process Improvement

3. Collect data in the projects

4. Track and evaluate the Test Performance Indicators

5. Distribute to the stakeholders

Goal oriented!

78


� Product Quality� Number of defects found� Requirements coverage; reduction of defects� Improvement of defect detection percentage (DDP)

� Efficiency


� Efficiency� Test lead time; # of test cases automated� Test Effort / total development effort

� Predictability� Test lead-time slippage; effort slippage

� Maturity level

79

19. manage defects

objectives

� defect administration�follow up status defect found

�documentation defects found and solution chosen

� test management



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


�overview number of open defects per project / per domain / general

�test repair time: test planning

�test release management: entry / exit criteria (gates) between test levels

� regression testing

� product and process quality

� defect analysis

80

19. manage defects

defect logging & traceability

Defect logging - IEEE 829-1998

defect identification and description

id description

project date

test object reproducible

environment status



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


environment status

test procedure assigned to

test case / test step tester

impact information

severity of defect impact of defect

repair information

priority diagnosis information

estimated fix time

traceability

link defects with test cases

link test cases with requirements

81

19. manage defects

defect logging: 10 Golden Rules (Rex Black)

1. structure

2. reproduce

3. isolate

4. generalize



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


4. generalize

5. compare

6. summarize

7. condense

8. disambiguate

9. neutralize

10. review

82

19. manage defectsDefect Tracking Flow

New

Open

Assigned to



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management

Statuses the Role Can Assign

Role

Ne

w

Op

en

Re

jec

ted

As

sig

ne

d t

o

(te

ste

rs o

r d

eve

lop

ers

)

Po

stp

on

ed

Clo

se

d

Fix

ed

ad

y f

or

rete

st

Re

op

en


Fixed

Ready for

retest

Postponed

Reopen

Closed Rejected

83

Role

Ne

w

Op

en

Re

jec

ted

As

sig

ne

d t

o(t

este

rs o

r d

eve

lop

ers

)

Po

stp

on

ed

Clo

se

d

Fix

ed

Re

ad

y f

or

rete

st

Re

op

en

Test Manager � � � � � � � � �

Testers � � �

Developers � � � �

Analysts � � � �

Project Team � � � �

Decision Forum Representative

�



Test Build


Test Management

Quality

Management


Test Execution


Test Management

Quality

Management

quality management QUALITY

MANAGEMENT

22. Review

Test Basis

FROM 3

FROM 5

Review

Report

Defect

s

22. review test basis23. review test deliverables

& organize approval



Test Build

Test Execution



Test Management

Quality

Management



Test Build

Test Execution



Test Management

Quality

Management


23. Review

Test

Deliverables

and Organize

Approval

TO 14

85

Reviewing

Distribution of Bugs


Source: Bender RBT Inc.

86

Review Techniques

� Informal Review� Informal technique

� Walkthrough� What are we building?


� What are we building?

� Technical Review� How are we building it?

� Inspection� Is everything correctly documented?

87

Benefits of reviewing

� Early defects are often the most important

� 56% of defects are already created before coding starts

� Finds other defects then dynamic testing (design mistakes)


mistakes)

� Development productivity improves

� Reduced Lead-times

� Reduced Testing cost & time� Less Test Cycles and retesting

� Improved Communications

88


People

roles and responsibilities


test organizations

� function based test organization

� independent test organization

� role based test organization

� test competence center

� outsourced testing


� outsourced testing

...... ...... ...

...ORGANIZATIONAL MODELS

91


Technology

technology

� test infrastructure� test environment

� office environment

� test infrastructure procedures

� test tools


test infrastructure

� test environment� hardware

� operating system & system software

� network

� other applications installed (interfaces)


� test databases

94

test infrastructure

� test infrastructure procedures� organization (ownership, responsibilities, etc.)

� provision (SLA’s regarding response times, availability, etc.)

� maintenance (upgrades hardware & system software)

� troubleshooting


� troubleshooting

� configuration & change management

95

test tools

� what are tools?

• tools are no more than pieces of equipment, meant to support the manual as well as the mental work. We use the possibilities of a computer to improve the quality of testing


• Computer Aided Software Testing: supporting the testing of software by means of a computer

� a tool is a means and shouldn’t be a goal in itself

96

test tools

CAST IMPROVEMENT OF PRODUCTIVITY

tools


tools

increase the efficiency and effectiveness!!!

structured testing

test tools

classification of test tools


Automated Unit Testing

� Regression Testing purposes� Testing of a previously tested program following modification to ensure

that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed. (ISTQB glossary)

� Every new build


� Every new build

� Different Tools available

99

Test Management Tools

� Managing your Test Cases/requirements/…

� Reporting

� (Defect) Management

� Different Tools available:


Test Automation Tools

� Regression Testing purposes (system (integration) testing)� Testing of a previously tested program following modification to ensure

that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed. (ISTQB glossary)


� Every new build


101

Load and Performance Test Tools

� Performance Testing

� Load/Stress Testing




Bert Jagers

Pre-Sales Consultant Application Solutions & Testing

[email protected]

103

Documents

Testing: improving quality of software, projects and processeslore.ua.ac.be/Teaching/STestenMaster/08TestImproving... · · 2009-05-13content of the presentation introduction