powerpoint template for testing training

© Allstate Northern Ireland Limited Proprietary and Confidential

Testing Training MaterialJohn Roddy

9th October 2009

Proprietary and ConfidentialMarch 4, 20152

What is Software Testing?

“…the process of executing a program with the intent to certify its quality”

Mills

“…the process of executing a program with the intent of finding failures / faults”

Myers

“…the process of exercising software to detect errors & to verify that it satisfies specified requirements”

“Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets required results.”

Bill Hetzel, 1983

March 4, 20153 Proprietary and Confidential

What is Software Testing (continued)?

Software testing is a process:The input is often stakeholder requirementsThe output is quality informationThe process methodically probes the application

from various angles


What is Software Testing (continued)?

Software testing comprises aspects of:• Engineering

– We must design tests that are effective in identifying software failures

• Literature– We must understand the stakeholders needs and desires.– A stakeholder is a person or entity who has a vested interest in the success

of a project. They may be end-users, financial backers, company sponsors or corporate members.

• Communication– We must present our findings in a way that our clients can make informed

decisions upon


Get with the Lingo!

Terms: Error - a human action producing incorrect result

Fault - a manifestation of error in software

Failure - deviation of software from expected delivery or service

Do 100 I=1.10X

Points to

Points to

produces

produces


Testing Terminology

DEFECTThe departure of a quality characteristic from its specified value that results in a product or

service not satisfying its normal usage requirements

RELIABILITYThe probability that software will not cause the failure of a system for a specified period of time

under specified conditions

QUALITYThe totality of the characteristics of an entity that bears on its ability to satisfy or implied needs

QUALITY ASSURANCEAll those planned actions used to fulfill the requirements for quality.


Misconceptions

Testing is an unnecessary expenseA common development perception

Software testers are ten a pennyHuge difference between good and bad testing

Software testing is not difficultMost testing activities are not rocket science but …

• Testers have a juggle many things at once• Make new decisions constantly• Think methodically and scientifically• Questions things other people take for granted


Cost of Software Failures

A single failure may incur little cost or millionsIn extreme cases software failure may cost LIVES (safety

critical systems – Airline, Nuclear Power)The cost of failures increases proportionately (tenfold)

with the passing of each successive stage in the system development process before they are detected

To correct a problem a requirements stage may cost £1To correct the problem post-implementation may cost

millions


Summary

The purpose of testing is to find faults. Faults can be fixed therefore making better software. Better software is more reliable, less prone to failures.

Testing enables us to measure the quality of the software

This enables us to understand and manage the risk to the business


MORNING BREAK


The Test Process

The test process has 5 steps: Test Planning Test Specification Test Execution Test Recording Checking for Test Completion


Test Planning and Test Specification

Test Planning• The Test Plan describes how the Test Strategy is implemented• A project plan for testing• Defines what is to be tested, how it is to be tested and what is needed

for testing

Test Specification has 3 steps• Preparation & Analysis• Building Test Cases• Defining expected Results


Test Execution Checklist

Test execution schedule / logIdentify which tests are to be runTest environment primed and readyResources ready, willing and ableBack-up and Recovery Processes in placeAny batch runs planned and scheduled

When all the above are in place we are ready to run the tests


Test Recording and Test Completion

Test Recording Test Log should record

Software and Test Versions Specifications / Requirements used as Test Base Test Timings Test Results (Actual Results v Expected Result) Any Defect Details

Test Completion Have we fulfilled the Test Exit Criteria Used to determine when to stop this phase of testing

Key Functionality Tested Test Coverage Defect detection rate


The Psychology of Testing

In this session we will: Understand what qualities make a good tester Look at a testers relationship with developers Look at a tester relationship with management Understand the issues with testing independence



What makes a Tester?

Testing is primarily to find faults Can be regarded as “destructive” Development is “constructive” Testing asks questions Testers need to ask questions

A tester needs many qualities



What makes a tester:

Intellectual Qualities Can absorb incomplete data Can work with incomplete data Can level quickly on manual levels Good verbal communication Good written communication Ability to prioritise

Proprietary and Confidential



Knowledge How projects work How computer systems and business needs interact Test Techniques Testing Processes Testing best practices To be able to thin outside the box



What makes a good tester? More Skills to acquire

Ability to find faults – planning, preparation & execution Ability to understand systems Ability to read specifications Ability to extract testable functionality Ability to work efficiently Ability to focus on essentials




Communication with Developers A good relationship is vital Developers need to keep testers up-to-date with changes to

application Testers need to inform developers of defects to allow fixes to be

applied



What makes a good tester? Communication with Management

Managers need progress reports

The best way is through Metrics Number of Tests planned and prepared Number of tests executed to date Number of defects raised & fixed How long planning and preparation and execution stages take



Testing Independence It is important that testing is separate from development The developer is likely to confirm adherence not deviation The developer will make assumptions

Levels of Independence Low – Developers write their own tests Medium – tests are written by another developer High – Tests are written by an independent body Utopia – tests generated automatically



Testers require a particular set of skills The desire to break things The desire to explore and experiment Communication Questioning

Testing requires a different mentally to development “Destroying” things rather than creating them Testing should be separate from development


LUNCH


Re-Testing and Regression Testing

Re-Testing is the re-running of failed tests once a fix has been implemented to check that the fix has worked

Regression Testing is running a wider test suite to check for unexpected errors in unchanged code


Re-Testing

The need for re-testing needs to be planned and designed for: Schedules need to allow for re-testing Tests need to be easily re-run Test Data needs to be re-usableThe environment needs to be easily restored


Regression Testing

Tests will need to be re-run when checking software upgradesRegression tests should be run whenever there is a change to

the software or the environment Regression tests are executed to prove aspects of a system

have not changedRegression testing is a vital testing technique

Selecting cases for regression Tests for areas that change regularlyTests of functions that have a high level of faults

Regression Testing is the ideal foundation for Automation


Regression Testing Strategy

Several factors to determine strategy: How many test cases in regression test set? What criteria should be used to select them? When will regression testing be performed? Who’s responsibility is it?

Why should it be a continuous activity?


Expected Results

In this session we will:

Understand the need to define expected results Understand where expected results can be found


Expected Results

Expected Results = Expected Outcomes Identify required behaviour If the expected outcome of a test is not defined

the actual output may be misinterpreted Running a test with only a general concept of the

outcome is fatal It is vital the Expected Results are defined with

the tests before they can be used You cannot decide whether a test has passed

just because it looks right


Expected Results

Summary

Without defining expected results how do you know if a test has passed or failed?

Expected Results can be found in the system specifications and by asking experienced business users


Prioritisation of Tests

In this session we will

Understand why we need to prioritise tests

Understand how we decide the priority of individual tests



It is not possible to test everything

Therefore errors will get through to the Live System

We must do the best testing possible in the time available

This means we must prioritize and focus testing on the priorities



Aspects to Consider Severity Probability Visibility Priority of Requirements Frequency of Change Vulnerability to error Technical Complexity Time and Resources



Business Criticality What elements of the application are essential to the success of the organization?

Customer Factors How visible would a failure be? What does the customer want?

Technical Factors How complex is it? How likely is an error? How often does this change?



Summary

There will never be enough time to complete all tests

Therefore those tests that cover those areas deemed most important (to the business, highest risk) must be tested first where possible


Models for Testing

There are many approaches to testing.

To models widely use are:Verification, Validation and Testing (VV&T)The V-Model

V-Model The most commonly used model in testing It represents the software development lifecycle Shows the varies stages in development and testing Shows the relationship between the various stages


Models for Testing

VV&T Verification

• The process of evaluating a system or component to determine whether the products of the given development phase satisfies the conditions imposed at the start of the phase

Validation• Determination of the correctness of the products of software

development with respect to the user needs and requirements

Testing• The process of exercising software to verify that it satisfies

specified requirements and to detect faults


Testing in the Lifecycle

Types of Lifecycle Sequential Lifecycle – Waterfall, V-Model Iterative Lifecycles – Spiral, Pre-planned incremental

delivery Rapid Development Models – RAD, DSDM (Agile) Evolutionary Object Oriented (OO) Extreme Programming (XP)


Waterfall Model


A typical development lifecycle V-Model


Rapid Application Development (Agile)


BREAK


High Level Test Planning


Look at how a test plan is put together

Understand how it should be used and maintained

Understand why they are so important to a testing project



What is a test plan?

“A document describing the scope, approach, resources and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, who will do each task and any risks required contingency planning”

A project plan for testing Covering all aspects of testing A “living” document that should change as testing progresses



IEEE 829 – Standard for Test DocumentationInstitute of Electrical & Electronic Engineers

• Standard for Test Documentation• Includes documentation templates for:

– Test Plan: Plan how the testing will proceed. – Test Design Specification: Decide what needs to be tested. – Test Case Specification: Create the tests to be run. – Test Script: Describe how the tests are run. – Test Item Transmittal Report: List of items released for testing.– Test Log: Record the details of tests in time order. – Test Incident Report: Details events that need to be investigated.– Test Summary Report: Summarise and evaluate tests.



1. Test plan identifier.2. Introduction.3. Test items.4. Features to be tested.5. Features not to be tested.6. Approach.7. Item pass/fail criteria.8. Suspension/Resumption.

9. Test deliverables. 10. Testing tasks. 11. Environmental needs. 12. Responsibilities. 13. Staffing and training needs. 14. Schedule. 15. Risks and contingencies. 16. Approvals.


High Level Test Planning : Who

Test Staff•Who will perform the testing•How will we address skill gaps•Contingency planning

Support Staff•Who will install operating systems•Who will set up and manage test databases


High Level Test Planning : What

Test items What components need testing

Features to be tested What features are subject to testing

Feature not to be tested What features are not ready for testing What features have already been tested What features will be tested at a later level

Software & Testing Risk Contingency planning


High Level Test Planning : When

Test ScheduleUse of Gant charts Initially based on activities found in test process

Test DeliverablesWhen do they need to be deliveredWhat is their frequency of delivery

Work BreakdownList of tasks required to be performedAll tasks have an associated period of time


High Level Test Planning : How

Test Environment Physical location Workspace requirements Hardware Requirements Software Requirements Network Requirements



Test Approach

Arguably the most important section of a test plan document

Relates to the details of test case designRelates to the details of test case designOften categorized by testing typeOften categorized by testing typeSpecifiesSpecifies

Techniques to be used in designTechniques to be used in design Test completion criteriaTest completion criteria Measurement techniquesMeasurement techniques ApproachApproach



Summary Test plans are project plans for testing

They identify why testing is needed, what will be tested, the scope of this phase of testing, what deliverables testing will provide and what is required to enable testing to succeed

They are “living” documents that must evolve as the project progresses


DEMO OF DMEA TEST PLAN


Day 2

Topics:

Types of Testing White Box Testing Black Box Testing Reviews


Key Testing Phases

Component TestingThe Testing of individual software components. The main focus is on the internal program

structure Component Integration TestingProcess of combining components into larger assembles. Looks at the interaction between

components and their interfaces. Functional System TestingThe process of testing an integrated system to verify that it meets specify requirements Non-Functional System TestingTesting of those requirements that do not relate to functionality - performance System Integration TestingTesting performed to expose faults in the interfaces and in the interaction between integrated

systems Acceptance TestingFormal testing conducted to enable a user, customer or other authorized entity whether to

accept a system or component


Acceptance Testing

In this session we will Understand what acceptance testing is

Why you would want to do it How you would plan it and prepare for it What you need to actually do it

Understand the different types of acceptance testing


Acceptance Testing

What is Acceptance Testing?

“Formal testing conducted to enable a user, customer or other authorized entity to determine whether to accept a system or component”


Acceptance Testing

What is User Acceptance Testing (UAT)?Exactly what it says it is

The set-up will represent a working environment

Users of the end product conducting the tests

Covers all aspects of the project, not just the system

Also known as Business Acceptance Testing or Business Process Testing


Acceptance Testing

Planning UAT Why Plan?

Things to Consider – what environment is going to be replicated

Preparing Tests Manual Automated


Acceptance Testing

Why Plan? If you don’t, then how do you know you have achieved

what you set out to do

Avoids repetition

Test according to code releases

Makes efficient and effective use of time and resources


Acceptance Testing

Things to consider:

Timescales

Resources

Availability


Acceptance Testing

Preparing your tests:

Take a logical approach

Identify the business processes

Build into everyday business scenarios


Acceptance Testing

Data Requirements Copied Environments Created Environments

Running the Tests: Order of Tests Confidence Checks Automated and Manual test runs


Acceptance Testing

Contract Acceptance Testing

“A demonstration of the acceptance criteria”

Acceptance Criteria will have been defined in the contract

Before the software is accepted it is necessary to show that it matches its specification as defined in the criteria


Acceptance Testing

Other types if Acceptance Testing: Alpha Testing – customers do the testing

Beta Testing

Factory Acceptance Testing (FAT)


Acceptance Testing

Summary

Before software is released it should be subjected to Acceptance Testing

User representation in testing is VITAL

If the product does not pass UAT then a decision about implementation needs to be made


Functional System Test


Understand what functional system testing is

Understand the benefits of functional system testing


Functional System Testing

What is functional system testing?

Testing of the complete system

Ideally done by an independent test team

Two type – functional and non-functional



A Functional Requirement is“ A requirement that specifies a function that a system

or system component must perform”

Functional System testing is geared to checking the function of the system against specification

My be requirement based or business process based


Functional System Testing

Testing based on requirements

Requirements specification used to derive test case

System is testing to ensure the requirements are met



Testing carried out against Business Processes

Based on expected use of the system

Builds use cases – test cases that reflect actual or expected use of the system


Non-Functional System Testing


Understand what non-functional System Testing is

Understand the need for Non-Functional System Testing

Look at the various types of Non-Functional System Testing


Non-Functional System Testing

Non-Functional System Testing is defined as:

“ Testing of those requirements that do not relate to the Functionality e.g. Performance, Load, Usability, Security etc”


Non-Functional System Test

Security Testing“Testing whether the system meets specified security requirements”

Usability Testing“Testing the ease with which users can learn and use the product”

Load Testing“Testing geared to assessing the applications ability to deal with the

expected throughput of data and users

Stress Testing“Assesses individual components by exercising them to and beyond

the limits of expected use”

Performance Testing“Tests the efficiency of individual components of an application”



Other Non-Functional System Test

Volume Testing Recovery Testing Documentation Testing Storage Testing Installability Testing



Summary

Just because a systems functions have been tested doesn’t mean that testing is complete

There are a range of non-functional tests that need to be performed upon a system


Component Testing


Understand what Component Testing is

Look at the Component Testing Process

Look at the myriad types of Component Testing


Component Testing

What is Component Testing?“ The testing of an individual software component.

This is also known as Unit Testing”

What is a component? A minimal software item for which a separate specification

is available


Component Testing

Component Testing is the lowest form of testing

At the bottom of the V-Model

Each component is tested in isolation Prior to being integrated into the system

Involves testing the code itself Test the code in the greatest detail Usually done by the components developer


Component Testing

BS7925-2 Standard for Component Testing

Component Test Process: Component Test Planning Component Test Specification Component Test Execution Component Test Recording Checking for Component Test Completion


Maintenance Testing


Look at the challenges that testers face when an application changes post implementation

See how to ensure that maintenance applied to the system does not cause failures


Maintenance Testing

What is Maintenance Testing?

Testing of changes to existing, established systems. It is regression testing in the “Live” environment

Checking that the fixes have been made and that the system has not regressed


Maintenance Testing

Summary All established systems need maintenance from time to time

The changed code / function will need to be tested

A regression test will need to be executed to ensure that the change(s) have been made correctly and that they have not affected other parts of the system

Impact Analysis is key


MORNING BREAK


Black and White Box Testing

In this session we will Understand the differences between Black and

White Box testing and where they feature in the testing lifecycle

Understand how a systematic approach provides confidence

Understand how tools can be used to improve and increase productivity


Black and White Box Testing

What is Black Box Testing?

“ Test Case selection that is based on an analysis of the specification of a component without reference to it’s internal workings”

What is White Box Testing?

“ Test Case selection based on an analysis of the internal structure of a component”


Black Box Testing

Concentrates on the Business Function

Can be used throughout testing cycle

Dominates the later stages of testing although is relevant throughout the development lifecycle

Little / No Knowledge of the underlying code needed


White Box Testing

Also known as Structural Testing

Focuses on Lines of Code

Looks at specific conditions

Looks at the mechanics of the Application

Useful in the early stages of testing


Summary

Black box testing focuses of functionality

White box testing focuses on code

A systematic approach is needed for both:

Tests need to be planned, prepared, executed and verified Expected results need to be defined and understood Tools can help increase productivity and quality


Black Box Test Techniques

Who do we need black box test techniques:

Exhaustive testing is not possible Due to the constraints of time, money and resources

Therefore we must create a sub-set of test There must be achievable, but should not reduce coverage

We should also focus on areas of likely risk Those places where mistakes may occur



Each black box test technique has A method

how to do it

A test case design approach How to create test cases using the approach

A measurement technique Except Random & Syntax

See BS7925-2 for detailed information



Equivalence Partitioning Boundary Value Analysis State Transition Cause & Effect Graphing Syntax Testing Random Testing


Equivalence Partitioning

Uses a model of the component to partition input and output values into sets

Such that each value within a set can be reasonably expected to be treated in the same manner

Therefore only one example of that set needs to be input to or resultant from the component


Equivalence Partitioning

Equivalence Partitioning – Test Case Design

Inputs to the component

Partitions Exercised: Valid Partitions Invalid Partitions

Expected Results


BVP and EP

BREAK


Boundary Value Analysis

Uses a model of the computer to identify the values close to and on partitions boundaries

For input and output, valid and invalid data

Chosen specifically to exercise the boundaries of the area under test


Boundary Value Analysis – Test Case Design

Inputs to the component

Boundaries to be exercised Just below On Just Above

Expected Results


Example: Loan Amount

Loan Amount:

Invalid 499 500 Valid 9000 9001 Invalid

Valid Amounts: Integers

Condit ions Valid Part it ions

Invalid Part it ions

Valid Boundaries

Invalid Boundaries

Loan Amount

500 - 9000 < 500 500

Integers > 9000 9000

Non-integer

32767

-32768


Example: Car Insurance


Input Partitions & Boundaries

March 4, 2015101

Applicant’s Age: 21 25 41 60

0 - 20 21 - 24 25 - 40 41 - 60 >60

Parked

StreetGarage

Convictions

NoneParkingSpeedingBanned

Health

No IssuesSome IssuesSerious Issues


Output Partitions & Boundaries

Outputs

Total Score: 4 8 13 23

4 – 7

cheap

8 – 12

normal

23 – 37

refused

37

13 -22

pricey


Key to Partitions Table

Partition Input/output

Valid/Invalid

Description

P1 Input Valid Age = 0-20




P5 Input Valid Age = >60

P6 Input Valid Parked = Street

P7 Input Valid Parked = Garaged

P8 Input Valid Convictions = None

P9 Input Valid Convictions = Parking

P10 Input Valid Convictions = Speeding

P11 Input Valid Convictions = Banned

P12 Input Valid Health = No issues

P13 Input Valid Health = Some issues

P14 Input Valid Health = Serious issues

P15 Output Valid Output = Cheap Insurance

P16 Output Valid Output = Normal Insurance

P17 Output Valid Output = Expensive Insurance

P18 Output Valid Output = Refused


Key to Boundaries Table

KeyID or ‘Tag’

Input or Output

Defined in Spec?

Descriptive text

Boundary Input/output

Valid/Invalid

Description

B1 Input Valid Input age ‘0’





B6 Output Valid Output total ‘3’





Proprietary and Confidential105

Boundary Value Analysis TCs

Test Case 1 2 3 4 5 6

Input Age -1 0 1 20 21 22

Input Parked Garaged Garaged Garaged Garaged Garaged Garaged

Input Convictions None None None None None None

Input Health No issues No issues No issues No issues No issues No issues

Boundary tested B1 (Age=0) B2 (Age=21)

Valid/Invalid I V V V V V

Total - 13 13 13 8 8

Exp. Output - Expensive Expensive Expensive Normal Normal


Boundary Value Analysis TCs

Test Case 1 2 3 4 5 6

Input Age -1 0 1 20 21 22

Input Parked Garaged Garaged Garaged Garaged Garaged Garaged

Input Convictions None None None None None None

Input Health No issues No issues No issues No issues No issues No issues

Boundary tested B1 (Age=0) B2 (Age=21)

Valid/Invalid I V V V V V

Total - 13 13 13 8 8

Exp. Output - Expensive Expensive Expensive Normal Normal


Progress = Number of distinct boundary values executed

Total number of boundary values* 100%

Progress: Testing Techniques

Progress = Number of distinct partitions executed

Total number of partitions* 100%


Why Do Both EP And BVA?

Invalid partitions maybe easily missed.

If the test fails, is the whole partition wrong, or is a boundary in the wrong place…have to test mid-partition anyway.

With BVA we are testing extremes. This does not give confidence for typical scenarios.


LUNCH


State Transition Testing

May be useful to model the software

State Transition Diagram shows: States the software can occupy Transitions between the states Events that cause the transitions Actions that result from transitions


The Specification

Example

Windows ‘Folder’ Program


State Transition Diagram

Invert Selection (IS)

Create (C)

Delete (D)


0 Files(0F) 1 Selected File(SF) 1 Unselected File (UF)



Level 0-Switch Test Cases

Trans# Start State

Event Action Finish State

1 0F Create 1 File Selected

SF

2 0F Invert Selection

No files selected

0F

3 UF Invert Selection

1 File selected

SF

4 SF Delete 1 File Deleted

0F

5 SF Invert Selection

No files selected

UF


Logical Execution Order

Trans# Start State

Event Action Finish State

1 0F Invert Selection

No files selected

0F

2 0F Create 1 File Selected

SF

3 SF Invert Selection

No files selected

UF

4 UF Invert Selection

1 File selected

SF

5 SF Delete 1 File Deleted

0F


Level 1-Switch Test Cases

Test Case 1 2 3 4 5 6 7 8 9

Start State 0F SF UF UF SF SF 0F 0F 0F

Input IS IS IS IS D D C C IS

Next State 0F UF SF SF 0F 0F SF SF 0F

Input C IS IS D C IS D IS IS

Finish State SF SF UF 0F SF 0F 0F UF 0F


State Tables

R S CM

S1 AT/S3 N/S1 D/S2

S2 AD/S4 N/S2 T/S1

S3 N/S3 T/S1 N/S3

S4 N/S4 D/S2 N/S4

N PRESENTS NO OUTPUT THEREFORE AN INVALID TRANSITION


Test Basis: Digital Watch

Reset (R)

Set (S) Change Mode (CM)

Welcome to the digital era!


Change

Date (S4)

Display

Date (S2)

Display

Time (S1)

Change

Time (S3)

Reset (R)

Alter time (AT)

set (S)

display time (T)

Reset (R)

Alter Date (AD)

set (S)

display date (D)

Change mode (CM)

display date (D)

Change mode (CM)

display time (T)

State Transition Diagram


Summary – Black Box Techniques

Black box Testing concentrates on testing the features of the systems

Techniques enable us to maximise testing Creates an achievable set of tests that offer maximum

coverage Ensure possible areas of Risk are tested

Black box testing is relevant throughout the testing process


White Box Testing


Understand what White Box Testing is

Look at some of the different types of White Box Testing


White Box Testing

What is White Box Testing?

“Test case selection based on an analysis of the internal structure of a component”

Also known as Glass Box testing or Clear Box testing


White Box Testing

Why do we need White Box Techniques?

Provide formal structure to testing code

Enable us to measure how much of a component has been tested

Example <100 lines of code 100,000,000,000,000 possible paths At 1,000 tests per second would still take 3,170 years to test all paths


White Box Testing

To plan and design effective cases requires a knowledge of the

Programming language usedf

Database(s) used

Operating System(s) used

And ideally knowledge of the code itself


White Box Testing Techniques

BS7925-2 lists all the white box test techniques

Statement Testing Branch / Decision Testing Branch Condition Testing Branch Condition Combination Testing Modified Condition Decision Testing Linear Code Sequence & Jump Data Flow Testing


Statement Testing

“ A test case design technique for a component in which test cases are designed to execute statements”

“ Test cases are designed and run with the intention of executing every statement in a component”


Branch / Decision Testing

A technique used to execute all branches the code may take based on decisions made

Test cases designed to ensure all branches & decision points are covered


EXAMPLES

PRACTICAL EXAMPLES


Summary

White box testing can be done immediately after code is written Doesn’t need the complete system Does need knowledge of the code

A combination of all techniques are required for a successful test Don’t rely on just one technique

Control Flow Graphing is a pre-requisitie


DAY 3


Reviews and Test Process

What is Static Testing?

“Testing of an object without execution on a computer”

How is this done

By reviewing the system deliverables


Reviews and the Test Process

Why Review? To identify errors as soon as possible in the development

lifecycle

Reviews offer the chance to find errors in the system specifications

This should lead to Development productivity improvements Reduced development time-scales Lifetime cost reductions Reduced failure levels



When to review? As soon as an object is ready, before it is used as a product or the basis for the next step in development

What to review? Anything and everything can be reviewed Requirements, System and Program Specifications should be reviewed prior to publication

System Design deliverables should be reviewed both in terms of functionality and technical robustness



What should be reviewed?

Program Specifications should be reviewed before construction

Code should be reviewed before execution

Test plans should be reviewed before creating tests

Test Results should be reviewed before implementation



The cost of failures On-going reviews cost approximately 15% of development budget

This includes activities such as the review itself, metric analysis & process improvement

Reviews are highly cost effective Finding and fixing faults in reviews is significantly cheaper

than finding and fixing faults in later stages of testing


Reviews & the Test Process

The benefits of reviews

Reviews save time and money Development productivity improvements

People take greater care They have more pride in doing good work They have a greater understanding of what they are required to

deliver

Reduced development costs Reduced fault levels Reduced lifetime costs


Summary

Reviews enable us to ensure that the systems specification is correct and relates to the user requirements

Anything generated by a project can be reviewed

In order for them to be effective, reviews must be well managed


Types of Reviews

A process or meeting during which a work product, or set of work products, is presented to project personnel, managers, users or other interested parties for comment or approval’

BS7925-1


Informal Reviews

Informal ReviewsAs the least formal of all reviews, it can occur at any time and is largely unplanned. It will include conversations at the photocopier, coffee machine and during breaks. It large consist of questions like ‘ What do you think of this?’ and normal occurs between peers. No formal documentation is produce and no record is keep of the review.


Walkthroughs

Walkthroughs A review process in which a professional leads one or more members of the development team through a segment of a document that he or she has written while the other members ask questions and make comments about technique, style, possible error, violation of development standards, and other problems.


Technical Reviews

Technical/Peer Reviews A formal meeting at which work products, are presented to interested parties for comments and approval. Participants are often peers with no management participation. A technical expert may be present.


Inspections

Inspections A formal evaluation technique in which documents are examined in detail by a person or group other than the author to detect errors, violations of development standards, and other problems.


The Inspection Process

Fagan’s Inspection Process

Kic

k-of

f

Che

ckin

g

Logg

ing

Edi

ting

En

try

Exi

t

Process Improvement

Change Request

Software Development Stage

Next Development Stage


Review Purposes

Purposes:

Fault Finding

Education

Conse

nsus

Walkthroughs

Technical/Peer Reviews

Inspections


Summary

There are various types of reviews

Companies must decide which one(s) are best for them

In order to gain maximum benefit from them they must be organized and implemented


BREAK


Static Analysis


Understand what Static Analysis is

Look at some of the elements of Static Analysis Compiler Static Analysis Tools Data Flow Analysis Control Flow Analysis Complexity Analysis


Static Analysis

What is Static Analysis? Analysis of the code without dynamic execution

Attempting to identify errors in the code

Provides metrics on flow through SUT and its complexity

A form of automation – usually done with tools


Static Analysis

What do we how to find?

Unreachable code Undeclared variables Parameter type mismatches Uncalled functions and procedures Possible array boundary violations


Incident Management

An “incident” is when testing cannot progress as planned

“any significant, unplanned event that occurs during the testing that requires subsequent investigation and / or correction The application does not function as anticipated When actual results differ from expected results Components are missing

They must be raised against documentation, the SUT, the test environment or the tests themselves


Raising An Incident

Immediately when a test case fails• More than one test case may fail due to the same reason,

resulting in duplicate incidents• But … information is fresh in your mind

After the execution of the test set• Allows the one-to-many relationship between incidents and test

cases to be facilitated better• But … information is not so fresh in the mind


Incident Record


Incident Record

Other Incident Information Name of Tester Test Case ID Test Specification ID & version Test Script ID & version Software Build Traced Requirements Resolution Details (added by developer


Incident Lifecycle

Lifecycle ensures that an appropriate process for incident closure is followed May be used to monitor and improve testing Status and change histories should be maintained Resolution progress should be tracked


Incident Lifecycle

IEEE 1044 – Standard for Software anomalies

Submitted Assigned Opened Resolved Closed

Duplicate

Submit

Assign Open Resolve Valid

Reject

DuplicateDuplicate


Incident Management Systems

Use a database platform Each incident record is stored as a record in

the incident database Facilities to ensure incident lifecycle is

followed Some test management systems have built in

incident management systems


Quality Center

DEMO of Quality Center


Summary

An incident is “any significant, unplanned event that occurs during the testing that requires subsequent investigation and / or correction”

Incidents should be recorded and tracked Analysis of incidents will enable us to see

where problems arose and to aid in test process improvement


LUNCH


Risk-Based Testing

Currently a software testing buzz word

Risk determines the testing schedule

Higher risk test items are tested first

Higher risk test items are tested well

Dependant on thorough Risk Management


Three main activities: Identification of risk

– Does it exist?

Analysis of risk– How serious could it be?

Mitigation of risk– What should we do about it?


Risk Analysis

Risk Severity = Likelihood x Impact Risk Exposure = Severity x Frequency

If we use our quantitative values for risk then we are said to be doing ‘Risk-based testing’


Project and Product Risks

Example Project Risks Development team in different location from independent testing

team. New, unfamiliar technology introduced to the project. Project team inexperienced or not mature. Unrealistic time constraints.

Example Product Risks Failure will cause loss of revenue Failure will cause loss of life Failure will affect future business


Testing and Risk

Risk Management affects testing by: Defining an ordering of tests Suggesting that some tests are more important

than others Dictating the depth of testing to be performed Directing focus to most important tests (MITs) –

Prioritising Tests


Possible Risk Criterion

Impact of failure to stakeholders Likelihood of failure Visibility to the wider world Business priority Quantity of change undergone How error prone Criticality to business Technical complexity Difficulty of testing Cost of testing


DAY 4


Test Estimation, Monitoring & Control


Understand how we estimate how long we need for testing

Understand how we monitor the progress of testing once it has started

Understand what steps we take to ensure that the effort progresses as smoothly as possible


Test Estimation

The same as estimating for any project

We need to identify the number of tasks to be performed, the length of each test, the skills and resources required and the various dependencies

Testing has a high degree of dependencyYou cannot test something until it has been delivered Faults found need to be fixed and re-tested The environment must be available whenever a test is to

be run


Test Estimation

Factors to consider

Risk of failure Complexity of Code Criticality Coverage Stability of System Under Test


Test Estimation

Testing is a tool to aid Risk Management

Consider the cost of the business of failure of a feature

Test the most important features as soon as possible

Be prepared to repeat these tests


Test Estimation Techniques

Expert Based Techniques Previous Experience

Formula Based Techniques % of development effort Lines of code Function Point Analysis Test Point Analysis


BREAK


DEMO of Traceability Matrix


DEMO of DMEA Mainframe Application


DEMO of DMEA Web Application


LUNCH


Revision and Mock Exam

March 4, 2015176

Documents

powerpoint template for testing training