Page 1, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Ivica Crnkovic
Mälardalen University
Department of Computer Engineering
Page 2, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Software faults and failuresSoftware faults and failures
What does it mean that the software has failed?
The software does not do what the requirements describe
Reasons to the failures
Requirements incomplete, inconsistent, impossible to implement
Wrong interpretation of the requirements
The system design may contain a fault (misunderstanding requirements, wrong architecture, incomplete design)
Program design may contain a fault - wrong algorithm
The program code may be wrong - improper, incomplete, inconsistent implementation
Documentation can be wrong - can describe incorrectly the behavior of the system
Page 3, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Some other reasons to the failuresSome other reasons to the failures
Integration problems
The components integrated into a systems are not compatible (wrong components, wrong component versions)
The performance of the entire system does not correspond to the requirements (expectations)
The system requires too much resources
Not failures but still unsuccessful
Bad Graphical User Interface
Complicated process
Missed points with requirements (users actually wanted something else)
Page 4, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Defect classificationsDefect classifications
What type of defect?
When it occurred (when failure, when fault?)
Severity level?
What is the foot cause of the defect?
….
Page 5, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Specification/requirements
Environment/support
Documentation OtherDesign Code
ORIGIN: WHERE?
MODE:WHY?
TYPE
: W
HA
T?
Missing Unclear Wrong Changed Better way
HW interface
SW interface
User interface
Functionaldescription
Test HW
Test SW
Integration SW
Developmenttools
Logic
Computation
Data handling
Moduleinterface/
implementation
Standards
(Inter-)Processcommunications
Data definition
Module designLogic
description
Error checking
Standards
Fault classification (HP)Fault classification (HP)
Requirementsor
specifications
Functionality
Page 6, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Data handling6%
Documentation19%
Requirements5%
Hardware4%
Process/interprocess
5%Logic32%
Computation18%
Other code11%
Example:Faults statistics (HP)
What are the most frequent faultsWhat are the most frequent faults
Page 7, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Costs to repair the failuresCosts to repair the failures
Specification/requirements
IntegrationDesign Code
$$$ to fix the problems
Origin of problems
Page 8, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
How to avoid a late detection of the faults?How to avoid a late detection of the faults?
Use the development process that suits the best
Do proper planning
Ensure the quality in each process step
perform reviews
perform tests
Page 9, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
REQUIREMENTSANALYSIS
SYSTEMDESIGN
PROGRAM DESIGN
CODING
UNIT & INTE-GRATION TESTING
SYSTEMTESTING
OPERATION& MAINTENANCE
Verify design
Validate requirementsACCEPTANCE
TESTING
V Model - document producedV Model - document produced
RS
Acceptance testdescription
SD, FSSystem design test description
DS, CodeModule test description & code
Test analysis report.
Page 10, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
The testing processThe testing process
Sub-systemtesting
Moduletesting
Unittesting
Systemtesting
Acceptancetesting
Componenttesting
Integration testing Usertesting
Page 11, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Unittest
Unittest
Unittest
Integrationtest
Functiontest
Performancetest
Acceptancetest
Installationtest
Com
pone
nt c
ode
Com
pone
nt c
ode
.
.
.
Tested component
Test
ed c
ompo
nent
Integratedmodules
Functioningsystem
Verified,validatedsoftware
Acceptedsystem
SYSTEMIN USE!
Designspecifications
Systemfunctional
requirements
Othersoftware
requirements
Customerrequirementsspecification
Userenvironment
The testing process - detailsThe testing process - details
Page 12, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Program authors (module tests)
The developer group (module tests, component tests)
Test teams
Integration tests, subsystem tests
System tests
“Big configuration test”
Special customers (running alpha, beta releases)
Customers with support people (acceptance tests)
Who performs the tests?Who performs the tests?
New functions
Development Alpha releases Beta releases
Fixing defects
Officialrelease
Page 13, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Unit TestingUnit Testing
Different methods to test the code
Code review - go trough the code
Code author, or a review group in the developer team (peer review)
Code inspection
Inspect the code against a prepared list of concerns (data-types, style guide…)
Proving code correct
Formal verification
Choosing test cases
Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification
Page 14, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Black-box testingBlack-box testing
Approach to testing where the program is considered as a ‘black-box’
The program test cases are based on the system specification
Test planning can begin early in the software process
Ie
Input test data
OeOutput test results
System
Inputs causinganomalousbehaviour
Outputs which revealthe presence ofdefects
Page 15, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Interface testingInterface testing
Takes place when modules or sub-systems are integrated to create larger systems
Objectives are to detect faults due to interface errors or invalid assumptions about interfaces
Particularly important for object-oriented development as objects are defined by their interfaces
Testcases
BA
C
The test is applied to sub-systems, not to individual modules
Page 16, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Interfaces typesInterfaces types
Parameter interfaces
Data passed from one procedure to another
Shared memory interfaces
Block of memory is shared between procedures
Procedural interfaces
Sub-system encapsulates a set of procedures to be called by other sub-systems
Message passing interfaces
Sub-systems request services from other sub-systems
Page 17, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
White-box testing (structural testing)White-box testing (structural testing)
Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases
Objective is to exercise all program statements (not all path combinations)
Componentcode
Testoutputs
Test data
DerivesTests
Page 18, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Path testing Path testing
A white-box testing
Testing different combinations of program paths
Measuring the coverage percentile of the test
if-then-else loop-while case-of
Page 19, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
A
DCB
FE G
Integration testingIntegration testing
Test on the sub-systems levels
Testing how the components work together
Page 20, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
TestE
TestF
TestG
TestB,E,F
TestC
TestD,G
TestA,B,C,D,
E,F,G
Integration test - bottom-up principleIntegration test - bottom-up principle
A
DCB
FE G
Page 21, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
TestA
TestA,B,C,D
TestA,B,C,D,
E,F,G
Top-down testingTop-down testing
Test from the top level
all components are not necessary completed (and tested)
writing STUBS (which simulates the component)
A
DCB
FE G
Page 22, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
TestA
TestA,B,C,D
TestA,B,C,D,
E,F,G
TestB
TestC
TestD
TestE
TestC
TestD
TestF
TestG
Combination of Top-down and Down-top modelCombination of Top-down and Down-top model
A
DCB
FE G
Page 23, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
TestA
TestB,E,F Test
A,B,C,D,E,F,G
TestE
TestD,G
TestF
TestG
A
DCB
FE G
Sandwich IntegrationSandwich Integration
Page 24, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Milestone 1: Most critical featuresand shared components Design, code, prototype
Usability testing Daily builds
Feature integration Eliminate severe faults
Milestone 2: Desirable features Design, code, prototype
Usability testing Daily builds
Feature integration
Milestone 3: Least critical features Design, code, prototype
Usability testing Daily builds
Feature integration and completionRelease to manufacturing
Microsoft synch-and stabilize approach
Page 25, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Object-oriented systems testingObject-oriented systems testing
Less closely coupled systems. Objects are not necessarily integrated into sub-systems
Cluster testing. Test a group of cooperating objects
Thread testing. Test a processing thread as it weaves from object to object. Discussed later in real-time system testing
Complication with inheritance
Page 26, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Object-oriented systems testingObject-oriented systems testing
Modularity
Small methods
Reuse
Interfaces identified earlier
Encapsulation
Inheritance
Polymorphism
Dynamic binding
Complex Interfaces
More integration
Easier parts of testing Harder parts of testing
Page 27, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Automated testing toolsAutomated testing tools
Static Analyzers
Code analysisShow some potential program errors and low quality code (too many
branches, complicated structure, bad naming policy
Generate different graphs showing the program structureData analyzer (initialization, usage, etc.)
Dynamic Analyzers
Checking a running program
Memory leakage
non proper use of variables
Measuring timing
Measuring execution coverage
Page 28, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Probabilityof existenceof additional
faults
Number of faults found to date
When to stop with testing?When to stop with testing?
Page 29, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Requirementsanalysis
Systemdesign
Programdesign
Programimplementation
Unit/integration testing
System testing
Maintenance
Incorrect, missing or unclear requirementsIncorrect or unclear translation
Incorrect or unclear design specification
Incorrect or unclear design specificationMisinterpretation of system design
Misinterpretation of program designIncorrect documentationIncorrect syntax or semantics
Incomplete test procedures New faults introduced when old ones corrected
Incomplete test procedures
Incorrect user documentationPoor human factorsNew faults introduced when old one correctedChanges in requirements
Page 30, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
Integratedmodules
SYSTEMIN USE!
Functiontest
Performancetest
Acceptancetest
Installationtest
Functioningsystem
Verified,validatedsoftware
Acceptedsystem
Systemfunctional
requirements
Othersoftware
requirements
Customerrequirementsspecification
Userenvironment
Page 31, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
TEST PLAN
SYSTEM TESTFUNCTION
Function 1 1Function 2 3,4 . .
TEST SPECIFICATION TEST 1Requirements tested
Functions tested
Methods
Conditions
TEST SPECIFICATION TEST 2Requirements tested
Functions tested
Methods
Conditions
TEST DESCRIPTION TEST 1Test data
Test procedures1.
2.
TEST DESCRIPTION TEST 2Test data
Test procedures1.
2.
TEST ANALYSISREPORT
TEST 1ResultsPerform
test 1
Performtest 2
TEST ANALYSISREPORT
TEST 2Results
Page 32, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
S.P0204.6.10.3016FAULT REPORT
ORIGINATOR: Joe Bloggs
BRIEF TITLE: Exception 1 in dps_c.c line 620 raised by NAS
FULL DESCRIPTION Started NAS endurance and allowed it to run for a few minutes. Disabled the active NASlink (emulator switched to standby link), then re-enabled the disabled link and CDIS exceptioned as above. (I
think the re-enabling is a red herring.) (during database load)
ASSIGNED FOR EVALUATION TO: DATE:
CATEGORISATION: 0 1 2 3 Design Spec DocnSEND COPIES FOR INFORMATION TO:
EVALUATOR: DATE:
CONFIGURATION ID ASSIGNED TO PART
COMMENTS: dpo_s.c appears to try to use an invalid CID, instead of rejecting the message. AWJ
dpo_s.c
8/7/92
ITEMS CHANGED
CONFIGURATION ID IMPLEMENTOR/DATE REVIEWER/DATE BUILD/ISSUE NUM INTEGRATOR/DATE
COMMENTS:
CLOSED
FAULT CONTROLLER: DATE: 9/7/92
dpo_s.c v.10 AWJ 8/7/92 MAR 8/7/92 6.120 RA 8-7-92
Page 33, 04/19/23SE course, TestingMdh/IDt, Ivica Crnkovic ©
DISCREPANCY REPORT FORM
DRF Number:__________________________________________ Tester name:____________________________Date: ___________________________________ Time: ________________________________Test Number: ______________________________Script step executed when failure occurred: __________________________________________________________Descripton of failure: _____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Activities before occurrence of failure:______________________________________________________________________________________________________________________________________________________________________________Expected results:______________________________________________________________________________________________________________________________________________________________________________Requirements affected:______________________________________________________________________________________________________________________________________________________________________________Effect of failure on test:______________________________________________________________________________________________________________________________________________________________________________Effect of failure on system:______________________________________________________________________________________________________________________________________________________________________________Severity level:(LOW) 1 2 3 4 5 (HIGH)